Note: This is test shard 6 of 8.
[==========] Running 9 tests from 5 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from AdminCliTest
[ RUN ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250625 20:03:04.416000 9337 test_util.cc:276] Using random seed: -288504194
W20250625 20:03:05.570632 9337 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.108s user 0.423s sys 0.682s
W20250625 20:03:05.570982 9337 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.108s user 0.423s sys 0.682s
I20250625 20:03:05.573477 9337 ts_itest-base.cc:115] Starting cluster with:
I20250625 20:03:05.573669 9337 ts_itest-base.cc:116] --------------
I20250625 20:03:05.573819 9337 ts_itest-base.cc:117] 4 tablet servers
I20250625 20:03:05.573992 9337 ts_itest-base.cc:118] 3 replicas per TS
I20250625 20:03:05.574182 9337 ts_itest-base.cc:119] --------------
2025-06-25T20:03:05Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-25T20:03:05Z Disabled control of system clock
I20250625 20:03:05.611840 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.30.126:39337
--webserver_interface=127.9.30.126
--webserver_port=0
--builtin_ntp_servers=127.9.30.84:38121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.30.126:39337 with env {}
W20250625 20:03:05.928454 9351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:03:05.929009 9351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:03:05.929430 9351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:03:05.960809 9351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 20:03:05.961118 9351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:03:05.961401 9351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 20:03:05.961621 9351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 20:03:05.997442 9351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:38121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.30.126:39337
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.30.126:39337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.9.30.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:03:05.998790 9351 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:03:06.000559 9351 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:03:06.018014 9358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:03:06.018021 9357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:03:06.019160 9360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:03:07.132021 9359 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250625 20:03:07.132170 9351 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 20:03:07.135793 9351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:03:07.138828 9351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:03:07.140296 9351 hybrid_clock.cc:648] HybridClock initialized: now 1750881787140252 us; error 57 us; skew 500 ppm
I20250625 20:03:07.141104 9351 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:03:07.148339 9351 webserver.cc:469] Webserver started at http://127.9.30.126:46791/ using document root <none> and password file <none>
I20250625 20:03:07.149302 9351 fs_manager.cc:362] Metadata directory not provided
I20250625 20:03:07.149509 9351 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:03:07.149932 9351 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 20:03:07.154520 9351 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "539a9bc847d44d62aff87db143754dea"
format_stamp: "Formatted at 2025-06-25 20:03:07 on dist-test-slave-jhxp"
I20250625 20:03:07.155718 9351 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "539a9bc847d44d62aff87db143754dea"
format_stamp: "Formatted at 2025-06-25 20:03:07 on dist-test-slave-jhxp"
I20250625 20:03:07.163054 9351 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20250625 20:03:07.169373 9367 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:03:07.170493 9351 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250625 20:03:07.170822 9351 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
uuid: "539a9bc847d44d62aff87db143754dea"
format_stamp: "Formatted at 2025-06-25 20:03:07 on dist-test-slave-jhxp"
I20250625 20:03:07.171154 9351 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:03:07.219234 9351 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:03:07.220732 9351 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:03:07.221153 9351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:03:07.301232 9351 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.126:39337
I20250625 20:03:07.301298 9418 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.126:39337 every 8 connection(s)
I20250625 20:03:07.304059 9351 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250625 20:03:07.309739 9419 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:03:07.311901 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 9351
I20250625 20:03:07.312320 9337 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250625 20:03:07.330629 9419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea: Bootstrap starting.
I20250625 20:03:07.336700 9419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea: Neither blocks nor log segments found. Creating new log.
I20250625 20:03:07.339044 9419 log.cc:826] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea: Log is configured to *not* fsync() on all Append() calls
I20250625 20:03:07.344566 9419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea: No bootstrap required, opened a new log
I20250625 20:03:07.364912 9419 raft_consensus.cc:357] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "539a9bc847d44d62aff87db143754dea" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 39337 } }
I20250625 20:03:07.365615 9419 raft_consensus.cc:383] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:03:07.365849 9419 raft_consensus.cc:738] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 539a9bc847d44d62aff87db143754dea, State: Initialized, Role: FOLLOWER
I20250625 20:03:07.366590 9419 consensus_queue.cc:260] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "539a9bc847d44d62aff87db143754dea" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 39337 } }
I20250625 20:03:07.367115 9419 raft_consensus.cc:397] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 20:03:07.367419 9419 raft_consensus.cc:491] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 20:03:07.367746 9419 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:03:07.372848 9419 raft_consensus.cc:513] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "539a9bc847d44d62aff87db143754dea" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 39337 } }
I20250625 20:03:07.373648 9419 leader_election.cc:304] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 539a9bc847d44d62aff87db143754dea; no voters:
I20250625 20:03:07.375567 9419 leader_election.cc:290] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 20:03:07.377310 9424 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:03:07.379575 9419 sys_catalog.cc:564] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea [sys.catalog]: configured and running, proceeding with master startup.
I20250625 20:03:07.379702 9424 raft_consensus.cc:695] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea [term 1 LEADER]: Becoming Leader. State: Replica: 539a9bc847d44d62aff87db143754dea, State: Running, Role: LEADER
I20250625 20:03:07.380470 9424 consensus_queue.cc:237] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "539a9bc847d44d62aff87db143754dea" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 39337 } }
I20250625 20:03:07.391636 9425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "539a9bc847d44d62aff87db143754dea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "539a9bc847d44d62aff87db143754dea" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 39337 } } }
I20250625 20:03:07.392235 9425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea [sys.catalog]: This master's current role is: LEADER
I20250625 20:03:07.393100 9426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 539a9bc847d44d62aff87db143754dea. Latest consensus state: current_term: 1 leader_uuid: "539a9bc847d44d62aff87db143754dea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "539a9bc847d44d62aff87db143754dea" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 39337 } } }
I20250625 20:03:07.394677 9426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea [sys.catalog]: This master's current role is: LEADER
I20250625 20:03:07.398861 9432 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 20:03:07.413391 9432 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 20:03:07.432528 9432 catalog_manager.cc:1349] Generated new cluster ID: 4cb5f21cfbaa41ef87824a823a38699e
I20250625 20:03:07.432855 9432 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 20:03:07.454478 9432 catalog_manager.cc:1372] Generated new certificate authority record
I20250625 20:03:07.456024 9432 catalog_manager.cc:1506] Loading token signing keys...
I20250625 20:03:07.471259 9432 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea: Generated new TSK 0
I20250625 20:03:07.472200 9432 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 20:03:07.483587 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.30.65:0
--local_ip_for_outbound_sockets=127.9.30.65
--webserver_interface=127.9.30.65
--webserver_port=0
--tserver_master_addrs=127.9.30.126:39337
--builtin_ntp_servers=127.9.30.84:38121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250625 20:03:07.830014 9443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:03:07.830523 9443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:03:07.831015 9443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:03:07.862931 9443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:03:07.863852 9443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.30.65
I20250625 20:03:07.899849 9443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:38121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.30.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.9.30.65
--webserver_port=0
--tserver_master_addrs=127.9.30.126:39337
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.30.65
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:03:07.901180 9443 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:03:07.902809 9443 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:03:07.921322 9449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:03:07.924310 9452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:03:07.921747 9450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:03:09.027809 9443 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250625 20:03:09.028338 9451 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250625 20:03:09.031821 9443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:03:09.034605 9443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:03:09.036182 9443 hybrid_clock.cc:648] HybridClock initialized: now 1750881789036149 us; error 45 us; skew 500 ppm
I20250625 20:03:09.037281 9443 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:03:09.057312 9443 webserver.cc:469] Webserver started at http://127.9.30.65:45433/ using document root <none> and password file <none>
I20250625 20:03:09.058598 9443 fs_manager.cc:362] Metadata directory not provided
I20250625 20:03:09.058889 9443 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:03:09.059474 9443 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 20:03:09.065644 9443 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "f647acd9e30449209271b97bb83df9be"
format_stamp: "Formatted at 2025-06-25 20:03:09 on dist-test-slave-jhxp"
I20250625 20:03:09.066876 9443 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "f647acd9e30449209271b97bb83df9be"
format_stamp: "Formatted at 2025-06-25 20:03:09 on dist-test-slave-jhxp"
I20250625 20:03:09.074714 9443 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250625 20:03:09.081476 9459 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:03:09.082648 9443 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.003s
I20250625 20:03:09.082986 9443 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f647acd9e30449209271b97bb83df9be"
format_stamp: "Formatted at 2025-06-25 20:03:09 on dist-test-slave-jhxp"
I20250625 20:03:09.083361 9443 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:03:09.141834 9443 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:03:09.143438 9443 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:03:09.144495 9443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:03:09.147672 9443 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 20:03:09.152223 9443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 20:03:09.152473 9443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 20:03:09.152748 9443 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 20:03:09.152910 9443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 20:03:09.334801 9443 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.65:44139
I20250625 20:03:09.334936 9571 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.65:44139 every 8 connection(s)
I20250625 20:03:09.337471 9443 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250625 20:03:09.342209 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 9443
I20250625 20:03:09.342633 9337 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250625 20:03:09.348606 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.30.66:0
--local_ip_for_outbound_sockets=127.9.30.66
--webserver_interface=127.9.30.66
--webserver_port=0
--tserver_master_addrs=127.9.30.126:39337
--builtin_ntp_servers=127.9.30.84:38121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 20:03:09.360934 9572 heartbeater.cc:344] Connected to a master server at 127.9.30.126:39337
I20250625 20:03:09.361393 9572 heartbeater.cc:461] Registering TS with master...
I20250625 20:03:09.362408 9572 heartbeater.cc:507] Master 127.9.30.126:39337 requested a full tablet report, sending...
I20250625 20:03:09.365263 9384 ts_manager.cc:194] Registered new tserver with Master: f647acd9e30449209271b97bb83df9be (127.9.30.65:44139)
I20250625 20:03:09.367199 9384 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.30.65:35979
W20250625 20:03:09.653292 9576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:03:09.653818 9576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:03:09.654363 9576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:03:09.686033 9576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:03:09.686930 9576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.30.66
I20250625 20:03:09.722311 9576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:38121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.30.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.9.30.66
--webserver_port=0
--tserver_master_addrs=127.9.30.126:39337
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.30.66
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:03:09.723709 9576 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:03:09.725463 9576 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:03:09.743961 9583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:03:10.371152 9572 heartbeater.cc:499] Master 127.9.30.126:39337 was elected leader, sending a full tablet report...
W20250625 20:03:09.745172 9585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:03:09.743968 9582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:03:10.830842 9584 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250625 20:03:10.830886 9576 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 20:03:10.835281 9576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:03:10.838591 9576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:03:10.840025 9576 hybrid_clock.cc:648] HybridClock initialized: now 1750881790839986 us; error 60 us; skew 500 ppm
I20250625 20:03:10.840850 9576 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:03:10.848109 9576 webserver.cc:469] Webserver started at http://127.9.30.66:42321/ using document root <none> and password file <none>
I20250625 20:03:10.849061 9576 fs_manager.cc:362] Metadata directory not provided
I20250625 20:03:10.849287 9576 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:03:10.849745 9576 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 20:03:10.854265 9576 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "e8217b9056ab48e7ae7b115a4a50b26b"
format_stamp: "Formatted at 2025-06-25 20:03:10 on dist-test-slave-jhxp"
I20250625 20:03:10.855430 9576 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "e8217b9056ab48e7ae7b115a4a50b26b"
format_stamp: "Formatted at 2025-06-25 20:03:10 on dist-test-slave-jhxp"
I20250625 20:03:10.862605 9576 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20250625 20:03:10.868858 9592 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:03:10.869951 9576 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20250625 20:03:10.870275 9576 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "e8217b9056ab48e7ae7b115a4a50b26b"
format_stamp: "Formatted at 2025-06-25 20:03:10 on dist-test-slave-jhxp"
I20250625 20:03:10.870621 9576 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:03:10.924115 9576 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:03:10.925606 9576 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:03:10.926082 9576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:03:10.929100 9576 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 20:03:10.933221 9576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 20:03:10.933466 9576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 20:03:10.933709 9576 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 20:03:10.933862 9576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 20:03:11.092067 9576 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.66:45301
I20250625 20:03:11.092146 9704 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.66:45301 every 8 connection(s)
I20250625 20:03:11.094622 9576 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250625 20:03:11.098371 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 9576
I20250625 20:03:11.098863 9337 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250625 20:03:11.106355 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.30.67:0
--local_ip_for_outbound_sockets=127.9.30.67
--webserver_interface=127.9.30.67
--webserver_port=0
--tserver_master_addrs=127.9.30.126:39337
--builtin_ntp_servers=127.9.30.84:38121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 20:03:11.119334 9705 heartbeater.cc:344] Connected to a master server at 127.9.30.126:39337
I20250625 20:03:11.119917 9705 heartbeater.cc:461] Registering TS with master...
I20250625 20:03:11.121218 9705 heartbeater.cc:507] Master 127.9.30.126:39337 requested a full tablet report, sending...
I20250625 20:03:11.124197 9384 ts_manager.cc:194] Registered new tserver with Master: e8217b9056ab48e7ae7b115a4a50b26b (127.9.30.66:45301)
I20250625 20:03:11.125416 9384 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.30.66:54553
W20250625 20:03:11.417887 9709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:03:11.418401 9709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:03:11.418960 9709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:03:11.450131 9709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:03:11.450947 9709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.30.67
I20250625 20:03:11.486198 9709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:38121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.30.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.9.30.67
--webserver_port=0
--tserver_master_addrs=127.9.30.126:39337
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.30.67
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:03:11.487466 9709 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:03:11.489140 9709 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:03:11.507678 9715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:03:12.128696 9705 heartbeater.cc:499] Master 127.9.30.126:39337 was elected leader, sending a full tablet report...
W20250625 20:03:11.508414 9718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:03:11.508343 9716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:03:11.510218 9709 server_base.cc:1048] running on GCE node
I20250625 20:03:12.600461 9709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:03:12.603148 9709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:03:12.604657 9709 hybrid_clock.cc:648] HybridClock initialized: now 1750881792604617 us; error 61 us; skew 500 ppm
I20250625 20:03:12.605433 9709 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:03:12.612520 9709 webserver.cc:469] Webserver started at http://127.9.30.67:40621/ using document root <none> and password file <none>
I20250625 20:03:12.613433 9709 fs_manager.cc:362] Metadata directory not provided
I20250625 20:03:12.613620 9709 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:03:12.614045 9709 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 20:03:12.618670 9709 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "1f98e8ed16524aee9210e66284681b6c"
format_stamp: "Formatted at 2025-06-25 20:03:12 on dist-test-slave-jhxp"
I20250625 20:03:12.620003 9709 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "1f98e8ed16524aee9210e66284681b6c"
format_stamp: "Formatted at 2025-06-25 20:03:12 on dist-test-slave-jhxp"
I20250625 20:03:12.627723 9709 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20250625 20:03:12.633352 9725 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:03:12.634462 9709 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.003s
I20250625 20:03:12.634771 9709 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "1f98e8ed16524aee9210e66284681b6c"
format_stamp: "Formatted at 2025-06-25 20:03:12 on dist-test-slave-jhxp"
I20250625 20:03:12.635102 9709 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:03:12.691661 9709 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:03:12.693130 9709 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:03:12.693565 9709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:03:12.696579 9709 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 20:03:12.701287 9709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 20:03:12.701524 9709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 20:03:12.701798 9709 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 20:03:12.701963 9709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 20:03:12.854971 9709 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.67:41473
I20250625 20:03:12.855115 9837 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.67:41473 every 8 connection(s)
I20250625 20:03:12.857646 9709 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250625 20:03:12.868217 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 9709
I20250625 20:03:12.868707 9337 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250625 20:03:12.875397 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.30.68:0
--local_ip_for_outbound_sockets=127.9.30.68
--webserver_interface=127.9.30.68
--webserver_port=0
--tserver_master_addrs=127.9.30.126:39337
--builtin_ntp_servers=127.9.30.84:38121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 20:03:12.881700 9838 heartbeater.cc:344] Connected to a master server at 127.9.30.126:39337
I20250625 20:03:12.882117 9838 heartbeater.cc:461] Registering TS with master...
I20250625 20:03:12.883093 9838 heartbeater.cc:507] Master 127.9.30.126:39337 requested a full tablet report, sending...
I20250625 20:03:12.885174 9384 ts_manager.cc:194] Registered new tserver with Master: 1f98e8ed16524aee9210e66284681b6c (127.9.30.67:41473)
I20250625 20:03:12.886857 9384 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.30.67:53715
W20250625 20:03:13.202186 9842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:03:13.202697 9842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:03:13.203179 9842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:03:13.235221 9842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:03:13.236301 9842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.30.68
I20250625 20:03:13.271590 9842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:38121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.30.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.9.30.68
--webserver_port=0
--tserver_master_addrs=127.9.30.126:39337
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.30.68
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:03:13.272965 9842 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:03:13.274657 9842 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:03:13.291273 9849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:03:13.889909 9838 heartbeater.cc:499] Master 127.9.30.126:39337 was elected leader, sending a full tablet report...
W20250625 20:03:13.294395 9851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:03:13.291266 9848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:03:14.387739 9850 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250625 20:03:14.387822 9842 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 20:03:14.391667 9842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:03:14.393808 9842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:03:14.395164 9842 hybrid_clock.cc:648] HybridClock initialized: now 1750881794395129 us; error 54 us; skew 500 ppm
I20250625 20:03:14.396005 9842 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:03:14.405855 9842 webserver.cc:469] Webserver started at http://127.9.30.68:36593/ using document root <none> and password file <none>
I20250625 20:03:14.406814 9842 fs_manager.cc:362] Metadata directory not provided
I20250625 20:03:14.407042 9842 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:03:14.407498 9842 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 20:03:14.412119 9842 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "63ad3b1903ef4d248b94c3d22b8d7577"
format_stamp: "Formatted at 2025-06-25 20:03:14 on dist-test-slave-jhxp"
I20250625 20:03:14.413241 9842 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "63ad3b1903ef4d248b94c3d22b8d7577"
format_stamp: "Formatted at 2025-06-25 20:03:14 on dist-test-slave-jhxp"
I20250625 20:03:14.420626 9842 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20250625 20:03:14.426522 9859 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:03:14.427783 9842 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250625 20:03:14.428189 9842 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "63ad3b1903ef4d248b94c3d22b8d7577"
format_stamp: "Formatted at 2025-06-25 20:03:14 on dist-test-slave-jhxp"
I20250625 20:03:14.428524 9842 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:03:14.470469 9842 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:03:14.471941 9842 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:03:14.472358 9842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:03:14.474895 9842 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 20:03:14.479049 9842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 20:03:14.479269 9842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 20:03:14.479475 9842 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 20:03:14.479635 9842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250625 20:03:14.629050 9842 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.68:46783
I20250625 20:03:14.629675 9971 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.68:46783 every 8 connection(s)
I20250625 20:03:14.631520 9842 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250625 20:03:14.638836 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 9842
I20250625 20:03:14.639515 9337 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250625 20:03:14.654912 9972 heartbeater.cc:344] Connected to a master server at 127.9.30.126:39337
I20250625 20:03:14.655366 9972 heartbeater.cc:461] Registering TS with master...
I20250625 20:03:14.656432 9972 heartbeater.cc:507] Master 127.9.30.126:39337 requested a full tablet report, sending...
I20250625 20:03:14.658710 9384 ts_manager.cc:194] Registered new tserver with Master: 63ad3b1903ef4d248b94c3d22b8d7577 (127.9.30.68:46783)
I20250625 20:03:14.660255 9384 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.30.68:48207
I20250625 20:03:14.661233 9337 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250625 20:03:14.704720 9384 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55594:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250625 20:03:14.776573 9507 tablet_service.cc:1468] Processing CreateTablet for tablet f68caba2fc6f493692b164b37de304ff (DEFAULT_TABLE table=TestTable [id=33ad668e77234212aaecf2d067a2ae45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:03:14.776572 9640 tablet_service.cc:1468] Processing CreateTablet for tablet f68caba2fc6f493692b164b37de304ff (DEFAULT_TABLE table=TestTable [id=33ad668e77234212aaecf2d067a2ae45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:03:14.778717 9507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f68caba2fc6f493692b164b37de304ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:03:14.778852 9640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f68caba2fc6f493692b164b37de304ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:03:14.787518 9773 tablet_service.cc:1468] Processing CreateTablet for tablet f68caba2fc6f493692b164b37de304ff (DEFAULT_TABLE table=TestTable [id=33ad668e77234212aaecf2d067a2ae45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:03:14.789661 9773 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f68caba2fc6f493692b164b37de304ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:03:14.804200 9991 tablet_bootstrap.cc:492] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b: Bootstrap starting.
I20250625 20:03:14.806938 9992 tablet_bootstrap.cc:492] T f68caba2fc6f493692b164b37de304ff P f647acd9e30449209271b97bb83df9be: Bootstrap starting.
I20250625 20:03:14.813838 9992 tablet_bootstrap.cc:654] T f68caba2fc6f493692b164b37de304ff P f647acd9e30449209271b97bb83df9be: Neither blocks nor log segments found. Creating new log.
I20250625 20:03:14.815347 9991 tablet_bootstrap.cc:654] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b: Neither blocks nor log segments found. Creating new log.
I20250625 20:03:14.816202 9992 log.cc:826] T f68caba2fc6f493692b164b37de304ff P f647acd9e30449209271b97bb83df9be: Log is configured to *not* fsync() on all Append() calls
I20250625 20:03:14.818046 9991 log.cc:826] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b: Log is configured to *not* fsync() on all Append() calls
I20250625 20:03:14.822207 9993 tablet_bootstrap.cc:492] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c: Bootstrap starting.
I20250625 20:03:14.823136 9992 tablet_bootstrap.cc:492] T f68caba2fc6f493692b164b37de304ff P f647acd9e30449209271b97bb83df9be: No bootstrap required, opened a new log
I20250625 20:03:14.823709 9992 ts_tablet_manager.cc:1397] T f68caba2fc6f493692b164b37de304ff P f647acd9e30449209271b97bb83df9be: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.005s
I20250625 20:03:14.824801 9991 tablet_bootstrap.cc:492] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b: No bootstrap required, opened a new log
I20250625 20:03:14.825330 9991 ts_tablet_manager.cc:1397] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b: Time spent bootstrapping tablet: real 0.022s user 0.013s sys 0.003s
I20250625 20:03:14.828234 9993 tablet_bootstrap.cc:654] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c: Neither blocks nor log segments found. Creating new log.
I20250625 20:03:14.830575 9993 log.cc:826] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c: Log is configured to *not* fsync() on all Append() calls
I20250625 20:03:14.836107 9993 tablet_bootstrap.cc:492] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c: No bootstrap required, opened a new log
I20250625 20:03:14.836570 9993 ts_tablet_manager.cc:1397] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20250625 20:03:14.852762 9992 raft_consensus.cc:357] T f68caba2fc6f493692b164b37de304ff P f647acd9e30449209271b97bb83df9be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45301 } } peers { permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 } } peers { permanent_uuid: "f647acd9e30449209271b97bb83df9be" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44139 } }
I20250625 20:03:14.853787 9992 raft_consensus.cc:383] T f68caba2fc6f493692b164b37de304ff P f647acd9e30449209271b97bb83df9be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:03:14.854125 9992 raft_consensus.cc:738] T f68caba2fc6f493692b164b37de304ff P f647acd9e30449209271b97bb83df9be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f647acd9e30449209271b97bb83df9be, State: Initialized, Role: FOLLOWER
I20250625 20:03:14.855579 9991 raft_consensus.cc:357] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45301 } } peers { permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 } } peers { permanent_uuid: "f647acd9e30449209271b97bb83df9be" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44139 } }
I20250625 20:03:14.856598 9991 raft_consensus.cc:383] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:03:14.856166 9992 consensus_queue.cc:260] T f68caba2fc6f493692b164b37de304ff P f647acd9e30449209271b97bb83df9be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45301 } } peers { permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 } } peers { permanent_uuid: "f647acd9e30449209271b97bb83df9be" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44139 } }
I20250625 20:03:14.856940 9991 raft_consensus.cc:738] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8217b9056ab48e7ae7b115a4a50b26b, State: Initialized, Role: FOLLOWER
I20250625 20:03:14.857867 9991 consensus_queue.cc:260] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45301 } } peers { permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 } } peers { permanent_uuid: "f647acd9e30449209271b97bb83df9be" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44139 } }
I20250625 20:03:14.859731 9993 raft_consensus.cc:357] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45301 } } peers { permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 } } peers { permanent_uuid: "f647acd9e30449209271b97bb83df9be" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44139 } }
I20250625 20:03:14.860828 9993 raft_consensus.cc:383] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:03:14.861166 9993 raft_consensus.cc:738] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f98e8ed16524aee9210e66284681b6c, State: Initialized, Role: FOLLOWER
I20250625 20:03:14.862236 9993 consensus_queue.cc:260] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45301 } } peers { permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 } } peers { permanent_uuid: "f647acd9e30449209271b97bb83df9be" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44139 } }
I20250625 20:03:14.872803 9993 ts_tablet_manager.cc:1428] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c: Time spent starting tablet: real 0.036s user 0.027s sys 0.007s
I20250625 20:03:14.879374 9991 ts_tablet_manager.cc:1428] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b: Time spent starting tablet: real 0.054s user 0.034s sys 0.017s
I20250625 20:03:14.883790 9992 ts_tablet_manager.cc:1428] T f68caba2fc6f493692b164b37de304ff P f647acd9e30449209271b97bb83df9be: Time spent starting tablet: real 0.060s user 0.032s sys 0.027s
I20250625 20:03:14.899477 9997 raft_consensus.cc:491] T f68caba2fc6f493692b164b37de304ff P f647acd9e30449209271b97bb83df9be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:03:14.899961 9997 raft_consensus.cc:513] T f68caba2fc6f493692b164b37de304ff P f647acd9e30449209271b97bb83df9be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45301 } } peers { permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 } } peers { permanent_uuid: "f647acd9e30449209271b97bb83df9be" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44139 } }
I20250625 20:03:14.902462 9997 leader_election.cc:290] T f68caba2fc6f493692b164b37de304ff P f647acd9e30449209271b97bb83df9be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8217b9056ab48e7ae7b115a4a50b26b (127.9.30.66:45301), 1f98e8ed16524aee9210e66284681b6c (127.9.30.67:41473)
I20250625 20:03:14.916544 9660 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f68caba2fc6f493692b164b37de304ff" candidate_uuid: "f647acd9e30449209271b97bb83df9be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" is_pre_election: true
I20250625 20:03:14.917403 9660 raft_consensus.cc:2466] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f647acd9e30449209271b97bb83df9be in term 0.
I20250625 20:03:14.917791 9793 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f68caba2fc6f493692b164b37de304ff" candidate_uuid: "f647acd9e30449209271b97bb83df9be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f98e8ed16524aee9210e66284681b6c" is_pre_election: true
I20250625 20:03:14.918452 9793 raft_consensus.cc:2466] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f647acd9e30449209271b97bb83df9be in term 0.
I20250625 20:03:14.918503 9460 leader_election.cc:304] T f68caba2fc6f493692b164b37de304ff P f647acd9e30449209271b97bb83df9be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8217b9056ab48e7ae7b115a4a50b26b, f647acd9e30449209271b97bb83df9be; no voters:
I20250625 20:03:14.919225 9997 raft_consensus.cc:2802] T f68caba2fc6f493692b164b37de304ff P f647acd9e30449209271b97bb83df9be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:03:14.919535 9997 raft_consensus.cc:491] T f68caba2fc6f493692b164b37de304ff P f647acd9e30449209271b97bb83df9be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:03:14.919828 9997 raft_consensus.cc:3058] T f68caba2fc6f493692b164b37de304ff P f647acd9e30449209271b97bb83df9be [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:03:14.924656 9997 raft_consensus.cc:513] T f68caba2fc6f493692b164b37de304ff P f647acd9e30449209271b97bb83df9be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45301 } } peers { permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 } } peers { permanent_uuid: "f647acd9e30449209271b97bb83df9be" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44139 } }
I20250625 20:03:14.926070 9997 leader_election.cc:290] T f68caba2fc6f493692b164b37de304ff P f647acd9e30449209271b97bb83df9be [CANDIDATE]: Term 1 election: Requested vote from peers e8217b9056ab48e7ae7b115a4a50b26b (127.9.30.66:45301), 1f98e8ed16524aee9210e66284681b6c (127.9.30.67:41473)
I20250625 20:03:14.926872 9660 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f68caba2fc6f493692b164b37de304ff" candidate_uuid: "f647acd9e30449209271b97bb83df9be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8217b9056ab48e7ae7b115a4a50b26b"
I20250625 20:03:14.927047 9793 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f68caba2fc6f493692b164b37de304ff" candidate_uuid: "f647acd9e30449209271b97bb83df9be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f98e8ed16524aee9210e66284681b6c"
I20250625 20:03:14.927309 9660 raft_consensus.cc:3058] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:03:14.927513 9793 raft_consensus.cc:3058] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:03:14.932171 9660 raft_consensus.cc:2466] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f647acd9e30449209271b97bb83df9be in term 1.
I20250625 20:03:14.932904 9793 raft_consensus.cc:2466] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f647acd9e30449209271b97bb83df9be in term 1.
I20250625 20:03:14.933113 9460 leader_election.cc:304] T f68caba2fc6f493692b164b37de304ff P f647acd9e30449209271b97bb83df9be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8217b9056ab48e7ae7b115a4a50b26b, f647acd9e30449209271b97bb83df9be; no voters:
I20250625 20:03:14.933769 9997 raft_consensus.cc:2802] T f68caba2fc6f493692b164b37de304ff P f647acd9e30449209271b97bb83df9be [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:03:14.935227 9997 raft_consensus.cc:695] T f68caba2fc6f493692b164b37de304ff P f647acd9e30449209271b97bb83df9be [term 1 LEADER]: Becoming Leader. State: Replica: f647acd9e30449209271b97bb83df9be, State: Running, Role: LEADER
I20250625 20:03:14.936108 9997 consensus_queue.cc:237] T f68caba2fc6f493692b164b37de304ff P f647acd9e30449209271b97bb83df9be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45301 } } peers { permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 } } peers { permanent_uuid: "f647acd9e30449209271b97bb83df9be" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44139 } }
I20250625 20:03:14.948222 9384 catalog_manager.cc:5582] T f68caba2fc6f493692b164b37de304ff P f647acd9e30449209271b97bb83df9be reported cstate change: term changed from 0 to 1, leader changed from <none> to f647acd9e30449209271b97bb83df9be (127.9.30.65). New cstate: current_term: 1 leader_uuid: "f647acd9e30449209271b97bb83df9be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f647acd9e30449209271b97bb83df9be" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44139 } health_report { overall_health: HEALTHY } } }
I20250625 20:03:14.992820 9337 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250625 20:03:14.996695 9337 ts_itest-base.cc:246] Waiting for 1 tablets on tserver f647acd9e30449209271b97bb83df9be to finish bootstrapping
I20250625 20:03:15.012037 9337 ts_itest-base.cc:246] Waiting for 1 tablets on tserver e8217b9056ab48e7ae7b115a4a50b26b to finish bootstrapping
I20250625 20:03:15.025796 9337 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 1f98e8ed16524aee9210e66284681b6c to finish bootstrapping
I20250625 20:03:15.040088 9337 kudu-admin-test.cc:709] Waiting for Master to see the current replicas...
I20250625 20:03:15.044556 9337 kudu-admin-test.cc:716] Tablet locations:
tablet_locations {
tablet_id: "f68caba2fc6f493692b164b37de304ff"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 2
role: LEADER
}
}
ts_infos {
permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b"
rpc_addresses {
host: "127.9.30.66"
port: 45301
}
}
ts_infos {
permanent_uuid: "1f98e8ed16524aee9210e66284681b6c"
rpc_addresses {
host: "127.9.30.67"
port: 41473
}
}
ts_infos {
permanent_uuid: "f647acd9e30449209271b97bb83df9be"
rpc_addresses {
host: "127.9.30.65"
port: 44139
}
}
W20250625 20:03:15.098824 9573 tablet.cc:2378] T f68caba2fc6f493692b164b37de304ff P f647acd9e30449209271b97bb83df9be: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250625 20:03:15.103633 9706 tablet.cc:2378] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250625 20:03:15.115339 9839 tablet.cc:2378] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 20:03:15.320983 9997 consensus_queue.cc:1035] T f68caba2fc6f493692b164b37de304ff P f647acd9e30449209271b97bb83df9be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:03:15.349489 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 9443
W20250625 20:03:15.375309 9727 connection.cc:537] server connection from 127.9.30.65:49365 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 20:03:15.375324 9594 connection.cc:537] server connection from 127.9.30.65:36191 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250625 20:03:15.376701 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 9351
I20250625 20:03:15.403414 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.30.126:39337
--webserver_interface=127.9.30.126
--webserver_port=46791
--builtin_ntp_servers=127.9.30.84:38121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.30.126:39337 with env {}
I20250625 20:03:15.664119 9972 heartbeater.cc:499] Master 127.9.30.126:39337 was elected leader, sending a full tablet report...
W20250625 20:03:15.668093 9972 heartbeater.cc:646] Failed to heartbeat to 127.9.30.126:39337 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.30.126:39337: connect: Connection refused (error 111)
W20250625 20:03:15.698738 10009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:03:15.699347 10009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:03:15.700074 10009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:03:15.731590 10009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 20:03:15.731902 10009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:03:15.732113 10009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 20:03:15.732311 10009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 20:03:15.767823 10009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:38121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.30.126:39337
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.30.126:39337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.9.30.126
--webserver_port=46791
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:03:15.769160 10009 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:03:15.770761 10009 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:03:15.784540 10016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:03:15.895668 9705 heartbeater.cc:646] Failed to heartbeat to 127.9.30.126:39337 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.30.126:39337: connect: Connection refused (error 111)
W20250625 20:03:16.364166 9838 heartbeater.cc:646] Failed to heartbeat to 127.9.30.126:39337 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.30.126:39337: connect: Connection refused (error 111)
I20250625 20:03:16.772131 10025 raft_consensus.cc:491] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:03:16.772514 10025 raft_consensus.cc:513] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45301 } } peers { permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 } } peers { permanent_uuid: "f647acd9e30449209271b97bb83df9be" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44139 } }
I20250625 20:03:16.774643 10025 leader_election.cc:290] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f98e8ed16524aee9210e66284681b6c (127.9.30.67:41473), f647acd9e30449209271b97bb83df9be (127.9.30.65:44139)
W20250625 20:03:16.780222 9596 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.30.65:44139: connect: Connection refused (error 111)
W20250625 20:03:16.785893 9596 leader_election.cc:336] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f647acd9e30449209271b97bb83df9be (127.9.30.65:44139): Network error: Client connection negotiation failed: client connection to 127.9.30.65:44139: connect: Connection refused (error 111)
I20250625 20:03:16.787931 9793 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f68caba2fc6f493692b164b37de304ff" candidate_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f98e8ed16524aee9210e66284681b6c" is_pre_election: true
I20250625 20:03:16.789567 9593 leader_election.cc:304] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8217b9056ab48e7ae7b115a4a50b26b; no voters: 1f98e8ed16524aee9210e66284681b6c, f647acd9e30449209271b97bb83df9be
I20250625 20:03:16.790426 10025 raft_consensus.cc:2747] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 20:03:16.836828 10030 raft_consensus.cc:491] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f647acd9e30449209271b97bb83df9be)
I20250625 20:03:16.837248 10030 raft_consensus.cc:513] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45301 } } peers { permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 } } peers { permanent_uuid: "f647acd9e30449209271b97bb83df9be" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44139 } }
I20250625 20:03:16.839953 10030 leader_election.cc:290] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e8217b9056ab48e7ae7b115a4a50b26b (127.9.30.66:45301), f647acd9e30449209271b97bb83df9be (127.9.30.65:44139)
W20250625 20:03:16.844931 9729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.30.65:44139: connect: Connection refused (error 111)
W20250625 20:03:16.851626 9729 leader_election.cc:336] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f647acd9e30449209271b97bb83df9be (127.9.30.65:44139): Network error: Client connection negotiation failed: client connection to 127.9.30.65:44139: connect: Connection refused (error 111)
I20250625 20:03:16.862890 9660 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f68caba2fc6f493692b164b37de304ff" candidate_uuid: "1f98e8ed16524aee9210e66284681b6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" is_pre_election: true
I20250625 20:03:16.863513 9660 raft_consensus.cc:2466] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f98e8ed16524aee9210e66284681b6c in term 1.
I20250625 20:03:16.864955 9726 leader_election.cc:304] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f98e8ed16524aee9210e66284681b6c, e8217b9056ab48e7ae7b115a4a50b26b; no voters: f647acd9e30449209271b97bb83df9be
I20250625 20:03:16.865826 10030 raft_consensus.cc:2802] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250625 20:03:16.866254 10030 raft_consensus.cc:491] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c [term 1 FOLLOWER]: Starting leader election (detected failure of leader f647acd9e30449209271b97bb83df9be)
I20250625 20:03:16.866595 10030 raft_consensus.cc:3058] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c [term 1 FOLLOWER]: Advancing to term 2
I20250625 20:03:16.873163 10030 raft_consensus.cc:513] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45301 } } peers { permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 } } peers { permanent_uuid: "f647acd9e30449209271b97bb83df9be" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44139 } }
I20250625 20:03:16.875922 9660 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f68caba2fc6f493692b164b37de304ff" candidate_uuid: "1f98e8ed16524aee9210e66284681b6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e8217b9056ab48e7ae7b115a4a50b26b"
I20250625 20:03:16.876479 9660 raft_consensus.cc:3058] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [term 1 FOLLOWER]: Advancing to term 2
I20250625 20:03:16.878923 10030 leader_election.cc:290] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c [CANDIDATE]: Term 2 election: Requested vote from peers e8217b9056ab48e7ae7b115a4a50b26b (127.9.30.66:45301), f647acd9e30449209271b97bb83df9be (127.9.30.65:44139)
W20250625 20:03:16.879480 9729 leader_election.cc:336] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f647acd9e30449209271b97bb83df9be (127.9.30.65:44139): Network error: Client connection negotiation failed: client connection to 127.9.30.65:44139: connect: Connection refused (error 111)
I20250625 20:03:16.883046 9660 raft_consensus.cc:2466] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f98e8ed16524aee9210e66284681b6c in term 2.
I20250625 20:03:16.884109 9726 leader_election.cc:304] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f98e8ed16524aee9210e66284681b6c, e8217b9056ab48e7ae7b115a4a50b26b; no voters: f647acd9e30449209271b97bb83df9be
I20250625 20:03:16.884845 10030 raft_consensus.cc:2802] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c [term 2 FOLLOWER]: Leader election won for term 2
I20250625 20:03:16.886722 10030 raft_consensus.cc:695] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c [term 2 LEADER]: Becoming Leader. State: Replica: 1f98e8ed16524aee9210e66284681b6c, State: Running, Role: LEADER
I20250625 20:03:16.887648 10030 consensus_queue.cc:237] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45301 } } peers { permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 } } peers { permanent_uuid: "f647acd9e30449209271b97bb83df9be" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44139 } }
W20250625 20:03:15.785305 10019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:03:15.786897 10009 server_base.cc:1048] running on GCE node
W20250625 20:03:15.785231 10017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:03:16.908023 10009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:03:16.910640 10009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:03:16.912016 10009 hybrid_clock.cc:648] HybridClock initialized: now 1750881796911977 us; error 54 us; skew 500 ppm
I20250625 20:03:16.912786 10009 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:03:16.918841 10009 webserver.cc:469] Webserver started at http://127.9.30.126:46791/ using document root <none> and password file <none>
I20250625 20:03:16.919829 10009 fs_manager.cc:362] Metadata directory not provided
I20250625 20:03:16.920051 10009 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:03:16.928020 10009 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.001s
I20250625 20:03:16.932556 10040 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:03:16.933665 10009 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250625 20:03:16.933991 10009 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
uuid: "539a9bc847d44d62aff87db143754dea"
format_stamp: "Formatted at 2025-06-25 20:03:07 on dist-test-slave-jhxp"
I20250625 20:03:16.935930 10009 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:03:17.008270 10009 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:03:17.009706 10009 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:03:17.010111 10009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:03:17.078594 10009 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.126:39337
I20250625 20:03:17.078667 10091 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.126:39337 every 8 connection(s)
I20250625 20:03:17.081442 10009 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250625 20:03:17.084450 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 10009
I20250625 20:03:17.085048 9337 kudu-admin-test.cc:735] Forcing unsafe config change on tserver e8217b9056ab48e7ae7b115a4a50b26b
I20250625 20:03:17.094579 10092 sys_catalog.cc:263] Verifying existing consensus state
I20250625 20:03:17.099437 10092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea: Bootstrap starting.
I20250625 20:03:17.138043 10092 log.cc:826] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea: Log is configured to *not* fsync() on all Append() calls
I20250625 20:03:17.161502 10092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 20:03:17.162313 10092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea: Bootstrap complete.
I20250625 20:03:17.184661 10092 raft_consensus.cc:357] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "539a9bc847d44d62aff87db143754dea" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 39337 } }
I20250625 20:03:17.186959 10092 raft_consensus.cc:738] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 539a9bc847d44d62aff87db143754dea, State: Initialized, Role: FOLLOWER
I20250625 20:03:17.187885 10092 consensus_queue.cc:260] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "539a9bc847d44d62aff87db143754dea" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 39337 } }
I20250625 20:03:17.188413 10092 raft_consensus.cc:397] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 20:03:17.188699 10092 raft_consensus.cc:491] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 20:03:17.189008 10092 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea [term 1 FOLLOWER]: Advancing to term 2
I20250625 20:03:17.194976 10092 raft_consensus.cc:513] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "539a9bc847d44d62aff87db143754dea" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 39337 } }
I20250625 20:03:17.195782 10092 leader_election.cc:304] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 539a9bc847d44d62aff87db143754dea; no voters:
I20250625 20:03:17.197860 10092 leader_election.cc:290] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea [CANDIDATE]: Term 2 election: Requested vote from peers
I20250625 20:03:17.198424 10096 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea [term 2 FOLLOWER]: Leader election won for term 2
I20250625 20:03:17.201607 10096 raft_consensus.cc:695] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea [term 2 LEADER]: Becoming Leader. State: Replica: 539a9bc847d44d62aff87db143754dea, State: Running, Role: LEADER
I20250625 20:03:17.202560 10096 consensus_queue.cc:237] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "539a9bc847d44d62aff87db143754dea" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 39337 } }
I20250625 20:03:17.203197 10092 sys_catalog.cc:564] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea [sys.catalog]: configured and running, proceeding with master startup.
I20250625 20:03:17.212735 10097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "539a9bc847d44d62aff87db143754dea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "539a9bc847d44d62aff87db143754dea" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 39337 } } }
I20250625 20:03:17.214604 10097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea [sys.catalog]: This master's current role is: LEADER
I20250625 20:03:17.213991 10098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 539a9bc847d44d62aff87db143754dea. Latest consensus state: current_term: 2 leader_uuid: "539a9bc847d44d62aff87db143754dea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "539a9bc847d44d62aff87db143754dea" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 39337 } } }
I20250625 20:03:17.216231 10098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea [sys.catalog]: This master's current role is: LEADER
I20250625 20:03:17.219053 10103 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 20:03:17.231523 10103 catalog_manager.cc:671] Loaded metadata for table TestTable [id=33ad668e77234212aaecf2d067a2ae45]
I20250625 20:03:17.238960 10103 tablet_loader.cc:96] loaded metadata for tablet f68caba2fc6f493692b164b37de304ff (table TestTable [id=33ad668e77234212aaecf2d067a2ae45])
I20250625 20:03:17.240521 10103 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 20:03:17.245185 10103 catalog_manager.cc:1261] Loaded cluster ID: 4cb5f21cfbaa41ef87824a823a38699e
I20250625 20:03:17.245582 10103 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 20:03:17.254213 10103 catalog_manager.cc:1506] Loading token signing keys...
I20250625 20:03:17.259047 10103 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 539a9bc847d44d62aff87db143754dea: Loaded TSK: 0
I20250625 20:03:17.260831 10103 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 20:03:17.302966 9660 raft_consensus.cc:1273] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [term 2 FOLLOWER]: Refusing update from remote peer 1f98e8ed16524aee9210e66284681b6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250625 20:03:17.304556 10030 consensus_queue.cc:1035] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
W20250625 20:03:17.324411 9729 consensus_peers.cc:487] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c -> Peer f647acd9e30449209271b97bb83df9be (127.9.30.65:44139): Couldn't send request to peer f647acd9e30449209271b97bb83df9be. Status: Network error: Client connection negotiation failed: client connection to 127.9.30.65:44139: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250625 20:03:17.340535 9705 heartbeater.cc:344] Connected to a master server at 127.9.30.126:39337
I20250625 20:03:17.348843 10057 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" instance_seqno: 1750881791056046) as {username='slave'} at 127.9.30.66:57901; Asking this server to re-register.
I20250625 20:03:17.350872 9705 heartbeater.cc:461] Registering TS with master...
I20250625 20:03:17.351537 9705 heartbeater.cc:507] Master 127.9.30.126:39337 requested a full tablet report, sending...
I20250625 20:03:17.355544 10057 ts_manager.cc:194] Registered new tserver with Master: e8217b9056ab48e7ae7b115a4a50b26b (127.9.30.66:45301)
I20250625 20:03:17.359896 10057 catalog_manager.cc:5582] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b reported cstate change: term changed from 1 to 2, leader changed from f647acd9e30449209271b97bb83df9be (127.9.30.65) to 1f98e8ed16524aee9210e66284681b6c (127.9.30.67). New cstate: current_term: 2 leader_uuid: "1f98e8ed16524aee9210e66284681b6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45301 } } peers { permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 } } peers { permanent_uuid: "f647acd9e30449209271b97bb83df9be" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44139 } } }
I20250625 20:03:17.369194 9838 heartbeater.cc:344] Connected to a master server at 127.9.30.126:39337
I20250625 20:03:17.373278 10056 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" instance_seqno: 1750881792819862) as {username='slave'} at 127.9.30.67:54729; Asking this server to re-register.
I20250625 20:03:17.375115 9838 heartbeater.cc:461] Registering TS with master...
I20250625 20:03:17.375859 9838 heartbeater.cc:507] Master 127.9.30.126:39337 requested a full tablet report, sending...
I20250625 20:03:17.379287 10056 ts_manager.cc:194] Registered new tserver with Master: 1f98e8ed16524aee9210e66284681b6c (127.9.30.67:41473)
W20250625 20:03:17.490010 10094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:03:17.490617 10094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:03:17.522379 10094 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250625 20:03:17.697953 9972 heartbeater.cc:344] Connected to a master server at 127.9.30.126:39337
I20250625 20:03:17.698549 9972 heartbeater.cc:499] Master 127.9.30.126:39337 was elected leader, sending a full tablet report...
I20250625 20:03:17.700847 10057 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "63ad3b1903ef4d248b94c3d22b8d7577" instance_seqno: 1750881794593508) as {username='slave'} at 127.9.30.68:40191; Asking this server to re-register.
I20250625 20:03:17.702636 9972 heartbeater.cc:461] Registering TS with master...
I20250625 20:03:17.703339 9972 heartbeater.cc:507] Master 127.9.30.126:39337 requested a full tablet report, sending...
I20250625 20:03:17.706701 10057 ts_manager.cc:194] Registered new tserver with Master: 63ad3b1903ef4d248b94c3d22b8d7577 (127.9.30.68:46783)
W20250625 20:03:18.878161 10094 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.313s user 0.502s sys 0.763s
W20250625 20:03:18.878566 10094 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.313s user 0.502s sys 0.763s
I20250625 20:03:18.932073 9660 tablet_service.cc:1905] Received UnsafeChangeConfig RPC: dest_uuid: "e8217b9056ab48e7ae7b115a4a50b26b"
tablet_id: "f68caba2fc6f493692b164b37de304ff"
caller_id: "kudu-tools"
new_config {
peers {
permanent_uuid: "1f98e8ed16524aee9210e66284681b6c"
}
peers {
permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b"
}
}
from {username='slave'} at 127.0.0.1:33408
W20250625 20:03:18.933177 9660 raft_consensus.cc:2216] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [term 2 FOLLOWER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45301 } } peers { permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 } } peers { permanent_uuid: "f647acd9e30449209271b97bb83df9be" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44139 } }NEW CONFIG: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45301 } } peers { permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 } } unsafe_config_change: true
I20250625 20:03:18.934023 9660 raft_consensus.cc:3058] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [term 2 FOLLOWER]: Advancing to term 3
I20250625 20:03:19.263793 9660 raft_consensus.cc:1238] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [term 3 FOLLOWER]: Rejecting Update request from peer 1f98e8ed16524aee9210e66284681b6c for earlier term 2. Current term is 3. Ops: []
I20250625 20:03:19.264891 10030 consensus_queue.cc:1046] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45301 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250625 20:03:19.266402 10142 raft_consensus.cc:3053] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c [term 2 LEADER]: Stepping down as leader of term 2
I20250625 20:03:19.266641 10142 raft_consensus.cc:738] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1f98e8ed16524aee9210e66284681b6c, State: Running, Role: LEADER
I20250625 20:03:19.267230 10142 consensus_queue.cc:260] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45301 } } peers { permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 } } peers { permanent_uuid: "f647acd9e30449209271b97bb83df9be" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44139 } }
I20250625 20:03:19.268311 10142 raft_consensus.cc:3058] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c [term 2 FOLLOWER]: Advancing to term 3
I20250625 20:03:20.440943 10146 raft_consensus.cc:491] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools)
I20250625 20:03:20.441498 10146 raft_consensus.cc:513] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45301 } } peers { permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 } } unsafe_config_change: true
I20250625 20:03:20.442746 10146 leader_election.cc:290] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1f98e8ed16524aee9210e66284681b6c (127.9.30.67:41473)
I20250625 20:03:20.443883 9793 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f68caba2fc6f493692b164b37de304ff" candidate_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1f98e8ed16524aee9210e66284681b6c" is_pre_election: true
I20250625 20:03:20.444363 9793 raft_consensus.cc:2466] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8217b9056ab48e7ae7b115a4a50b26b in term 3.
I20250625 20:03:20.445350 9593 leader_election.cc:304] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1f98e8ed16524aee9210e66284681b6c, e8217b9056ab48e7ae7b115a4a50b26b; no voters:
I20250625 20:03:20.445977 10146 raft_consensus.cc:2802] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250625 20:03:20.446244 10146 raft_consensus.cc:491] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [term 3 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools)
I20250625 20:03:20.446470 10146 raft_consensus.cc:3058] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [term 3 FOLLOWER]: Advancing to term 4
I20250625 20:03:20.451774 10146 raft_consensus.cc:513] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45301 } } peers { permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 } } unsafe_config_change: true
I20250625 20:03:20.452809 10146 leader_election.cc:290] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [CANDIDATE]: Term 4 election: Requested vote from peers 1f98e8ed16524aee9210e66284681b6c (127.9.30.67:41473)
I20250625 20:03:20.453796 9793 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f68caba2fc6f493692b164b37de304ff" candidate_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1f98e8ed16524aee9210e66284681b6c"
I20250625 20:03:20.454222 9793 raft_consensus.cc:3058] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c [term 3 FOLLOWER]: Advancing to term 4
I20250625 20:03:20.458230 9793 raft_consensus.cc:2466] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8217b9056ab48e7ae7b115a4a50b26b in term 4.
I20250625 20:03:20.459120 9593 leader_election.cc:304] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1f98e8ed16524aee9210e66284681b6c, e8217b9056ab48e7ae7b115a4a50b26b; no voters:
I20250625 20:03:20.459769 10146 raft_consensus.cc:2802] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [term 4 FOLLOWER]: Leader election won for term 4
I20250625 20:03:20.460561 10146 raft_consensus.cc:695] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [term 4 LEADER]: Becoming Leader. State: Replica: e8217b9056ab48e7ae7b115a4a50b26b, State: Running, Role: LEADER
I20250625 20:03:20.461380 10146 consensus_queue.cc:237] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45301 } } peers { permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 } } unsafe_config_change: true
I20250625 20:03:20.467949 10057 catalog_manager.cc:5582] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b reported cstate change: term changed from 2 to 4, leader changed from 1f98e8ed16524aee9210e66284681b6c (127.9.30.67) to e8217b9056ab48e7ae7b115a4a50b26b (127.9.30.66), now has a pending config: VOTER e8217b9056ab48e7ae7b115a4a50b26b (127.9.30.66), VOTER 1f98e8ed16524aee9210e66284681b6c (127.9.30.67). New cstate: current_term: 4 leader_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f647acd9e30449209271b97bb83df9be" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44139 } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45301 } } peers { permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 } } unsafe_config_change: true }
I20250625 20:03:21.008025 9793 raft_consensus.cc:1273] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c [term 4 FOLLOWER]: Refusing update from remote peer e8217b9056ab48e7ae7b115a4a50b26b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250625 20:03:21.009321 10153 consensus_queue.cc:1035] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250625 20:03:21.018677 10156 raft_consensus.cc:2953] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [term 4 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER f647acd9e30449209271b97bb83df9be (127.9.30.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45301 } } peers { permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 } } unsafe_config_change: true }
I20250625 20:03:21.020498 9793 raft_consensus.cc:2953] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c [term 4 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER f647acd9e30449209271b97bb83df9be (127.9.30.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45301 } } peers { permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 } } unsafe_config_change: true }
I20250625 20:03:21.032313 10057 catalog_manager.cc:5582] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c reported cstate change: config changed from index -1 to 3, VOTER f647acd9e30449209271b97bb83df9be (127.9.30.65) evicted, no longer has a pending config: VOTER e8217b9056ab48e7ae7b115a4a50b26b (127.9.30.66), VOTER 1f98e8ed16524aee9210e66284681b6c (127.9.30.67). New cstate: current_term: 4 leader_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45301 } } peers { permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 } } unsafe_config_change: true }
W20250625 20:03:21.041792 10057 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet f68caba2fc6f493692b164b37de304ff on TS f647acd9e30449209271b97bb83df9be: Not found: failed to reset TS proxy: Could not find TS for UUID f647acd9e30449209271b97bb83df9be
I20250625 20:03:21.059845 9660 consensus_queue.cc:237] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45301 } } peers { permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 } } peers { permanent_uuid: "63ad3b1903ef4d248b94c3d22b8d7577" member_type: NON_VOTER last_known_addr { host: "127.9.30.68" port: 46783 } attrs { promote: true } } unsafe_config_change: true
I20250625 20:03:21.066684 9793 raft_consensus.cc:1273] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c [term 4 FOLLOWER]: Refusing update from remote peer e8217b9056ab48e7ae7b115a4a50b26b: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250625 20:03:21.067968 10153 consensus_queue.cc:1035] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250625 20:03:21.073498 10156 raft_consensus.cc:2953] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER 63ad3b1903ef4d248b94c3d22b8d7577 (127.9.30.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45301 } } peers { permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 } } peers { permanent_uuid: "63ad3b1903ef4d248b94c3d22b8d7577" member_type: NON_VOTER last_known_addr { host: "127.9.30.68" port: 46783 } attrs { promote: true } } unsafe_config_change: true }
W20250625 20:03:21.075815 9593 consensus_peers.cc:487] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b -> Peer 63ad3b1903ef4d248b94c3d22b8d7577 (127.9.30.68:46783): Couldn't send request to peer 63ad3b1903ef4d248b94c3d22b8d7577. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f68caba2fc6f493692b164b37de304ff. This is attempt 1: this message will repeat every 5th retry.
I20250625 20:03:21.075116 9793 raft_consensus.cc:2953] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER 63ad3b1903ef4d248b94c3d22b8d7577 (127.9.30.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45301 } } peers { permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 } } peers { permanent_uuid: "63ad3b1903ef4d248b94c3d22b8d7577" member_type: NON_VOTER last_known_addr { host: "127.9.30.68" port: 46783 } attrs { promote: true } } unsafe_config_change: true }
I20250625 20:03:21.082815 10041 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f68caba2fc6f493692b164b37de304ff with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250625 20:03:21.084860 10057 catalog_manager.cc:5582] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c reported cstate change: config changed from index 3 to 5, NON_VOTER 63ad3b1903ef4d248b94c3d22b8d7577 (127.9.30.68) added. New cstate: current_term: 4 leader_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45301 } } peers { permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 } } peers { permanent_uuid: "63ad3b1903ef4d248b94c3d22b8d7577" member_type: NON_VOTER last_known_addr { host: "127.9.30.68" port: 46783 } attrs { promote: true } } unsafe_config_change: true }
W20250625 20:03:21.094468 10042 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet f68caba2fc6f493692b164b37de304ff on TS f647acd9e30449209271b97bb83df9be failed: Not found: failed to reset TS proxy: Could not find TS for UUID f647acd9e30449209271b97bb83df9be
I20250625 20:03:21.538293 10167 ts_tablet_manager.cc:927] T f68caba2fc6f493692b164b37de304ff P 63ad3b1903ef4d248b94c3d22b8d7577: Initiating tablet copy from peer e8217b9056ab48e7ae7b115a4a50b26b (127.9.30.66:45301)
I20250625 20:03:21.540581 10167 tablet_copy_client.cc:323] T f68caba2fc6f493692b164b37de304ff P 63ad3b1903ef4d248b94c3d22b8d7577: tablet copy: Beginning tablet copy session from remote peer at address 127.9.30.66:45301
I20250625 20:03:21.550627 9680 tablet_copy_service.cc:140] P e8217b9056ab48e7ae7b115a4a50b26b: Received BeginTabletCopySession request for tablet f68caba2fc6f493692b164b37de304ff from peer 63ad3b1903ef4d248b94c3d22b8d7577 ({username='slave'} at 127.9.30.68:39273)
I20250625 20:03:21.551110 9680 tablet_copy_service.cc:161] P e8217b9056ab48e7ae7b115a4a50b26b: Beginning new tablet copy session on tablet f68caba2fc6f493692b164b37de304ff from peer 63ad3b1903ef4d248b94c3d22b8d7577 at {username='slave'} at 127.9.30.68:39273: session id = 63ad3b1903ef4d248b94c3d22b8d7577-f68caba2fc6f493692b164b37de304ff
I20250625 20:03:21.556443 9680 tablet_copy_source_session.cc:215] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b: Tablet Copy: opened 0 blocks and 1 log segments
I20250625 20:03:21.561559 10167 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f68caba2fc6f493692b164b37de304ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:03:21.579514 10167 tablet_copy_client.cc:806] T f68caba2fc6f493692b164b37de304ff P 63ad3b1903ef4d248b94c3d22b8d7577: tablet copy: Starting download of 0 data blocks...
I20250625 20:03:21.580121 10167 tablet_copy_client.cc:670] T f68caba2fc6f493692b164b37de304ff P 63ad3b1903ef4d248b94c3d22b8d7577: tablet copy: Starting download of 1 WAL segments...
I20250625 20:03:21.583757 10167 tablet_copy_client.cc:538] T f68caba2fc6f493692b164b37de304ff P 63ad3b1903ef4d248b94c3d22b8d7577: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250625 20:03:21.590062 10167 tablet_bootstrap.cc:492] T f68caba2fc6f493692b164b37de304ff P 63ad3b1903ef4d248b94c3d22b8d7577: Bootstrap starting.
I20250625 20:03:21.601565 10167 log.cc:826] T f68caba2fc6f493692b164b37de304ff P 63ad3b1903ef4d248b94c3d22b8d7577: Log is configured to *not* fsync() on all Append() calls
I20250625 20:03:21.612274 10167 tablet_bootstrap.cc:492] T f68caba2fc6f493692b164b37de304ff P 63ad3b1903ef4d248b94c3d22b8d7577: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 20:03:21.612944 10167 tablet_bootstrap.cc:492] T f68caba2fc6f493692b164b37de304ff P 63ad3b1903ef4d248b94c3d22b8d7577: Bootstrap complete.
I20250625 20:03:21.613502 10167 ts_tablet_manager.cc:1397] T f68caba2fc6f493692b164b37de304ff P 63ad3b1903ef4d248b94c3d22b8d7577: Time spent bootstrapping tablet: real 0.024s user 0.023s sys 0.000s
I20250625 20:03:21.630720 10167 raft_consensus.cc:357] T f68caba2fc6f493692b164b37de304ff P 63ad3b1903ef4d248b94c3d22b8d7577 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45301 } } peers { permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 } } peers { permanent_uuid: "63ad3b1903ef4d248b94c3d22b8d7577" member_type: NON_VOTER last_known_addr { host: "127.9.30.68" port: 46783 } attrs { promote: true } } unsafe_config_change: true
I20250625 20:03:21.631561 10167 raft_consensus.cc:738] T f68caba2fc6f493692b164b37de304ff P 63ad3b1903ef4d248b94c3d22b8d7577 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 63ad3b1903ef4d248b94c3d22b8d7577, State: Initialized, Role: LEARNER
I20250625 20:03:21.632222 10167 consensus_queue.cc:260] T f68caba2fc6f493692b164b37de304ff P 63ad3b1903ef4d248b94c3d22b8d7577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45301 } } peers { permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 } } peers { permanent_uuid: "63ad3b1903ef4d248b94c3d22b8d7577" member_type: NON_VOTER last_known_addr { host: "127.9.30.68" port: 46783 } attrs { promote: true } } unsafe_config_change: true
I20250625 20:03:21.636075 10167 ts_tablet_manager.cc:1428] T f68caba2fc6f493692b164b37de304ff P 63ad3b1903ef4d248b94c3d22b8d7577: Time spent starting tablet: real 0.022s user 0.024s sys 0.000s
I20250625 20:03:21.638254 9680 tablet_copy_service.cc:342] P e8217b9056ab48e7ae7b115a4a50b26b: Request end of tablet copy session 63ad3b1903ef4d248b94c3d22b8d7577-f68caba2fc6f493692b164b37de304ff received from {username='slave'} at 127.9.30.68:39273
I20250625 20:03:21.638707 9680 tablet_copy_service.cc:434] P e8217b9056ab48e7ae7b115a4a50b26b: ending tablet copy session 63ad3b1903ef4d248b94c3d22b8d7577-f68caba2fc6f493692b164b37de304ff on tablet f68caba2fc6f493692b164b37de304ff with peer 63ad3b1903ef4d248b94c3d22b8d7577
I20250625 20:03:21.914425 9927 raft_consensus.cc:1215] T f68caba2fc6f493692b164b37de304ff P 63ad3b1903ef4d248b94c3d22b8d7577 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.4->[4.5-4.5] Dedup: 4.5->[]
W20250625 20:03:22.261773 10042 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet f68caba2fc6f493692b164b37de304ff on TS f647acd9e30449209271b97bb83df9be failed: Not found: failed to reset TS proxy: Could not find TS for UUID f647acd9e30449209271b97bb83df9be
I20250625 20:03:22.419981 10172 raft_consensus.cc:1062] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b: attempting to promote NON_VOTER 63ad3b1903ef4d248b94c3d22b8d7577 to VOTER
I20250625 20:03:22.421608 10172 consensus_queue.cc:237] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45301 } } peers { permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 } } peers { permanent_uuid: "63ad3b1903ef4d248b94c3d22b8d7577" member_type: VOTER last_known_addr { host: "127.9.30.68" port: 46783 } attrs { promote: false } } unsafe_config_change: true
I20250625 20:03:22.426517 9927 raft_consensus.cc:1273] T f68caba2fc6f493692b164b37de304ff P 63ad3b1903ef4d248b94c3d22b8d7577 [term 4 LEARNER]: Refusing update from remote peer e8217b9056ab48e7ae7b115a4a50b26b: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250625 20:03:22.427145 9793 raft_consensus.cc:1273] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c [term 4 FOLLOWER]: Refusing update from remote peer e8217b9056ab48e7ae7b115a4a50b26b: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250625 20:03:22.427929 10172 consensus_queue.cc:1035] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [LEADER]: Connected to new peer: Peer: permanent_uuid: "63ad3b1903ef4d248b94c3d22b8d7577" member_type: VOTER last_known_addr { host: "127.9.30.68" port: 46783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250625 20:03:22.428627 10174 consensus_queue.cc:1035] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250625 20:03:22.436064 10176 raft_consensus.cc:2953] T f68caba2fc6f493692b164b37de304ff P e8217b9056ab48e7ae7b115a4a50b26b [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 63ad3b1903ef4d248b94c3d22b8d7577 (127.9.30.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45301 } } peers { permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 } } peers { permanent_uuid: "63ad3b1903ef4d248b94c3d22b8d7577" member_type: VOTER last_known_addr { host: "127.9.30.68" port: 46783 } attrs { promote: false } } unsafe_config_change: true }
I20250625 20:03:22.437484 9793 raft_consensus.cc:2953] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 63ad3b1903ef4d248b94c3d22b8d7577 (127.9.30.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45301 } } peers { permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 } } peers { permanent_uuid: "63ad3b1903ef4d248b94c3d22b8d7577" member_type: VOTER last_known_addr { host: "127.9.30.68" port: 46783 } attrs { promote: false } } unsafe_config_change: true }
I20250625 20:03:22.437709 9927 raft_consensus.cc:2953] T f68caba2fc6f493692b164b37de304ff P 63ad3b1903ef4d248b94c3d22b8d7577 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 63ad3b1903ef4d248b94c3d22b8d7577 (127.9.30.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45301 } } peers { permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 } } peers { permanent_uuid: "63ad3b1903ef4d248b94c3d22b8d7577" member_type: VOTER last_known_addr { host: "127.9.30.68" port: 46783 } attrs { promote: false } } unsafe_config_change: true }
I20250625 20:03:22.447662 10057 catalog_manager.cc:5582] T f68caba2fc6f493692b164b37de304ff P 1f98e8ed16524aee9210e66284681b6c reported cstate change: config changed from index 5 to 6, 63ad3b1903ef4d248b94c3d22b8d7577 (127.9.30.68) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45301 } } peers { permanent_uuid: "1f98e8ed16524aee9210e66284681b6c" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 41473 } } peers { permanent_uuid: "63ad3b1903ef4d248b94c3d22b8d7577" member_type: VOTER last_known_addr { host: "127.9.30.68" port: 46783 } attrs { promote: false } } unsafe_config_change: true }
I20250625 20:03:22.471079 9337 kudu-admin-test.cc:751] Waiting for Master to see new config...
I20250625 20:03:22.485095 9337 kudu-admin-test.cc:756] Tablet locations:
tablet_locations {
tablet_id: "f68caba2fc6f493692b164b37de304ff"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: LEADER
}
interned_replicas {
ts_info_idx: 1
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "e8217b9056ab48e7ae7b115a4a50b26b"
rpc_addresses {
host: "127.9.30.66"
port: 45301
}
}
ts_infos {
permanent_uuid: "1f98e8ed16524aee9210e66284681b6c"
rpc_addresses {
host: "127.9.30.67"
port: 41473
}
}
ts_infos {
permanent_uuid: "63ad3b1903ef4d248b94c3d22b8d7577"
rpc_addresses {
host: "127.9.30.68"
port: 46783
}
}
I20250625 20:03:22.487299 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 9576
I20250625 20:03:22.508158 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 9709
I20250625 20:03:22.527832 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 9842
I20250625 20:03:22.550652 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 10009
2025-06-25T20:03:22Z chronyd exiting
[ OK ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes (18191 ms)
[ RUN ] AdminCliTest.TestGracefulSpecificLeaderStepDown
I20250625 20:03:22.605772 9337 test_util.cc:276] Using random seed: -270314323
I20250625 20:03:22.611811 9337 ts_itest-base.cc:115] Starting cluster with:
I20250625 20:03:22.612001 9337 ts_itest-base.cc:116] --------------
I20250625 20:03:22.612166 9337 ts_itest-base.cc:117] 3 tablet servers
I20250625 20:03:22.612304 9337 ts_itest-base.cc:118] 3 replicas per TS
I20250625 20:03:22.612443 9337 ts_itest-base.cc:119] --------------
2025-06-25T20:03:22Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-25T20:03:22Z Disabled control of system clock
I20250625 20:03:22.648627 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.30.126:36001
--webserver_interface=127.9.30.126
--webserver_port=0
--builtin_ntp_servers=127.9.30.84:38045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.30.126:36001
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250625 20:03:22.940480 10194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:03:22.941069 10194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:03:22.941524 10194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:03:22.972520 10194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 20:03:22.972865 10194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:03:22.973116 10194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 20:03:22.973344 10194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 20:03:23.015448 10194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:38045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.9.30.126:36001
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.30.126:36001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.9.30.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:03:23.017233 10194 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:03:23.019634 10194 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:03:23.036566 10200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:03:24.584914 10194 thread.cc:640] OpenStack (cloud detector) Time spent creating pthread: real 1.548s user 0.608s sys 0.937s
W20250625 20:03:24.585376 10194 thread.cc:606] OpenStack (cloud detector) Time spent starting thread: real 1.549s user 0.608s sys 0.937s
W20250625 20:03:24.435657 10199 debug-util.cc:398] Leaking SignalData structure 0x7b0800008680 after lost signal to thread 10194
W20250625 20:03:23.036584 10201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:03:24.586275 10202 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1548 milliseconds
I20250625 20:03:24.587076 10194 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250625 20:03:24.587149 10203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:03:24.590543 10194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:03:24.593044 10194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:03:24.594411 10194 hybrid_clock.cc:648] HybridClock initialized: now 1750881804594371 us; error 48 us; skew 500 ppm
I20250625 20:03:24.595227 10194 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:03:24.601351 10194 webserver.cc:469] Webserver started at http://127.9.30.126:44483/ using document root <none> and password file <none>
I20250625 20:03:24.602295 10194 fs_manager.cc:362] Metadata directory not provided
I20250625 20:03:24.602520 10194 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:03:24.602962 10194 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 20:03:24.607831 10194 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "4b6f92fa9b0c4c72ac3aa32d44e869f2"
format_stamp: "Formatted at 2025-06-25 20:03:24 on dist-test-slave-jhxp"
I20250625 20:03:24.609032 10194 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "4b6f92fa9b0c4c72ac3aa32d44e869f2"
format_stamp: "Formatted at 2025-06-25 20:03:24 on dist-test-slave-jhxp"
I20250625 20:03:24.616231 10194 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20250625 20:03:24.621845 10211 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:03:24.622901 10194 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.001s
I20250625 20:03:24.623245 10194 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
uuid: "4b6f92fa9b0c4c72ac3aa32d44e869f2"
format_stamp: "Formatted at 2025-06-25 20:03:24 on dist-test-slave-jhxp"
I20250625 20:03:24.623579 10194 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:03:24.675019 10194 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:03:24.676544 10194 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:03:24.677021 10194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:03:24.746232 10194 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.126:36001
I20250625 20:03:24.746307 10262 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.126:36001 every 8 connection(s)
I20250625 20:03:24.748984 10194 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250625 20:03:24.752450 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 10194
I20250625 20:03:24.753091 9337 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250625 20:03:24.755141 10263 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:03:24.780239 10263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b6f92fa9b0c4c72ac3aa32d44e869f2: Bootstrap starting.
I20250625 20:03:24.785830 10263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b6f92fa9b0c4c72ac3aa32d44e869f2: Neither blocks nor log segments found. Creating new log.
I20250625 20:03:24.787533 10263 log.cc:826] T 00000000000000000000000000000000 P 4b6f92fa9b0c4c72ac3aa32d44e869f2: Log is configured to *not* fsync() on all Append() calls
I20250625 20:03:24.792279 10263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b6f92fa9b0c4c72ac3aa32d44e869f2: No bootstrap required, opened a new log
I20250625 20:03:24.810081 10263 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4b6f92fa9b0c4c72ac3aa32d44e869f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6f92fa9b0c4c72ac3aa32d44e869f2" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 36001 } }
I20250625 20:03:24.810750 10263 raft_consensus.cc:383] T 00000000000000000000000000000000 P 4b6f92fa9b0c4c72ac3aa32d44e869f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:03:24.810952 10263 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4b6f92fa9b0c4c72ac3aa32d44e869f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b6f92fa9b0c4c72ac3aa32d44e869f2, State: Initialized, Role: FOLLOWER
I20250625 20:03:24.811643 10263 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b6f92fa9b0c4c72ac3aa32d44e869f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6f92fa9b0c4c72ac3aa32d44e869f2" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 36001 } }
I20250625 20:03:24.812139 10263 raft_consensus.cc:397] T 00000000000000000000000000000000 P 4b6f92fa9b0c4c72ac3aa32d44e869f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 20:03:24.812367 10263 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4b6f92fa9b0c4c72ac3aa32d44e869f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 20:03:24.812614 10263 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4b6f92fa9b0c4c72ac3aa32d44e869f2 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:03:24.817106 10263 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4b6f92fa9b0c4c72ac3aa32d44e869f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6f92fa9b0c4c72ac3aa32d44e869f2" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 36001 } }
I20250625 20:03:24.817800 10263 leader_election.cc:304] T 00000000000000000000000000000000 P 4b6f92fa9b0c4c72ac3aa32d44e869f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b6f92fa9b0c4c72ac3aa32d44e869f2; no voters:
I20250625 20:03:24.819382 10263 leader_election.cc:290] T 00000000000000000000000000000000 P 4b6f92fa9b0c4c72ac3aa32d44e869f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 20:03:24.820091 10268 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4b6f92fa9b0c4c72ac3aa32d44e869f2 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:03:24.822149 10268 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4b6f92fa9b0c4c72ac3aa32d44e869f2 [term 1 LEADER]: Becoming Leader. State: Replica: 4b6f92fa9b0c4c72ac3aa32d44e869f2, State: Running, Role: LEADER
I20250625 20:03:24.822913 10268 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b6f92fa9b0c4c72ac3aa32d44e869f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6f92fa9b0c4c72ac3aa32d44e869f2" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 36001 } }
I20250625 20:03:24.823931 10263 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b6f92fa9b0c4c72ac3aa32d44e869f2 [sys.catalog]: configured and running, proceeding with master startup.
I20250625 20:03:24.832238 10270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b6f92fa9b0c4c72ac3aa32d44e869f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b6f92fa9b0c4c72ac3aa32d44e869f2. Latest consensus state: current_term: 1 leader_uuid: "4b6f92fa9b0c4c72ac3aa32d44e869f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6f92fa9b0c4c72ac3aa32d44e869f2" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 36001 } } }
I20250625 20:03:24.833196 10270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b6f92fa9b0c4c72ac3aa32d44e869f2 [sys.catalog]: This master's current role is: LEADER
I20250625 20:03:24.833310 10269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b6f92fa9b0c4c72ac3aa32d44e869f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b6f92fa9b0c4c72ac3aa32d44e869f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6f92fa9b0c4c72ac3aa32d44e869f2" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 36001 } } }
I20250625 20:03:24.833993 10269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b6f92fa9b0c4c72ac3aa32d44e869f2 [sys.catalog]: This master's current role is: LEADER
I20250625 20:03:24.841302 10277 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 20:03:24.852751 10277 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 20:03:24.868575 10277 catalog_manager.cc:1349] Generated new cluster ID: e7621bf597934fd0b678dca4f2c60cf8
I20250625 20:03:24.868907 10277 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 20:03:24.884020 10277 catalog_manager.cc:1372] Generated new certificate authority record
I20250625 20:03:24.885596 10277 catalog_manager.cc:1506] Loading token signing keys...
I20250625 20:03:24.900008 10277 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 4b6f92fa9b0c4c72ac3aa32d44e869f2: Generated new TSK 0
I20250625 20:03:24.901013 10277 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 20:03:24.912784 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.30.65:0
--local_ip_for_outbound_sockets=127.9.30.65
--webserver_interface=127.9.30.65
--webserver_port=0
--tserver_master_addrs=127.9.30.126:36001
--builtin_ntp_servers=127.9.30.84:38045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250625 20:03:25.228408 10287 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250625 20:03:25.229072 10287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:03:25.229333 10287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:03:25.229825 10287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:03:25.261770 10287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:03:25.262669 10287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.30.65
I20250625 20:03:25.300422 10287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:38045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.30.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.9.30.65
--webserver_port=0
--tserver_master_addrs=127.9.30.126:36001
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.30.65
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:03:25.301798 10287 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:03:25.303488 10287 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:03:25.321367 10294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:03:25.323189 10296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:03:25.321374 10293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:03:26.433902 10295 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1110 milliseconds
I20250625 20:03:26.434065 10287 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 20:03:26.435580 10287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:03:26.438426 10287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:03:26.439930 10287 hybrid_clock.cc:648] HybridClock initialized: now 1750881806439854 us; error 84 us; skew 500 ppm
I20250625 20:03:26.441015 10287 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:03:26.454921 10287 webserver.cc:469] Webserver started at http://127.9.30.65:38813/ using document root <none> and password file <none>
I20250625 20:03:26.456281 10287 fs_manager.cc:362] Metadata directory not provided
I20250625 20:03:26.456568 10287 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:03:26.457155 10287 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 20:03:26.463853 10287 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "72c05ce85e7342ed889e1ecc2678ece5"
format_stamp: "Formatted at 2025-06-25 20:03:26 on dist-test-slave-jhxp"
I20250625 20:03:26.465440 10287 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "72c05ce85e7342ed889e1ecc2678ece5"
format_stamp: "Formatted at 2025-06-25 20:03:26 on dist-test-slave-jhxp"
I20250625 20:03:26.474886 10287 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.009s sys 0.001s
I20250625 20:03:26.481076 10303 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:03:26.482262 10287 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20250625 20:03:26.482591 10287 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "72c05ce85e7342ed889e1ecc2678ece5"
format_stamp: "Formatted at 2025-06-25 20:03:26 on dist-test-slave-jhxp"
I20250625 20:03:26.482942 10287 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:03:26.535034 10287 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:03:26.536548 10287 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:03:26.537010 10287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:03:26.539937 10287 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 20:03:26.544114 10287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 20:03:26.544337 10287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 20:03:26.544574 10287 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 20:03:26.544739 10287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 20:03:26.703656 10287 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.65:46161
I20250625 20:03:26.703780 10415 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.65:46161 every 8 connection(s)
I20250625 20:03:26.706358 10287 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250625 20:03:26.715279 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 10287
I20250625 20:03:26.715756 9337 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250625 20:03:26.723436 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.30.66:0
--local_ip_for_outbound_sockets=127.9.30.66
--webserver_interface=127.9.30.66
--webserver_port=0
--tserver_master_addrs=127.9.30.126:36001
--builtin_ntp_servers=127.9.30.84:38045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250625 20:03:26.731729 10416 heartbeater.cc:344] Connected to a master server at 127.9.30.126:36001
I20250625 20:03:26.732281 10416 heartbeater.cc:461] Registering TS with master...
I20250625 20:03:26.733626 10416 heartbeater.cc:507] Master 127.9.30.126:36001 requested a full tablet report, sending...
I20250625 20:03:26.736963 10228 ts_manager.cc:194] Registered new tserver with Master: 72c05ce85e7342ed889e1ecc2678ece5 (127.9.30.65:46161)
I20250625 20:03:26.739806 10228 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.30.65:36741
W20250625 20:03:27.038511 10420 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250625 20:03:27.039160 10420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:03:27.039460 10420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:03:27.040046 10420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:03:27.071393 10420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:03:27.072305 10420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.30.66
I20250625 20:03:27.106789 10420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:38045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.30.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.9.30.66
--webserver_port=0
--tserver_master_addrs=127.9.30.126:36001
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.30.66
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:03:27.108171 10420 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:03:27.109918 10420 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:03:27.125972 10426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:03:27.744377 10416 heartbeater.cc:499] Master 127.9.30.126:36001 was elected leader, sending a full tablet report...
W20250625 20:03:27.125993 10427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:03:27.127005 10429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:03:27.128922 10420 server_base.cc:1048] running on GCE node
I20250625 20:03:28.204737 10420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:03:28.207345 10420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:03:28.208815 10420 hybrid_clock.cc:648] HybridClock initialized: now 1750881808208771 us; error 92 us; skew 500 ppm
I20250625 20:03:28.209609 10420 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:03:28.216161 10420 webserver.cc:469] Webserver started at http://127.9.30.66:39391/ using document root <none> and password file <none>
I20250625 20:03:28.217074 10420 fs_manager.cc:362] Metadata directory not provided
I20250625 20:03:28.217267 10420 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:03:28.217694 10420 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 20:03:28.222169 10420 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "d36847f6f8cd48bd9f5128ab2a3003bc"
format_stamp: "Formatted at 2025-06-25 20:03:28 on dist-test-slave-jhxp"
I20250625 20:03:28.223261 10420 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "d36847f6f8cd48bd9f5128ab2a3003bc"
format_stamp: "Formatted at 2025-06-25 20:03:28 on dist-test-slave-jhxp"
I20250625 20:03:28.230495 10420 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20250625 20:03:28.236259 10436 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:03:28.237366 10420 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250625 20:03:28.237680 10420 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d36847f6f8cd48bd9f5128ab2a3003bc"
format_stamp: "Formatted at 2025-06-25 20:03:28 on dist-test-slave-jhxp"
I20250625 20:03:28.237996 10420 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:03:28.291386 10420 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:03:28.293443 10420 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:03:28.294037 10420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:03:28.297209 10420 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 20:03:28.301412 10420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 20:03:28.301630 10420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 20:03:28.301898 10420 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 20:03:28.302059 10420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 20:03:28.432240 10420 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.66:33663
I20250625 20:03:28.432340 10548 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.66:33663 every 8 connection(s)
I20250625 20:03:28.434908 10420 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250625 20:03:28.436177 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 10420
I20250625 20:03:28.436666 9337 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250625 20:03:28.444811 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.30.67:0
--local_ip_for_outbound_sockets=127.9.30.67
--webserver_interface=127.9.30.67
--webserver_port=0
--tserver_master_addrs=127.9.30.126:36001
--builtin_ntp_servers=127.9.30.84:38045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250625 20:03:28.457132 10549 heartbeater.cc:344] Connected to a master server at 127.9.30.126:36001
I20250625 20:03:28.457563 10549 heartbeater.cc:461] Registering TS with master...
I20250625 20:03:28.458590 10549 heartbeater.cc:507] Master 127.9.30.126:36001 requested a full tablet report, sending...
I20250625 20:03:28.460950 10228 ts_manager.cc:194] Registered new tserver with Master: d36847f6f8cd48bd9f5128ab2a3003bc (127.9.30.66:33663)
I20250625 20:03:28.462761 10228 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.30.66:60443
W20250625 20:03:28.506726 10259 debug-util.cc:398] Leaking SignalData structure 0x7b08000a22e0 after lost signal to thread 10195
W20250625 20:03:28.767488 10553 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250625 20:03:28.768170 10553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:03:28.768428 10553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:03:28.768918 10553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:03:28.800387 10553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:03:28.801260 10553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.30.67
I20250625 20:03:28.837262 10553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:38045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.30.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.9.30.67
--webserver_port=0
--tserver_master_addrs=127.9.30.126:36001
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.30.67
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:03:28.838552 10553 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:03:28.840390 10553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:03:28.858292 10559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:03:29.466310 10549 heartbeater.cc:499] Master 127.9.30.126:36001 was elected leader, sending a full tablet report...
W20250625 20:03:28.859977 10562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:03:28.859148 10560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:03:30.255121 10561 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1395 milliseconds
I20250625 20:03:30.255254 10553 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 20:03:30.256693 10553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:03:30.259200 10553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:03:30.260661 10553 hybrid_clock.cc:648] HybridClock initialized: now 1750881810260587 us; error 89 us; skew 500 ppm
I20250625 20:03:30.261451 10553 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:03:30.268414 10553 webserver.cc:469] Webserver started at http://127.9.30.67:45555/ using document root <none> and password file <none>
I20250625 20:03:30.269748 10553 fs_manager.cc:362] Metadata directory not provided
I20250625 20:03:30.270047 10553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:03:30.270625 10553 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 20:03:30.277159 10553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "9099b8230df648b4989530bb42eef6d5"
format_stamp: "Formatted at 2025-06-25 20:03:30 on dist-test-slave-jhxp"
I20250625 20:03:30.278714 10553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "9099b8230df648b4989530bb42eef6d5"
format_stamp: "Formatted at 2025-06-25 20:03:30 on dist-test-slave-jhxp"
I20250625 20:03:30.288161 10553 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.011s sys 0.001s
I20250625 20:03:30.296016 10569 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:03:30.297886 10553 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20250625 20:03:30.298309 10553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9099b8230df648b4989530bb42eef6d5"
format_stamp: "Formatted at 2025-06-25 20:03:30 on dist-test-slave-jhxp"
I20250625 20:03:30.298733 10553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:03:30.352880 10553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:03:30.354328 10553 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:03:30.354774 10553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:03:30.357476 10553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 20:03:30.361984 10553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 20:03:30.362198 10553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 20:03:30.362430 10553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 20:03:30.362589 10553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 20:03:30.522513 10553 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.67:33899
I20250625 20:03:30.522614 10681 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.67:33899 every 8 connection(s)
I20250625 20:03:30.525250 10553 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250625 20:03:30.535311 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 10553
I20250625 20:03:30.535804 9337 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250625 20:03:30.547914 10682 heartbeater.cc:344] Connected to a master server at 127.9.30.126:36001
I20250625 20:03:30.548434 10682 heartbeater.cc:461] Registering TS with master...
I20250625 20:03:30.549708 10682 heartbeater.cc:507] Master 127.9.30.126:36001 requested a full tablet report, sending...
I20250625 20:03:30.552275 10228 ts_manager.cc:194] Registered new tserver with Master: 9099b8230df648b4989530bb42eef6d5 (127.9.30.67:33899)
I20250625 20:03:30.553653 10228 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.30.67:44851
I20250625 20:03:30.556880 9337 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 20:03:30.593567 10228 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54638:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250625 20:03:30.612485 10228 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 20:03:30.657478 10351 tablet_service.cc:1468] Processing CreateTablet for tablet 13bdb98bc62e4d3ab274d457ae7aa374 (DEFAULT_TABLE table=TestTable [id=86c2d55d486941ac8752bd1465f5dfba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:03:30.659677 10351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13bdb98bc62e4d3ab274d457ae7aa374. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:03:30.676360 10484 tablet_service.cc:1468] Processing CreateTablet for tablet 13bdb98bc62e4d3ab274d457ae7aa374 (DEFAULT_TABLE table=TestTable [id=86c2d55d486941ac8752bd1465f5dfba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:03:30.676360 10617 tablet_service.cc:1468] Processing CreateTablet for tablet 13bdb98bc62e4d3ab274d457ae7aa374 (DEFAULT_TABLE table=TestTable [id=86c2d55d486941ac8752bd1465f5dfba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:03:30.677891 10484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13bdb98bc62e4d3ab274d457ae7aa374. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:03:30.678201 10617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13bdb98bc62e4d3ab274d457ae7aa374. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:03:30.691817 10701 tablet_bootstrap.cc:492] T 13bdb98bc62e4d3ab274d457ae7aa374 P 72c05ce85e7342ed889e1ecc2678ece5: Bootstrap starting.
I20250625 20:03:30.699035 10702 tablet_bootstrap.cc:492] T 13bdb98bc62e4d3ab274d457ae7aa374 P d36847f6f8cd48bd9f5128ab2a3003bc: Bootstrap starting.
I20250625 20:03:30.699422 10701 tablet_bootstrap.cc:654] T 13bdb98bc62e4d3ab274d457ae7aa374 P 72c05ce85e7342ed889e1ecc2678ece5: Neither blocks nor log segments found. Creating new log.
I20250625 20:03:30.706300 10702 tablet_bootstrap.cc:654] T 13bdb98bc62e4d3ab274d457ae7aa374 P d36847f6f8cd48bd9f5128ab2a3003bc: Neither blocks nor log segments found. Creating new log.
I20250625 20:03:30.708665 10702 log.cc:826] T 13bdb98bc62e4d3ab274d457ae7aa374 P d36847f6f8cd48bd9f5128ab2a3003bc: Log is configured to *not* fsync() on all Append() calls
I20250625 20:03:30.708649 10701 log.cc:826] T 13bdb98bc62e4d3ab274d457ae7aa374 P 72c05ce85e7342ed889e1ecc2678ece5: Log is configured to *not* fsync() on all Append() calls
I20250625 20:03:30.710704 10703 tablet_bootstrap.cc:492] T 13bdb98bc62e4d3ab274d457ae7aa374 P 9099b8230df648b4989530bb42eef6d5: Bootstrap starting.
I20250625 20:03:30.719365 10703 tablet_bootstrap.cc:654] T 13bdb98bc62e4d3ab274d457ae7aa374 P 9099b8230df648b4989530bb42eef6d5: Neither blocks nor log segments found. Creating new log.
I20250625 20:03:30.723132 10703 log.cc:826] T 13bdb98bc62e4d3ab274d457ae7aa374 P 9099b8230df648b4989530bb42eef6d5: Log is configured to *not* fsync() on all Append() calls
I20250625 20:03:30.723366 10702 tablet_bootstrap.cc:492] T 13bdb98bc62e4d3ab274d457ae7aa374 P d36847f6f8cd48bd9f5128ab2a3003bc: No bootstrap required, opened a new log
I20250625 20:03:30.723968 10701 tablet_bootstrap.cc:492] T 13bdb98bc62e4d3ab274d457ae7aa374 P 72c05ce85e7342ed889e1ecc2678ece5: No bootstrap required, opened a new log
I20250625 20:03:30.724061 10702 ts_tablet_manager.cc:1397] T 13bdb98bc62e4d3ab274d457ae7aa374 P d36847f6f8cd48bd9f5128ab2a3003bc: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.017s
I20250625 20:03:30.724522 10701 ts_tablet_manager.cc:1397] T 13bdb98bc62e4d3ab274d457ae7aa374 P 72c05ce85e7342ed889e1ecc2678ece5: Time spent bootstrapping tablet: real 0.035s user 0.022s sys 0.003s
I20250625 20:03:30.734496 10703 tablet_bootstrap.cc:492] T 13bdb98bc62e4d3ab274d457ae7aa374 P 9099b8230df648b4989530bb42eef6d5: No bootstrap required, opened a new log
I20250625 20:03:30.735183 10703 ts_tablet_manager.cc:1397] T 13bdb98bc62e4d3ab274d457ae7aa374 P 9099b8230df648b4989530bb42eef6d5: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.012s
I20250625 20:03:30.752610 10702 raft_consensus.cc:357] T 13bdb98bc62e4d3ab274d457ae7aa374 P d36847f6f8cd48bd9f5128ab2a3003bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c05ce85e7342ed889e1ecc2678ece5" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 46161 } } peers { permanent_uuid: "d36847f6f8cd48bd9f5128ab2a3003bc" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 33663 } } peers { permanent_uuid: "9099b8230df648b4989530bb42eef6d5" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33899 } }
I20250625 20:03:30.753588 10702 raft_consensus.cc:738] T 13bdb98bc62e4d3ab274d457ae7aa374 P d36847f6f8cd48bd9f5128ab2a3003bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d36847f6f8cd48bd9f5128ab2a3003bc, State: Initialized, Role: FOLLOWER
I20250625 20:03:30.754338 10702 consensus_queue.cc:260] T 13bdb98bc62e4d3ab274d457ae7aa374 P d36847f6f8cd48bd9f5128ab2a3003bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c05ce85e7342ed889e1ecc2678ece5" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 46161 } } peers { permanent_uuid: "d36847f6f8cd48bd9f5128ab2a3003bc" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 33663 } } peers { permanent_uuid: "9099b8230df648b4989530bb42eef6d5" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33899 } }
I20250625 20:03:30.754611 10701 raft_consensus.cc:357] T 13bdb98bc62e4d3ab274d457ae7aa374 P 72c05ce85e7342ed889e1ecc2678ece5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c05ce85e7342ed889e1ecc2678ece5" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 46161 } } peers { permanent_uuid: "d36847f6f8cd48bd9f5128ab2a3003bc" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 33663 } } peers { permanent_uuid: "9099b8230df648b4989530bb42eef6d5" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33899 } }
I20250625 20:03:30.755198 10703 raft_consensus.cc:357] T 13bdb98bc62e4d3ab274d457ae7aa374 P 9099b8230df648b4989530bb42eef6d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c05ce85e7342ed889e1ecc2678ece5" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 46161 } } peers { permanent_uuid: "d36847f6f8cd48bd9f5128ab2a3003bc" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 33663 } } peers { permanent_uuid: "9099b8230df648b4989530bb42eef6d5" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33899 } }
I20250625 20:03:30.755672 10701 raft_consensus.cc:738] T 13bdb98bc62e4d3ab274d457ae7aa374 P 72c05ce85e7342ed889e1ecc2678ece5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72c05ce85e7342ed889e1ecc2678ece5, State: Initialized, Role: FOLLOWER
I20250625 20:03:30.755964 10703 raft_consensus.cc:738] T 13bdb98bc62e4d3ab274d457ae7aa374 P 9099b8230df648b4989530bb42eef6d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9099b8230df648b4989530bb42eef6d5, State: Initialized, Role: FOLLOWER
I20250625 20:03:30.756742 10703 consensus_queue.cc:260] T 13bdb98bc62e4d3ab274d457ae7aa374 P 9099b8230df648b4989530bb42eef6d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c05ce85e7342ed889e1ecc2678ece5" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 46161 } } peers { permanent_uuid: "d36847f6f8cd48bd9f5128ab2a3003bc" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 33663 } } peers { permanent_uuid: "9099b8230df648b4989530bb42eef6d5" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33899 } }
I20250625 20:03:30.758836 10702 ts_tablet_manager.cc:1428] T 13bdb98bc62e4d3ab274d457ae7aa374 P d36847f6f8cd48bd9f5128ab2a3003bc: Time spent starting tablet: real 0.034s user 0.027s sys 0.005s
I20250625 20:03:30.758803 10701 consensus_queue.cc:260] T 13bdb98bc62e4d3ab274d457ae7aa374 P 72c05ce85e7342ed889e1ecc2678ece5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c05ce85e7342ed889e1ecc2678ece5" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 46161 } } peers { permanent_uuid: "d36847f6f8cd48bd9f5128ab2a3003bc" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 33663 } } peers { permanent_uuid: "9099b8230df648b4989530bb42eef6d5" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33899 } }
I20250625 20:03:30.763324 10701 ts_tablet_manager.cc:1428] T 13bdb98bc62e4d3ab274d457ae7aa374 P 72c05ce85e7342ed889e1ecc2678ece5: Time spent starting tablet: real 0.038s user 0.028s sys 0.004s
I20250625 20:03:30.765569 10682 heartbeater.cc:499] Master 127.9.30.126:36001 was elected leader, sending a full tablet report...
I20250625 20:03:30.766566 10703 ts_tablet_manager.cc:1428] T 13bdb98bc62e4d3ab274d457ae7aa374 P 9099b8230df648b4989530bb42eef6d5: Time spent starting tablet: real 0.031s user 0.027s sys 0.000s
W20250625 20:03:30.779922 10683 tablet.cc:2378] T 13bdb98bc62e4d3ab274d457ae7aa374 P 9099b8230df648b4989530bb42eef6d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 20:03:30.803175 9337 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 20:03:30.806509 9337 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 72c05ce85e7342ed889e1ecc2678ece5 to finish bootstrapping
I20250625 20:03:30.819402 9337 ts_itest-base.cc:246] Waiting for 1 tablets on tserver d36847f6f8cd48bd9f5128ab2a3003bc to finish bootstrapping
I20250625 20:03:30.829990 9337 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 9099b8230df648b4989530bb42eef6d5 to finish bootstrapping
I20250625 20:03:30.872263 10371 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "13bdb98bc62e4d3ab274d457ae7aa374"
dest_uuid: "72c05ce85e7342ed889e1ecc2678ece5"
from {username='slave'} at 127.0.0.1:35518
I20250625 20:03:30.872836 10371 raft_consensus.cc:491] T 13bdb98bc62e4d3ab274d457ae7aa374 P 72c05ce85e7342ed889e1ecc2678ece5 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250625 20:03:30.873152 10371 raft_consensus.cc:3058] T 13bdb98bc62e4d3ab274d457ae7aa374 P 72c05ce85e7342ed889e1ecc2678ece5 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:03:30.877501 10371 raft_consensus.cc:513] T 13bdb98bc62e4d3ab274d457ae7aa374 P 72c05ce85e7342ed889e1ecc2678ece5 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c05ce85e7342ed889e1ecc2678ece5" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 46161 } } peers { permanent_uuid: "d36847f6f8cd48bd9f5128ab2a3003bc" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 33663 } } peers { permanent_uuid: "9099b8230df648b4989530bb42eef6d5" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33899 } }
I20250625 20:03:30.879841 10371 leader_election.cc:290] T 13bdb98bc62e4d3ab274d457ae7aa374 P 72c05ce85e7342ed889e1ecc2678ece5 [CANDIDATE]: Term 1 election: Requested vote from peers d36847f6f8cd48bd9f5128ab2a3003bc (127.9.30.66:33663), 9099b8230df648b4989530bb42eef6d5 (127.9.30.67:33899)
I20250625 20:03:30.889889 9337 cluster_itest_util.cc:257] Not converged past 1 yet: 0.0 0.0 0.0
I20250625 20:03:30.892132 10637 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13bdb98bc62e4d3ab274d457ae7aa374" candidate_uuid: "72c05ce85e7342ed889e1ecc2678ece5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9099b8230df648b4989530bb42eef6d5"
I20250625 20:03:30.892166 10504 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13bdb98bc62e4d3ab274d457ae7aa374" candidate_uuid: "72c05ce85e7342ed889e1ecc2678ece5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d36847f6f8cd48bd9f5128ab2a3003bc"
I20250625 20:03:30.892900 10637 raft_consensus.cc:3058] T 13bdb98bc62e4d3ab274d457ae7aa374 P 9099b8230df648b4989530bb42eef6d5 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:03:30.892901 10504 raft_consensus.cc:3058] T 13bdb98bc62e4d3ab274d457ae7aa374 P d36847f6f8cd48bd9f5128ab2a3003bc [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:03:30.899629 10637 raft_consensus.cc:2466] T 13bdb98bc62e4d3ab274d457ae7aa374 P 9099b8230df648b4989530bb42eef6d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72c05ce85e7342ed889e1ecc2678ece5 in term 1.
I20250625 20:03:30.899631 10504 raft_consensus.cc:2466] T 13bdb98bc62e4d3ab274d457ae7aa374 P d36847f6f8cd48bd9f5128ab2a3003bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72c05ce85e7342ed889e1ecc2678ece5 in term 1.
I20250625 20:03:30.901062 10304 leader_election.cc:304] T 13bdb98bc62e4d3ab274d457ae7aa374 P 72c05ce85e7342ed889e1ecc2678ece5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72c05ce85e7342ed889e1ecc2678ece5, d36847f6f8cd48bd9f5128ab2a3003bc; no voters:
I20250625 20:03:30.901918 10709 raft_consensus.cc:2802] T 13bdb98bc62e4d3ab274d457ae7aa374 P 72c05ce85e7342ed889e1ecc2678ece5 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:03:30.903499 10709 raft_consensus.cc:695] T 13bdb98bc62e4d3ab274d457ae7aa374 P 72c05ce85e7342ed889e1ecc2678ece5 [term 1 LEADER]: Becoming Leader. State: Replica: 72c05ce85e7342ed889e1ecc2678ece5, State: Running, Role: LEADER
I20250625 20:03:30.904338 10709 consensus_queue.cc:237] T 13bdb98bc62e4d3ab274d457ae7aa374 P 72c05ce85e7342ed889e1ecc2678ece5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c05ce85e7342ed889e1ecc2678ece5" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 46161 } } peers { permanent_uuid: "d36847f6f8cd48bd9f5128ab2a3003bc" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 33663 } } peers { permanent_uuid: "9099b8230df648b4989530bb42eef6d5" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33899 } }
I20250625 20:03:30.915148 10227 catalog_manager.cc:5582] T 13bdb98bc62e4d3ab274d457ae7aa374 P 72c05ce85e7342ed889e1ecc2678ece5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 72c05ce85e7342ed889e1ecc2678ece5 (127.9.30.65). New cstate: current_term: 1 leader_uuid: "72c05ce85e7342ed889e1ecc2678ece5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c05ce85e7342ed889e1ecc2678ece5" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 46161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d36847f6f8cd48bd9f5128ab2a3003bc" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 33663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9099b8230df648b4989530bb42eef6d5" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33899 } health_report { overall_health: UNKNOWN } } }
W20250625 20:03:30.942742 10550 tablet.cc:2378] T 13bdb98bc62e4d3ab274d457ae7aa374 P d36847f6f8cd48bd9f5128ab2a3003bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250625 20:03:30.965600 10417 tablet.cc:2378] T 13bdb98bc62e4d3ab274d457ae7aa374 P 72c05ce85e7342ed889e1ecc2678ece5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 20:03:30.995503 9337 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250625 20:03:31.200788 9337 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250625 20:03:31.370034 10709 consensus_queue.cc:1035] T 13bdb98bc62e4d3ab274d457ae7aa374 P 72c05ce85e7342ed889e1ecc2678ece5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d36847f6f8cd48bd9f5128ab2a3003bc" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 33663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:03:31.385810 10709 consensus_queue.cc:1035] T 13bdb98bc62e4d3ab274d457ae7aa374 P 72c05ce85e7342ed889e1ecc2678ece5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9099b8230df648b4989530bb42eef6d5" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250625 20:03:33.088582 10371 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "13bdb98bc62e4d3ab274d457ae7aa374"
dest_uuid: "72c05ce85e7342ed889e1ecc2678ece5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35520
I20250625 20:03:33.089263 10371 raft_consensus.cc:604] T 13bdb98bc62e4d3ab274d457ae7aa374 P 72c05ce85e7342ed889e1ecc2678ece5 [term 1 LEADER]: Received request to transfer leadership
I20250625 20:03:33.187857 10736 raft_consensus.cc:991] T 13bdb98bc62e4d3ab274d457ae7aa374 P 72c05ce85e7342ed889e1ecc2678ece5: : Instructing follower 9099b8230df648b4989530bb42eef6d5 to start an election
I20250625 20:03:33.188328 10748 raft_consensus.cc:1079] T 13bdb98bc62e4d3ab274d457ae7aa374 P 72c05ce85e7342ed889e1ecc2678ece5 [term 1 LEADER]: Signalling peer 9099b8230df648b4989530bb42eef6d5 to start an election
I20250625 20:03:33.189778 10637 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "13bdb98bc62e4d3ab274d457ae7aa374"
dest_uuid: "9099b8230df648b4989530bb42eef6d5"
from {username='slave'} at 127.9.30.65:38151
I20250625 20:03:33.190300 10637 raft_consensus.cc:491] T 13bdb98bc62e4d3ab274d457ae7aa374 P 9099b8230df648b4989530bb42eef6d5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250625 20:03:33.190546 10637 raft_consensus.cc:3058] T 13bdb98bc62e4d3ab274d457ae7aa374 P 9099b8230df648b4989530bb42eef6d5 [term 1 FOLLOWER]: Advancing to term 2
I20250625 20:03:33.194510 10637 raft_consensus.cc:513] T 13bdb98bc62e4d3ab274d457ae7aa374 P 9099b8230df648b4989530bb42eef6d5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c05ce85e7342ed889e1ecc2678ece5" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 46161 } } peers { permanent_uuid: "d36847f6f8cd48bd9f5128ab2a3003bc" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 33663 } } peers { permanent_uuid: "9099b8230df648b4989530bb42eef6d5" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33899 } }
I20250625 20:03:33.196595 10637 leader_election.cc:290] T 13bdb98bc62e4d3ab274d457ae7aa374 P 9099b8230df648b4989530bb42eef6d5 [CANDIDATE]: Term 2 election: Requested vote from peers 72c05ce85e7342ed889e1ecc2678ece5 (127.9.30.65:46161), d36847f6f8cd48bd9f5128ab2a3003bc (127.9.30.66:33663)
I20250625 20:03:33.208447 10371 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13bdb98bc62e4d3ab274d457ae7aa374" candidate_uuid: "9099b8230df648b4989530bb42eef6d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "72c05ce85e7342ed889e1ecc2678ece5"
I20250625 20:03:33.209091 10371 raft_consensus.cc:3053] T 13bdb98bc62e4d3ab274d457ae7aa374 P 72c05ce85e7342ed889e1ecc2678ece5 [term 1 LEADER]: Stepping down as leader of term 1
I20250625 20:03:33.209342 10371 raft_consensus.cc:738] T 13bdb98bc62e4d3ab274d457ae7aa374 P 72c05ce85e7342ed889e1ecc2678ece5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 72c05ce85e7342ed889e1ecc2678ece5, State: Running, Role: LEADER
I20250625 20:03:33.209426 10504 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13bdb98bc62e4d3ab274d457ae7aa374" candidate_uuid: "9099b8230df648b4989530bb42eef6d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d36847f6f8cd48bd9f5128ab2a3003bc"
I20250625 20:03:33.209882 10504 raft_consensus.cc:3058] T 13bdb98bc62e4d3ab274d457ae7aa374 P d36847f6f8cd48bd9f5128ab2a3003bc [term 1 FOLLOWER]: Advancing to term 2
I20250625 20:03:33.209844 10371 consensus_queue.cc:260] T 13bdb98bc62e4d3ab274d457ae7aa374 P 72c05ce85e7342ed889e1ecc2678ece5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c05ce85e7342ed889e1ecc2678ece5" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 46161 } } peers { permanent_uuid: "d36847f6f8cd48bd9f5128ab2a3003bc" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 33663 } } peers { permanent_uuid: "9099b8230df648b4989530bb42eef6d5" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33899 } }
I20250625 20:03:33.210712 10371 raft_consensus.cc:3058] T 13bdb98bc62e4d3ab274d457ae7aa374 P 72c05ce85e7342ed889e1ecc2678ece5 [term 1 FOLLOWER]: Advancing to term 2
I20250625 20:03:33.213868 10504 raft_consensus.cc:2466] T 13bdb98bc62e4d3ab274d457ae7aa374 P d36847f6f8cd48bd9f5128ab2a3003bc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9099b8230df648b4989530bb42eef6d5 in term 2.
I20250625 20:03:33.214668 10371 raft_consensus.cc:2466] T 13bdb98bc62e4d3ab274d457ae7aa374 P 72c05ce85e7342ed889e1ecc2678ece5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9099b8230df648b4989530bb42eef6d5 in term 2.
I20250625 20:03:33.214864 10570 leader_election.cc:304] T 13bdb98bc62e4d3ab274d457ae7aa374 P 9099b8230df648b4989530bb42eef6d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9099b8230df648b4989530bb42eef6d5, d36847f6f8cd48bd9f5128ab2a3003bc; no voters:
I20250625 20:03:33.216631 10752 raft_consensus.cc:2802] T 13bdb98bc62e4d3ab274d457ae7aa374 P 9099b8230df648b4989530bb42eef6d5 [term 2 FOLLOWER]: Leader election won for term 2
I20250625 20:03:33.217833 10752 raft_consensus.cc:695] T 13bdb98bc62e4d3ab274d457ae7aa374 P 9099b8230df648b4989530bb42eef6d5 [term 2 LEADER]: Becoming Leader. State: Replica: 9099b8230df648b4989530bb42eef6d5, State: Running, Role: LEADER
I20250625 20:03:33.218546 10752 consensus_queue.cc:237] T 13bdb98bc62e4d3ab274d457ae7aa374 P 9099b8230df648b4989530bb42eef6d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c05ce85e7342ed889e1ecc2678ece5" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 46161 } } peers { permanent_uuid: "d36847f6f8cd48bd9f5128ab2a3003bc" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 33663 } } peers { permanent_uuid: "9099b8230df648b4989530bb42eef6d5" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33899 } }
I20250625 20:03:33.225689 10226 catalog_manager.cc:5582] T 13bdb98bc62e4d3ab274d457ae7aa374 P 9099b8230df648b4989530bb42eef6d5 reported cstate change: term changed from 1 to 2, leader changed from 72c05ce85e7342ed889e1ecc2678ece5 (127.9.30.65) to 9099b8230df648b4989530bb42eef6d5 (127.9.30.67). New cstate: current_term: 2 leader_uuid: "9099b8230df648b4989530bb42eef6d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72c05ce85e7342ed889e1ecc2678ece5" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 46161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d36847f6f8cd48bd9f5128ab2a3003bc" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 33663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9099b8230df648b4989530bb42eef6d5" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33899 } health_report { overall_health: HEALTHY } } }
I20250625 20:03:33.601176 10504 raft_consensus.cc:1273] T 13bdb98bc62e4d3ab274d457ae7aa374 P d36847f6f8cd48bd9f5128ab2a3003bc [term 2 FOLLOWER]: Refusing update from remote peer 9099b8230df648b4989530bb42eef6d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250625 20:03:33.602447 10752 consensus_queue.cc:1035] T 13bdb98bc62e4d3ab274d457ae7aa374 P 9099b8230df648b4989530bb42eef6d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d36847f6f8cd48bd9f5128ab2a3003bc" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 33663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250625 20:03:33.614673 10371 raft_consensus.cc:1273] T 13bdb98bc62e4d3ab274d457ae7aa374 P 72c05ce85e7342ed889e1ecc2678ece5 [term 2 FOLLOWER]: Refusing update from remote peer 9099b8230df648b4989530bb42eef6d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250625 20:03:33.616469 10752 consensus_queue.cc:1035] T 13bdb98bc62e4d3ab274d457ae7aa374 P 9099b8230df648b4989530bb42eef6d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72c05ce85e7342ed889e1ecc2678ece5" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 46161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250625 20:03:35.695477 10371 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "13bdb98bc62e4d3ab274d457ae7aa374"
dest_uuid: "72c05ce85e7342ed889e1ecc2678ece5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35524
I20250625 20:03:35.696162 10371 raft_consensus.cc:604] T 13bdb98bc62e4d3ab274d457ae7aa374 P 72c05ce85e7342ed889e1ecc2678ece5 [term 2 FOLLOWER]: Received request to transfer leadership
I20250625 20:03:35.696480 10371 raft_consensus.cc:612] T 13bdb98bc62e4d3ab274d457ae7aa374 P 72c05ce85e7342ed889e1ecc2678ece5 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250625 20:03:36.728848 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 10287
I20250625 20:03:36.748818 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 10420
I20250625 20:03:36.769274 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 10553
I20250625 20:03:36.789968 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 10194
2025-06-25T20:03:36Z chronyd exiting
[ OK ] AdminCliTest.TestGracefulSpecificLeaderStepDown (14235 ms)
[ RUN ] AdminCliTest.TestDescribeTableColumnFlags
I20250625 20:03:36.841288 9337 test_util.cc:276] Using random seed: -256078805
I20250625 20:03:36.845506 9337 ts_itest-base.cc:115] Starting cluster with:
I20250625 20:03:36.845703 9337 ts_itest-base.cc:116] --------------
I20250625 20:03:36.845858 9337 ts_itest-base.cc:117] 3 tablet servers
I20250625 20:03:36.845996 9337 ts_itest-base.cc:118] 3 replicas per TS
I20250625 20:03:36.846148 9337 ts_itest-base.cc:119] --------------
2025-06-25T20:03:36Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-25T20:03:36Z Disabled control of system clock
I20250625 20:03:36.880455 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.30.126:44899
--webserver_interface=127.9.30.126
--webserver_port=0
--builtin_ntp_servers=127.9.30.84:44121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.30.126:44899 with env {}
W20250625 20:03:37.176643 10796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:03:37.177232 10796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:03:37.177701 10796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:03:37.209484 10796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 20:03:37.209812 10796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:03:37.210059 10796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 20:03:37.210283 10796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 20:03:37.245741 10796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:44121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.30.126:44899
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.30.126:44899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.9.30.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:03:37.247074 10796 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:03:37.248782 10796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:03:37.264483 10802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:03:37.264483 10803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:03:37.264494 10805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:03:38.356551 10804 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250625 20:03:38.356667 10796 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 20:03:38.360191 10796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:03:38.362689 10796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:03:38.364037 10796 hybrid_clock.cc:648] HybridClock initialized: now 1750881818364010 us; error 43 us; skew 500 ppm
I20250625 20:03:38.364847 10796 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:03:38.375974 10796 webserver.cc:469] Webserver started at http://127.9.30.126:35431/ using document root <none> and password file <none>
I20250625 20:03:38.376955 10796 fs_manager.cc:362] Metadata directory not provided
I20250625 20:03:38.377164 10796 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:03:38.377614 10796 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 20:03:38.382181 10796 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "5210673272d1410a84f086b7d5bf1e7e"
format_stamp: "Formatted at 2025-06-25 20:03:38 on dist-test-slave-jhxp"
I20250625 20:03:38.383394 10796 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "5210673272d1410a84f086b7d5bf1e7e"
format_stamp: "Formatted at 2025-06-25 20:03:38 on dist-test-slave-jhxp"
I20250625 20:03:38.390986 10796 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20250625 20:03:38.396757 10812 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:03:38.397859 10796 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250625 20:03:38.398213 10796 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
uuid: "5210673272d1410a84f086b7d5bf1e7e"
format_stamp: "Formatted at 2025-06-25 20:03:38 on dist-test-slave-jhxp"
I20250625 20:03:38.398602 10796 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:03:38.467650 10796 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:03:38.469218 10796 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:03:38.469717 10796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:03:38.539773 10796 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.126:44899
I20250625 20:03:38.539885 10863 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.126:44899 every 8 connection(s)
I20250625 20:03:38.542517 10796 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250625 20:03:38.547631 10864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:03:38.548987 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 10796
I20250625 20:03:38.549484 9337 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250625 20:03:38.573741 10864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5210673272d1410a84f086b7d5bf1e7e: Bootstrap starting.
I20250625 20:03:38.579288 10864 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5210673272d1410a84f086b7d5bf1e7e: Neither blocks nor log segments found. Creating new log.
I20250625 20:03:38.581029 10864 log.cc:826] T 00000000000000000000000000000000 P 5210673272d1410a84f086b7d5bf1e7e: Log is configured to *not* fsync() on all Append() calls
I20250625 20:03:38.585606 10864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5210673272d1410a84f086b7d5bf1e7e: No bootstrap required, opened a new log
I20250625 20:03:38.603382 10864 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5210673272d1410a84f086b7d5bf1e7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5210673272d1410a84f086b7d5bf1e7e" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 44899 } }
I20250625 20:03:38.604104 10864 raft_consensus.cc:383] T 00000000000000000000000000000000 P 5210673272d1410a84f086b7d5bf1e7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:03:38.604333 10864 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5210673272d1410a84f086b7d5bf1e7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5210673272d1410a84f086b7d5bf1e7e, State: Initialized, Role: FOLLOWER
I20250625 20:03:38.605055 10864 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5210673272d1410a84f086b7d5bf1e7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5210673272d1410a84f086b7d5bf1e7e" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 44899 } }
I20250625 20:03:38.605587 10864 raft_consensus.cc:397] T 00000000000000000000000000000000 P 5210673272d1410a84f086b7d5bf1e7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 20:03:38.605849 10864 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5210673272d1410a84f086b7d5bf1e7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 20:03:38.606127 10864 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5210673272d1410a84f086b7d5bf1e7e [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:03:38.610484 10864 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5210673272d1410a84f086b7d5bf1e7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5210673272d1410a84f086b7d5bf1e7e" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 44899 } }
I20250625 20:03:38.611375 10864 leader_election.cc:304] T 00000000000000000000000000000000 P 5210673272d1410a84f086b7d5bf1e7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5210673272d1410a84f086b7d5bf1e7e; no voters:
I20250625 20:03:38.613008 10864 leader_election.cc:290] T 00000000000000000000000000000000 P 5210673272d1410a84f086b7d5bf1e7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 20:03:38.613770 10869 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5210673272d1410a84f086b7d5bf1e7e [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:03:38.616014 10869 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5210673272d1410a84f086b7d5bf1e7e [term 1 LEADER]: Becoming Leader. State: Replica: 5210673272d1410a84f086b7d5bf1e7e, State: Running, Role: LEADER
I20250625 20:03:38.616978 10864 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5210673272d1410a84f086b7d5bf1e7e [sys.catalog]: configured and running, proceeding with master startup.
I20250625 20:03:38.616791 10869 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5210673272d1410a84f086b7d5bf1e7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5210673272d1410a84f086b7d5bf1e7e" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 44899 } }
I20250625 20:03:38.626964 10871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5210673272d1410a84f086b7d5bf1e7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5210673272d1410a84f086b7d5bf1e7e. Latest consensus state: current_term: 1 leader_uuid: "5210673272d1410a84f086b7d5bf1e7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5210673272d1410a84f086b7d5bf1e7e" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 44899 } } }
I20250625 20:03:38.627964 10871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5210673272d1410a84f086b7d5bf1e7e [sys.catalog]: This master's current role is: LEADER
I20250625 20:03:38.628005 10870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5210673272d1410a84f086b7d5bf1e7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5210673272d1410a84f086b7d5bf1e7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5210673272d1410a84f086b7d5bf1e7e" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 44899 } } }
I20250625 20:03:38.628662 10870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5210673272d1410a84f086b7d5bf1e7e [sys.catalog]: This master's current role is: LEADER
I20250625 20:03:38.631949 10878 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 20:03:38.642277 10878 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 20:03:38.658587 10878 catalog_manager.cc:1349] Generated new cluster ID: a615c99f66dc443a8e6440f648052c28
I20250625 20:03:38.658926 10878 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 20:03:38.687793 10878 catalog_manager.cc:1372] Generated new certificate authority record
I20250625 20:03:38.689328 10878 catalog_manager.cc:1506] Loading token signing keys...
I20250625 20:03:38.708045 10878 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 5210673272d1410a84f086b7d5bf1e7e: Generated new TSK 0
I20250625 20:03:38.709007 10878 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 20:03:38.730921 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.30.65:0
--local_ip_for_outbound_sockets=127.9.30.65
--webserver_interface=127.9.30.65
--webserver_port=0
--tserver_master_addrs=127.9.30.126:44899
--builtin_ntp_servers=127.9.30.84:44121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250625 20:03:39.086805 10888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:03:39.087303 10888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:03:39.087849 10888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:03:39.118986 10888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:03:39.119892 10888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.30.65
I20250625 20:03:39.155099 10888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:44121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.30.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.9.30.65
--webserver_port=0
--tserver_master_addrs=127.9.30.126:44899
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.30.65
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:03:39.156469 10888 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:03:39.158124 10888 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:03:39.174990 10894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:03:39.175096 10895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:03:39.189951 10897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:03:40.280339 10896 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1106 milliseconds
I20250625 20:03:40.280362 10888 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 20:03:40.285274 10888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:03:40.288049 10888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:03:40.289580 10888 hybrid_clock.cc:648] HybridClock initialized: now 1750881820289530 us; error 52 us; skew 500 ppm
I20250625 20:03:40.290681 10888 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:03:40.298375 10888 webserver.cc:469] Webserver started at http://127.9.30.65:40255/ using document root <none> and password file <none>
I20250625 20:03:40.299661 10888 fs_manager.cc:362] Metadata directory not provided
I20250625 20:03:40.299954 10888 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:03:40.300544 10888 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 20:03:40.307094 10888 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "7f82ee574bdf426a894cadbfd3de9e20"
format_stamp: "Formatted at 2025-06-25 20:03:40 on dist-test-slave-jhxp"
I20250625 20:03:40.308696 10888 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "7f82ee574bdf426a894cadbfd3de9e20"
format_stamp: "Formatted at 2025-06-25 20:03:40 on dist-test-slave-jhxp"
I20250625 20:03:40.317831 10888 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.005s sys 0.004s
I20250625 20:03:40.325098 10904 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:03:40.326323 10888 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20250625 20:03:40.326704 10888 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "7f82ee574bdf426a894cadbfd3de9e20"
format_stamp: "Formatted at 2025-06-25 20:03:40 on dist-test-slave-jhxp"
I20250625 20:03:40.327147 10888 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:03:40.391960 10888 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:03:40.393890 10888 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:03:40.394436 10888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:03:40.397609 10888 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 20:03:40.403131 10888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 20:03:40.403420 10888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 20:03:40.403746 10888 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 20:03:40.403975 10888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 20:03:40.553634 10888 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.65:39231
I20250625 20:03:40.553690 11016 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.65:39231 every 8 connection(s)
I20250625 20:03:40.557133 10888 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250625 20:03:40.558454 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 10888
I20250625 20:03:40.558892 9337 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250625 20:03:40.568178 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.30.66:0
--local_ip_for_outbound_sockets=127.9.30.66
--webserver_interface=127.9.30.66
--webserver_port=0
--tserver_master_addrs=127.9.30.126:44899
--builtin_ntp_servers=127.9.30.84:44121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 20:03:40.584602 11017 heartbeater.cc:344] Connected to a master server at 127.9.30.126:44899
I20250625 20:03:40.585041 11017 heartbeater.cc:461] Registering TS with master...
I20250625 20:03:40.586077 11017 heartbeater.cc:507] Master 127.9.30.126:44899 requested a full tablet report, sending...
I20250625 20:03:40.588634 10829 ts_manager.cc:194] Registered new tserver with Master: 7f82ee574bdf426a894cadbfd3de9e20 (127.9.30.65:39231)
I20250625 20:03:40.590562 10829 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.30.65:39001
W20250625 20:03:40.863566 11021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:03:40.864123 11021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:03:40.864634 11021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:03:40.896453 11021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:03:40.897316 11021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.30.66
I20250625 20:03:40.932296 11021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:44121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.30.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.9.30.66
--webserver_port=0
--tserver_master_addrs=127.9.30.126:44899
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.30.66
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:03:40.933611 11021 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:03:40.935251 11021 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:03:40.950311 11028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:03:41.594070 11017 heartbeater.cc:499] Master 127.9.30.126:44899 was elected leader, sending a full tablet report...
W20250625 20:03:40.950314 11027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:03:40.952605 11021 server_base.cc:1048] running on GCE node
W20250625 20:03:40.952695 11030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:03:42.034478 11021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:03:42.036969 11021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:03:42.038359 11021 hybrid_clock.cc:648] HybridClock initialized: now 1750881822038319 us; error 62 us; skew 500 ppm
I20250625 20:03:42.039147 11021 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:03:42.045406 11021 webserver.cc:469] Webserver started at http://127.9.30.66:38009/ using document root <none> and password file <none>
I20250625 20:03:42.046391 11021 fs_manager.cc:362] Metadata directory not provided
I20250625 20:03:42.046598 11021 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:03:42.047021 11021 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 20:03:42.051636 11021 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "6b2899541b6e4e969255d6eade08f1b3"
format_stamp: "Formatted at 2025-06-25 20:03:42 on dist-test-slave-jhxp"
I20250625 20:03:42.052793 11021 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "6b2899541b6e4e969255d6eade08f1b3"
format_stamp: "Formatted at 2025-06-25 20:03:42 on dist-test-slave-jhxp"
I20250625 20:03:42.060079 11021 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250625 20:03:42.065856 11037 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:03:42.067004 11021 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20250625 20:03:42.067309 11021 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "6b2899541b6e4e969255d6eade08f1b3"
format_stamp: "Formatted at 2025-06-25 20:03:42 on dist-test-slave-jhxp"
I20250625 20:03:42.067654 11021 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:03:42.125902 11021 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:03:42.127490 11021 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:03:42.128031 11021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:03:42.130519 11021 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 20:03:42.134714 11021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 20:03:42.134940 11021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 20:03:42.135142 11021 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 20:03:42.135273 11021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 20:03:42.265347 11021 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.66:43729
I20250625 20:03:42.265483 11149 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.66:43729 every 8 connection(s)
I20250625 20:03:42.267964 11021 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250625 20:03:42.271252 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 11021
I20250625 20:03:42.271791 9337 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250625 20:03:42.278544 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.30.67:0
--local_ip_for_outbound_sockets=127.9.30.67
--webserver_interface=127.9.30.67
--webserver_port=0
--tserver_master_addrs=127.9.30.126:44899
--builtin_ntp_servers=127.9.30.84:44121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 20:03:42.289889 11150 heartbeater.cc:344] Connected to a master server at 127.9.30.126:44899
I20250625 20:03:42.290314 11150 heartbeater.cc:461] Registering TS with master...
I20250625 20:03:42.291323 11150 heartbeater.cc:507] Master 127.9.30.126:44899 requested a full tablet report, sending...
I20250625 20:03:42.293437 10829 ts_manager.cc:194] Registered new tserver with Master: 6b2899541b6e4e969255d6eade08f1b3 (127.9.30.66:43729)
I20250625 20:03:42.294648 10829 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.30.66:36423
W20250625 20:03:42.585633 11154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:03:42.586133 11154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:03:42.586637 11154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:03:42.618468 11154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:03:42.619328 11154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.30.67
I20250625 20:03:42.655398 11154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:44121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.30.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.9.30.67
--webserver_port=0
--tserver_master_addrs=127.9.30.126:44899
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.30.67
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:03:42.656806 11154 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:03:42.658543 11154 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:03:42.674015 11160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:03:43.297724 11150 heartbeater.cc:499] Master 127.9.30.126:44899 was elected leader, sending a full tablet report...
W20250625 20:03:42.674065 11161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:03:42.675843 11163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:03:43.759708 11162 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250625 20:03:43.759768 11154 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 20:03:43.763386 11154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:03:43.766062 11154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:03:43.767480 11154 hybrid_clock.cc:648] HybridClock initialized: now 1750881823767429 us; error 68 us; skew 500 ppm
I20250625 20:03:43.768352 11154 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:03:43.779392 11154 webserver.cc:469] Webserver started at http://127.9.30.67:42099/ using document root <none> and password file <none>
I20250625 20:03:43.780388 11154 fs_manager.cc:362] Metadata directory not provided
I20250625 20:03:43.780615 11154 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:03:43.781092 11154 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 20:03:43.785661 11154 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "20559cb549264b999ba8d5c6c61770da"
format_stamp: "Formatted at 2025-06-25 20:03:43 on dist-test-slave-jhxp"
I20250625 20:03:43.786957 11154 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "20559cb549264b999ba8d5c6c61770da"
format_stamp: "Formatted at 2025-06-25 20:03:43 on dist-test-slave-jhxp"
I20250625 20:03:43.794337 11154 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.001s sys 0.004s
I20250625 20:03:43.800315 11170 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:03:43.801440 11154 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250625 20:03:43.801764 11154 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "20559cb549264b999ba8d5c6c61770da"
format_stamp: "Formatted at 2025-06-25 20:03:43 on dist-test-slave-jhxp"
I20250625 20:03:43.802078 11154 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:03:43.861227 11154 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:03:43.862668 11154 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:03:43.863075 11154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:03:43.865582 11154 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 20:03:43.869706 11154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 20:03:43.869920 11154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 20:03:43.870177 11154 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 20:03:43.870327 11154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 20:03:43.998160 11154 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.67:46723
I20250625 20:03:43.998267 11282 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.67:46723 every 8 connection(s)
I20250625 20:03:44.000777 11154 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250625 20:03:44.009552 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 11154
I20250625 20:03:44.010068 9337 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250625 20:03:44.022584 11283 heartbeater.cc:344] Connected to a master server at 127.9.30.126:44899
I20250625 20:03:44.023041 11283 heartbeater.cc:461] Registering TS with master...
I20250625 20:03:44.024103 11283 heartbeater.cc:507] Master 127.9.30.126:44899 requested a full tablet report, sending...
I20250625 20:03:44.026191 10829 ts_manager.cc:194] Registered new tserver with Master: 20559cb549264b999ba8d5c6c61770da (127.9.30.67:46723)
I20250625 20:03:44.027402 10829 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.30.67:34429
I20250625 20:03:44.030911 9337 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 20:03:44.062803 10828 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60640:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250625 20:03:44.081812 10828 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 20:03:44.132892 11085 tablet_service.cc:1468] Processing CreateTablet for tablet 320fe425de1d4ce195c7931e36dd4c58 (DEFAULT_TABLE table=TestTable [id=7d9934579e924b2b88a3a6ac78cbb757]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:03:44.133038 10952 tablet_service.cc:1468] Processing CreateTablet for tablet 320fe425de1d4ce195c7931e36dd4c58 (DEFAULT_TABLE table=TestTable [id=7d9934579e924b2b88a3a6ac78cbb757]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:03:44.134910 10952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 320fe425de1d4ce195c7931e36dd4c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:03:44.135838 11085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 320fe425de1d4ce195c7931e36dd4c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:03:44.141404 11218 tablet_service.cc:1468] Processing CreateTablet for tablet 320fe425de1d4ce195c7931e36dd4c58 (DEFAULT_TABLE table=TestTable [id=7d9934579e924b2b88a3a6ac78cbb757]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:03:44.143694 11218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 320fe425de1d4ce195c7931e36dd4c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:03:44.160928 11302 tablet_bootstrap.cc:492] T 320fe425de1d4ce195c7931e36dd4c58 P 6b2899541b6e4e969255d6eade08f1b3: Bootstrap starting.
I20250625 20:03:44.168318 11303 tablet_bootstrap.cc:492] T 320fe425de1d4ce195c7931e36dd4c58 P 7f82ee574bdf426a894cadbfd3de9e20: Bootstrap starting.
I20250625 20:03:44.168677 11302 tablet_bootstrap.cc:654] T 320fe425de1d4ce195c7931e36dd4c58 P 6b2899541b6e4e969255d6eade08f1b3: Neither blocks nor log segments found. Creating new log.
I20250625 20:03:44.171305 11302 log.cc:826] T 320fe425de1d4ce195c7931e36dd4c58 P 6b2899541b6e4e969255d6eade08f1b3: Log is configured to *not* fsync() on all Append() calls
I20250625 20:03:44.172075 11304 tablet_bootstrap.cc:492] T 320fe425de1d4ce195c7931e36dd4c58 P 20559cb549264b999ba8d5c6c61770da: Bootstrap starting.
I20250625 20:03:44.175809 11303 tablet_bootstrap.cc:654] T 320fe425de1d4ce195c7931e36dd4c58 P 7f82ee574bdf426a894cadbfd3de9e20: Neither blocks nor log segments found. Creating new log.
I20250625 20:03:44.177742 11303 log.cc:826] T 320fe425de1d4ce195c7931e36dd4c58 P 7f82ee574bdf426a894cadbfd3de9e20: Log is configured to *not* fsync() on all Append() calls
I20250625 20:03:44.178267 11302 tablet_bootstrap.cc:492] T 320fe425de1d4ce195c7931e36dd4c58 P 6b2899541b6e4e969255d6eade08f1b3: No bootstrap required, opened a new log
I20250625 20:03:44.178797 11302 ts_tablet_manager.cc:1397] T 320fe425de1d4ce195c7931e36dd4c58 P 6b2899541b6e4e969255d6eade08f1b3: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.005s
I20250625 20:03:44.179170 11304 tablet_bootstrap.cc:654] T 320fe425de1d4ce195c7931e36dd4c58 P 20559cb549264b999ba8d5c6c61770da: Neither blocks nor log segments found. Creating new log.
I20250625 20:03:44.181653 11304 log.cc:826] T 320fe425de1d4ce195c7931e36dd4c58 P 20559cb549264b999ba8d5c6c61770da: Log is configured to *not* fsync() on all Append() calls
I20250625 20:03:44.182480 11303 tablet_bootstrap.cc:492] T 320fe425de1d4ce195c7931e36dd4c58 P 7f82ee574bdf426a894cadbfd3de9e20: No bootstrap required, opened a new log
I20250625 20:03:44.182914 11303 ts_tablet_manager.cc:1397] T 320fe425de1d4ce195c7931e36dd4c58 P 7f82ee574bdf426a894cadbfd3de9e20: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.003s
I20250625 20:03:44.190696 11304 tablet_bootstrap.cc:492] T 320fe425de1d4ce195c7931e36dd4c58 P 20559cb549264b999ba8d5c6c61770da: No bootstrap required, opened a new log
I20250625 20:03:44.191272 11304 ts_tablet_manager.cc:1397] T 320fe425de1d4ce195c7931e36dd4c58 P 20559cb549264b999ba8d5c6c61770da: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.016s
I20250625 20:03:44.203277 11303 raft_consensus.cc:357] T 320fe425de1d4ce195c7931e36dd4c58 P 7f82ee574bdf426a894cadbfd3de9e20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b2899541b6e4e969255d6eade08f1b3" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 43729 } } peers { permanent_uuid: "7f82ee574bdf426a894cadbfd3de9e20" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 39231 } } peers { permanent_uuid: "20559cb549264b999ba8d5c6c61770da" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 46723 } }
I20250625 20:03:44.204041 11303 raft_consensus.cc:383] T 320fe425de1d4ce195c7931e36dd4c58 P 7f82ee574bdf426a894cadbfd3de9e20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:03:44.204298 11303 raft_consensus.cc:738] T 320fe425de1d4ce195c7931e36dd4c58 P 7f82ee574bdf426a894cadbfd3de9e20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f82ee574bdf426a894cadbfd3de9e20, State: Initialized, Role: FOLLOWER
I20250625 20:03:44.205049 11303 consensus_queue.cc:260] T 320fe425de1d4ce195c7931e36dd4c58 P 7f82ee574bdf426a894cadbfd3de9e20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b2899541b6e4e969255d6eade08f1b3" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 43729 } } peers { permanent_uuid: "7f82ee574bdf426a894cadbfd3de9e20" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 39231 } } peers { permanent_uuid: "20559cb549264b999ba8d5c6c61770da" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 46723 } }
I20250625 20:03:44.205714 11302 raft_consensus.cc:357] T 320fe425de1d4ce195c7931e36dd4c58 P 6b2899541b6e4e969255d6eade08f1b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b2899541b6e4e969255d6eade08f1b3" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 43729 } } peers { permanent_uuid: "7f82ee574bdf426a894cadbfd3de9e20" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 39231 } } peers { permanent_uuid: "20559cb549264b999ba8d5c6c61770da" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 46723 } }
I20250625 20:03:44.206707 11302 raft_consensus.cc:383] T 320fe425de1d4ce195c7931e36dd4c58 P 6b2899541b6e4e969255d6eade08f1b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:03:44.207051 11302 raft_consensus.cc:738] T 320fe425de1d4ce195c7931e36dd4c58 P 6b2899541b6e4e969255d6eade08f1b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b2899541b6e4e969255d6eade08f1b3, State: Initialized, Role: FOLLOWER
I20250625 20:03:44.208775 11303 ts_tablet_manager.cc:1428] T 320fe425de1d4ce195c7931e36dd4c58 P 7f82ee574bdf426a894cadbfd3de9e20: Time spent starting tablet: real 0.026s user 0.018s sys 0.009s
I20250625 20:03:44.209254 11302 consensus_queue.cc:260] T 320fe425de1d4ce195c7931e36dd4c58 P 6b2899541b6e4e969255d6eade08f1b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b2899541b6e4e969255d6eade08f1b3" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 43729 } } peers { permanent_uuid: "7f82ee574bdf426a894cadbfd3de9e20" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 39231 } } peers { permanent_uuid: "20559cb549264b999ba8d5c6c61770da" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 46723 } }
I20250625 20:03:44.213052 11302 ts_tablet_manager.cc:1428] T 320fe425de1d4ce195c7931e36dd4c58 P 6b2899541b6e4e969255d6eade08f1b3: Time spent starting tablet: real 0.034s user 0.022s sys 0.009s
I20250625 20:03:44.218819 11304 raft_consensus.cc:357] T 320fe425de1d4ce195c7931e36dd4c58 P 20559cb549264b999ba8d5c6c61770da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b2899541b6e4e969255d6eade08f1b3" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 43729 } } peers { permanent_uuid: "7f82ee574bdf426a894cadbfd3de9e20" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 39231 } } peers { permanent_uuid: "20559cb549264b999ba8d5c6c61770da" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 46723 } }
I20250625 20:03:44.220964 11304 raft_consensus.cc:383] T 320fe425de1d4ce195c7931e36dd4c58 P 20559cb549264b999ba8d5c6c61770da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:03:44.221341 11304 raft_consensus.cc:738] T 320fe425de1d4ce195c7931e36dd4c58 P 20559cb549264b999ba8d5c6c61770da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20559cb549264b999ba8d5c6c61770da, State: Initialized, Role: FOLLOWER
I20250625 20:03:44.222184 11304 consensus_queue.cc:260] T 320fe425de1d4ce195c7931e36dd4c58 P 20559cb549264b999ba8d5c6c61770da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b2899541b6e4e969255d6eade08f1b3" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 43729 } } peers { permanent_uuid: "7f82ee574bdf426a894cadbfd3de9e20" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 39231 } } peers { permanent_uuid: "20559cb549264b999ba8d5c6c61770da" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 46723 } }
I20250625 20:03:44.224889 11283 heartbeater.cc:499] Master 127.9.30.126:44899 was elected leader, sending a full tablet report...
I20250625 20:03:44.226198 11304 ts_tablet_manager.cc:1428] T 320fe425de1d4ce195c7931e36dd4c58 P 20559cb549264b999ba8d5c6c61770da: Time spent starting tablet: real 0.035s user 0.026s sys 0.007s
W20250625 20:03:44.255143 11284 tablet.cc:2378] T 320fe425de1d4ce195c7931e36dd4c58 P 20559cb549264b999ba8d5c6c61770da: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250625 20:03:44.274188 11151 tablet.cc:2378] T 320fe425de1d4ce195c7931e36dd4c58 P 6b2899541b6e4e969255d6eade08f1b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250625 20:03:44.319353 11018 tablet.cc:2378] T 320fe425de1d4ce195c7931e36dd4c58 P 7f82ee574bdf426a894cadbfd3de9e20: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 20:03:44.374171 11308 raft_consensus.cc:491] T 320fe425de1d4ce195c7931e36dd4c58 P 7f82ee574bdf426a894cadbfd3de9e20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:03:44.374665 11308 raft_consensus.cc:513] T 320fe425de1d4ce195c7931e36dd4c58 P 7f82ee574bdf426a894cadbfd3de9e20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b2899541b6e4e969255d6eade08f1b3" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 43729 } } peers { permanent_uuid: "7f82ee574bdf426a894cadbfd3de9e20" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 39231 } } peers { permanent_uuid: "20559cb549264b999ba8d5c6c61770da" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 46723 } }
I20250625 20:03:44.377246 11308 leader_election.cc:290] T 320fe425de1d4ce195c7931e36dd4c58 P 7f82ee574bdf426a894cadbfd3de9e20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b2899541b6e4e969255d6eade08f1b3 (127.9.30.66:43729), 20559cb549264b999ba8d5c6c61770da (127.9.30.67:46723)
I20250625 20:03:44.386816 11309 raft_consensus.cc:491] T 320fe425de1d4ce195c7931e36dd4c58 P 6b2899541b6e4e969255d6eade08f1b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:03:44.387431 11309 raft_consensus.cc:513] T 320fe425de1d4ce195c7931e36dd4c58 P 6b2899541b6e4e969255d6eade08f1b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b2899541b6e4e969255d6eade08f1b3" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 43729 } } peers { permanent_uuid: "7f82ee574bdf426a894cadbfd3de9e20" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 39231 } } peers { permanent_uuid: "20559cb549264b999ba8d5c6c61770da" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 46723 } }
I20250625 20:03:44.389216 11238 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "320fe425de1d4ce195c7931e36dd4c58" candidate_uuid: "7f82ee574bdf426a894cadbfd3de9e20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20559cb549264b999ba8d5c6c61770da" is_pre_election: true
I20250625 20:03:44.390015 11238 raft_consensus.cc:2466] T 320fe425de1d4ce195c7931e36dd4c58 P 20559cb549264b999ba8d5c6c61770da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f82ee574bdf426a894cadbfd3de9e20 in term 0.
I20250625 20:03:44.391212 11105 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "320fe425de1d4ce195c7931e36dd4c58" candidate_uuid: "7f82ee574bdf426a894cadbfd3de9e20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b2899541b6e4e969255d6eade08f1b3" is_pre_election: true
I20250625 20:03:44.391490 10905 leader_election.cc:304] T 320fe425de1d4ce195c7931e36dd4c58 P 7f82ee574bdf426a894cadbfd3de9e20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20559cb549264b999ba8d5c6c61770da, 7f82ee574bdf426a894cadbfd3de9e20; no voters:
I20250625 20:03:44.391853 11309 leader_election.cc:290] T 320fe425de1d4ce195c7931e36dd4c58 P 6b2899541b6e4e969255d6eade08f1b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f82ee574bdf426a894cadbfd3de9e20 (127.9.30.65:39231), 20559cb549264b999ba8d5c6c61770da (127.9.30.67:46723)
I20250625 20:03:44.392027 11105 raft_consensus.cc:2466] T 320fe425de1d4ce195c7931e36dd4c58 P 6b2899541b6e4e969255d6eade08f1b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f82ee574bdf426a894cadbfd3de9e20 in term 0.
I20250625 20:03:44.393033 11308 raft_consensus.cc:2802] T 320fe425de1d4ce195c7931e36dd4c58 P 7f82ee574bdf426a894cadbfd3de9e20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:03:44.393433 11308 raft_consensus.cc:491] T 320fe425de1d4ce195c7931e36dd4c58 P 7f82ee574bdf426a894cadbfd3de9e20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:03:44.393751 11308 raft_consensus.cc:3058] T 320fe425de1d4ce195c7931e36dd4c58 P 7f82ee574bdf426a894cadbfd3de9e20 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:03:44.400135 11308 raft_consensus.cc:513] T 320fe425de1d4ce195c7931e36dd4c58 P 7f82ee574bdf426a894cadbfd3de9e20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b2899541b6e4e969255d6eade08f1b3" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 43729 } } peers { permanent_uuid: "7f82ee574bdf426a894cadbfd3de9e20" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 39231 } } peers { permanent_uuid: "20559cb549264b999ba8d5c6c61770da" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 46723 } }
I20250625 20:03:44.402421 11308 leader_election.cc:290] T 320fe425de1d4ce195c7931e36dd4c58 P 7f82ee574bdf426a894cadbfd3de9e20 [CANDIDATE]: Term 1 election: Requested vote from peers 6b2899541b6e4e969255d6eade08f1b3 (127.9.30.66:43729), 20559cb549264b999ba8d5c6c61770da (127.9.30.67:46723)
I20250625 20:03:44.403390 11105 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "320fe425de1d4ce195c7931e36dd4c58" candidate_uuid: "7f82ee574bdf426a894cadbfd3de9e20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b2899541b6e4e969255d6eade08f1b3"
I20250625 20:03:44.403439 11238 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "320fe425de1d4ce195c7931e36dd4c58" candidate_uuid: "7f82ee574bdf426a894cadbfd3de9e20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20559cb549264b999ba8d5c6c61770da"
I20250625 20:03:44.403981 11105 raft_consensus.cc:3058] T 320fe425de1d4ce195c7931e36dd4c58 P 6b2899541b6e4e969255d6eade08f1b3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:03:44.404026 11238 raft_consensus.cc:3058] T 320fe425de1d4ce195c7931e36dd4c58 P 20559cb549264b999ba8d5c6c61770da [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:03:44.404697 11237 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "320fe425de1d4ce195c7931e36dd4c58" candidate_uuid: "6b2899541b6e4e969255d6eade08f1b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20559cb549264b999ba8d5c6c61770da" is_pre_election: true
I20250625 20:03:44.405442 10972 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "320fe425de1d4ce195c7931e36dd4c58" candidate_uuid: "6b2899541b6e4e969255d6eade08f1b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f82ee574bdf426a894cadbfd3de9e20" is_pre_election: true
I20250625 20:03:44.406411 10972 raft_consensus.cc:2391] T 320fe425de1d4ce195c7931e36dd4c58 P 7f82ee574bdf426a894cadbfd3de9e20 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b2899541b6e4e969255d6eade08f1b3 in current term 1: Already voted for candidate 7f82ee574bdf426a894cadbfd3de9e20 in this term.
I20250625 20:03:44.407742 11041 leader_election.cc:304] T 320fe425de1d4ce195c7931e36dd4c58 P 6b2899541b6e4e969255d6eade08f1b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b2899541b6e4e969255d6eade08f1b3; no voters: 20559cb549264b999ba8d5c6c61770da, 7f82ee574bdf426a894cadbfd3de9e20
I20250625 20:03:44.409695 11238 raft_consensus.cc:2466] T 320fe425de1d4ce195c7931e36dd4c58 P 20559cb549264b999ba8d5c6c61770da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f82ee574bdf426a894cadbfd3de9e20 in term 1.
I20250625 20:03:44.410029 11105 raft_consensus.cc:2466] T 320fe425de1d4ce195c7931e36dd4c58 P 6b2899541b6e4e969255d6eade08f1b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f82ee574bdf426a894cadbfd3de9e20 in term 1.
I20250625 20:03:44.410401 11309 raft_consensus.cc:2747] T 320fe425de1d4ce195c7931e36dd4c58 P 6b2899541b6e4e969255d6eade08f1b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 20:03:44.410562 10905 leader_election.cc:304] T 320fe425de1d4ce195c7931e36dd4c58 P 7f82ee574bdf426a894cadbfd3de9e20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20559cb549264b999ba8d5c6c61770da, 7f82ee574bdf426a894cadbfd3de9e20; no voters:
I20250625 20:03:44.411201 11308 raft_consensus.cc:2802] T 320fe425de1d4ce195c7931e36dd4c58 P 7f82ee574bdf426a894cadbfd3de9e20 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:03:44.413146 11308 raft_consensus.cc:695] T 320fe425de1d4ce195c7931e36dd4c58 P 7f82ee574bdf426a894cadbfd3de9e20 [term 1 LEADER]: Becoming Leader. State: Replica: 7f82ee574bdf426a894cadbfd3de9e20, State: Running, Role: LEADER
I20250625 20:03:44.413993 11308 consensus_queue.cc:237] T 320fe425de1d4ce195c7931e36dd4c58 P 7f82ee574bdf426a894cadbfd3de9e20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b2899541b6e4e969255d6eade08f1b3" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 43729 } } peers { permanent_uuid: "7f82ee574bdf426a894cadbfd3de9e20" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 39231 } } peers { permanent_uuid: "20559cb549264b999ba8d5c6c61770da" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 46723 } }
I20250625 20:03:44.425374 10827 catalog_manager.cc:5582] T 320fe425de1d4ce195c7931e36dd4c58 P 7f82ee574bdf426a894cadbfd3de9e20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f82ee574bdf426a894cadbfd3de9e20 (127.9.30.65). New cstate: current_term: 1 leader_uuid: "7f82ee574bdf426a894cadbfd3de9e20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b2899541b6e4e969255d6eade08f1b3" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 43729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f82ee574bdf426a894cadbfd3de9e20" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 39231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20559cb549264b999ba8d5c6c61770da" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 46723 } health_report { overall_health: UNKNOWN } } }
I20250625 20:03:44.448554 9337 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 20:03:44.451934 9337 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 7f82ee574bdf426a894cadbfd3de9e20 to finish bootstrapping
I20250625 20:03:44.464229 9337 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 6b2899541b6e4e969255d6eade08f1b3 to finish bootstrapping
I20250625 20:03:44.475016 9337 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 20559cb549264b999ba8d5c6c61770da to finish bootstrapping
I20250625 20:03:44.488754 10827 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60640:
name: "TestAnotherTable"
schema {
columns {
name: "foo"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "bar"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
comment: "comment for bar"
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "foo"
}
}
}
W20250625 20:03:44.490257 10827 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAnotherTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 20:03:44.507683 10952 tablet_service.cc:1468] Processing CreateTablet for tablet 9b82a92cc79047bbae080e7e7c21c821 (DEFAULT_TABLE table=TestAnotherTable [id=81b862adf80543769401d42d6d3850ac]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250625 20:03:44.508136 11085 tablet_service.cc:1468] Processing CreateTablet for tablet 9b82a92cc79047bbae080e7e7c21c821 (DEFAULT_TABLE table=TestAnotherTable [id=81b862adf80543769401d42d6d3850ac]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250625 20:03:44.508812 10952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b82a92cc79047bbae080e7e7c21c821. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:03:44.508636 11218 tablet_service.cc:1468] Processing CreateTablet for tablet 9b82a92cc79047bbae080e7e7c21c821 (DEFAULT_TABLE table=TestAnotherTable [id=81b862adf80543769401d42d6d3850ac]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250625 20:03:44.508980 11085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b82a92cc79047bbae080e7e7c21c821. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:03:44.509677 11218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b82a92cc79047bbae080e7e7c21c821. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:03:44.519150 11302 tablet_bootstrap.cc:492] T 9b82a92cc79047bbae080e7e7c21c821 P 6b2899541b6e4e969255d6eade08f1b3: Bootstrap starting.
I20250625 20:03:44.523545 11303 tablet_bootstrap.cc:492] T 9b82a92cc79047bbae080e7e7c21c821 P 7f82ee574bdf426a894cadbfd3de9e20: Bootstrap starting.
I20250625 20:03:44.524214 11302 tablet_bootstrap.cc:654] T 9b82a92cc79047bbae080e7e7c21c821 P 6b2899541b6e4e969255d6eade08f1b3: Neither blocks nor log segments found. Creating new log.
I20250625 20:03:44.526093 11304 tablet_bootstrap.cc:492] T 9b82a92cc79047bbae080e7e7c21c821 P 20559cb549264b999ba8d5c6c61770da: Bootstrap starting.
I20250625 20:03:44.529206 11303 tablet_bootstrap.cc:654] T 9b82a92cc79047bbae080e7e7c21c821 P 7f82ee574bdf426a894cadbfd3de9e20: Neither blocks nor log segments found. Creating new log.
I20250625 20:03:44.530509 11304 tablet_bootstrap.cc:654] T 9b82a92cc79047bbae080e7e7c21c821 P 20559cb549264b999ba8d5c6c61770da: Neither blocks nor log segments found. Creating new log.
I20250625 20:03:44.534528 11303 tablet_bootstrap.cc:492] T 9b82a92cc79047bbae080e7e7c21c821 P 7f82ee574bdf426a894cadbfd3de9e20: No bootstrap required, opened a new log
I20250625 20:03:44.535228 11303 ts_tablet_manager.cc:1397] T 9b82a92cc79047bbae080e7e7c21c821 P 7f82ee574bdf426a894cadbfd3de9e20: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.007s
I20250625 20:03:44.537690 11302 tablet_bootstrap.cc:492] T 9b82a92cc79047bbae080e7e7c21c821 P 6b2899541b6e4e969255d6eade08f1b3: No bootstrap required, opened a new log
I20250625 20:03:44.537552 11303 raft_consensus.cc:357] T 9b82a92cc79047bbae080e7e7c21c821 P 7f82ee574bdf426a894cadbfd3de9e20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f82ee574bdf426a894cadbfd3de9e20" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 39231 } } peers { permanent_uuid: "6b2899541b6e4e969255d6eade08f1b3" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 43729 } } peers { permanent_uuid: "20559cb549264b999ba8d5c6c61770da" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 46723 } }
I20250625 20:03:44.538180 11302 ts_tablet_manager.cc:1397] T 9b82a92cc79047bbae080e7e7c21c821 P 6b2899541b6e4e969255d6eade08f1b3: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.001s
I20250625 20:03:44.538277 11303 raft_consensus.cc:383] T 9b82a92cc79047bbae080e7e7c21c821 P 7f82ee574bdf426a894cadbfd3de9e20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:03:44.538556 11303 raft_consensus.cc:738] T 9b82a92cc79047bbae080e7e7c21c821 P 7f82ee574bdf426a894cadbfd3de9e20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f82ee574bdf426a894cadbfd3de9e20, State: Initialized, Role: FOLLOWER
I20250625 20:03:44.539440 11304 tablet_bootstrap.cc:492] T 9b82a92cc79047bbae080e7e7c21c821 P 20559cb549264b999ba8d5c6c61770da: No bootstrap required, opened a new log
I20250625 20:03:44.539242 11303 consensus_queue.cc:260] T 9b82a92cc79047bbae080e7e7c21c821 P 7f82ee574bdf426a894cadbfd3de9e20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f82ee574bdf426a894cadbfd3de9e20" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 39231 } } peers { permanent_uuid: "6b2899541b6e4e969255d6eade08f1b3" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 43729 } } peers { permanent_uuid: "20559cb549264b999ba8d5c6c61770da" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 46723 } }
I20250625 20:03:44.539938 11304 ts_tablet_manager.cc:1397] T 9b82a92cc79047bbae080e7e7c21c821 P 20559cb549264b999ba8d5c6c61770da: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.003s
I20250625 20:03:44.541625 11303 ts_tablet_manager.cc:1428] T 9b82a92cc79047bbae080e7e7c21c821 P 7f82ee574bdf426a894cadbfd3de9e20: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250625 20:03:44.541307 11302 raft_consensus.cc:357] T 9b82a92cc79047bbae080e7e7c21c821 P 6b2899541b6e4e969255d6eade08f1b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f82ee574bdf426a894cadbfd3de9e20" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 39231 } } peers { permanent_uuid: "6b2899541b6e4e969255d6eade08f1b3" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 43729 } } peers { permanent_uuid: "20559cb549264b999ba8d5c6c61770da" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 46723 } }
I20250625 20:03:44.542104 11302 raft_consensus.cc:383] T 9b82a92cc79047bbae080e7e7c21c821 P 6b2899541b6e4e969255d6eade08f1b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:03:44.542416 11302 raft_consensus.cc:738] T 9b82a92cc79047bbae080e7e7c21c821 P 6b2899541b6e4e969255d6eade08f1b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b2899541b6e4e969255d6eade08f1b3, State: Initialized, Role: FOLLOWER
I20250625 20:03:44.542583 11304 raft_consensus.cc:357] T 9b82a92cc79047bbae080e7e7c21c821 P 20559cb549264b999ba8d5c6c61770da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f82ee574bdf426a894cadbfd3de9e20" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 39231 } } peers { permanent_uuid: "6b2899541b6e4e969255d6eade08f1b3" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 43729 } } peers { permanent_uuid: "20559cb549264b999ba8d5c6c61770da" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 46723 } }
I20250625 20:03:44.543231 11304 raft_consensus.cc:383] T 9b82a92cc79047bbae080e7e7c21c821 P 20559cb549264b999ba8d5c6c61770da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:03:44.543511 11304 raft_consensus.cc:738] T 9b82a92cc79047bbae080e7e7c21c821 P 20559cb549264b999ba8d5c6c61770da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20559cb549264b999ba8d5c6c61770da, State: Initialized, Role: FOLLOWER
I20250625 20:03:44.543501 11302 consensus_queue.cc:260] T 9b82a92cc79047bbae080e7e7c21c821 P 6b2899541b6e4e969255d6eade08f1b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f82ee574bdf426a894cadbfd3de9e20" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 39231 } } peers { permanent_uuid: "6b2899541b6e4e969255d6eade08f1b3" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 43729 } } peers { permanent_uuid: "20559cb549264b999ba8d5c6c61770da" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 46723 } }
I20250625 20:03:44.544276 11304 consensus_queue.cc:260] T 9b82a92cc79047bbae080e7e7c21c821 P 20559cb549264b999ba8d5c6c61770da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f82ee574bdf426a894cadbfd3de9e20" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 39231 } } peers { permanent_uuid: "6b2899541b6e4e969255d6eade08f1b3" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 43729 } } peers { permanent_uuid: "20559cb549264b999ba8d5c6c61770da" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 46723 } }
I20250625 20:03:44.546398 11302 ts_tablet_manager.cc:1428] T 9b82a92cc79047bbae080e7e7c21c821 P 6b2899541b6e4e969255d6eade08f1b3: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20250625 20:03:44.548321 11304 ts_tablet_manager.cc:1428] T 9b82a92cc79047bbae080e7e7c21c821 P 20559cb549264b999ba8d5c6c61770da: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20250625 20:03:44.793125 11308 raft_consensus.cc:491] T 9b82a92cc79047bbae080e7e7c21c821 P 7f82ee574bdf426a894cadbfd3de9e20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:03:44.793503 11308 raft_consensus.cc:513] T 9b82a92cc79047bbae080e7e7c21c821 P 7f82ee574bdf426a894cadbfd3de9e20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f82ee574bdf426a894cadbfd3de9e20" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 39231 } } peers { permanent_uuid: "6b2899541b6e4e969255d6eade08f1b3" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 43729 } } peers { permanent_uuid: "20559cb549264b999ba8d5c6c61770da" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 46723 } }
I20250625 20:03:44.794940 11308 leader_election.cc:290] T 9b82a92cc79047bbae080e7e7c21c821 P 7f82ee574bdf426a894cadbfd3de9e20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b2899541b6e4e969255d6eade08f1b3 (127.9.30.66:43729), 20559cb549264b999ba8d5c6c61770da (127.9.30.67:46723)
I20250625 20:03:44.795825 11105 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9b82a92cc79047bbae080e7e7c21c821" candidate_uuid: "7f82ee574bdf426a894cadbfd3de9e20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b2899541b6e4e969255d6eade08f1b3" is_pre_election: true
I20250625 20:03:44.795987 11238 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9b82a92cc79047bbae080e7e7c21c821" candidate_uuid: "7f82ee574bdf426a894cadbfd3de9e20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20559cb549264b999ba8d5c6c61770da" is_pre_election: true
I20250625 20:03:44.796283 11105 raft_consensus.cc:2466] T 9b82a92cc79047bbae080e7e7c21c821 P 6b2899541b6e4e969255d6eade08f1b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f82ee574bdf426a894cadbfd3de9e20 in term 0.
I20250625 20:03:44.796460 11238 raft_consensus.cc:2466] T 9b82a92cc79047bbae080e7e7c21c821 P 20559cb549264b999ba8d5c6c61770da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f82ee574bdf426a894cadbfd3de9e20 in term 0.
I20250625 20:03:44.797255 10906 leader_election.cc:304] T 9b82a92cc79047bbae080e7e7c21c821 P 7f82ee574bdf426a894cadbfd3de9e20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b2899541b6e4e969255d6eade08f1b3, 7f82ee574bdf426a894cadbfd3de9e20; no voters:
I20250625 20:03:44.797852 11308 raft_consensus.cc:2802] T 9b82a92cc79047bbae080e7e7c21c821 P 7f82ee574bdf426a894cadbfd3de9e20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:03:44.798156 11308 raft_consensus.cc:491] T 9b82a92cc79047bbae080e7e7c21c821 P 7f82ee574bdf426a894cadbfd3de9e20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:03:44.798408 11308 raft_consensus.cc:3058] T 9b82a92cc79047bbae080e7e7c21c821 P 7f82ee574bdf426a894cadbfd3de9e20 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:03:44.802500 11308 raft_consensus.cc:513] T 9b82a92cc79047bbae080e7e7c21c821 P 7f82ee574bdf426a894cadbfd3de9e20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f82ee574bdf426a894cadbfd3de9e20" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 39231 } } peers { permanent_uuid: "6b2899541b6e4e969255d6eade08f1b3" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 43729 } } peers { permanent_uuid: "20559cb549264b999ba8d5c6c61770da" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 46723 } }
I20250625 20:03:44.803983 11308 leader_election.cc:290] T 9b82a92cc79047bbae080e7e7c21c821 P 7f82ee574bdf426a894cadbfd3de9e20 [CANDIDATE]: Term 1 election: Requested vote from peers 6b2899541b6e4e969255d6eade08f1b3 (127.9.30.66:43729), 20559cb549264b999ba8d5c6c61770da (127.9.30.67:46723)
I20250625 20:03:44.804777 11105 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9b82a92cc79047bbae080e7e7c21c821" candidate_uuid: "7f82ee574bdf426a894cadbfd3de9e20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b2899541b6e4e969255d6eade08f1b3"
I20250625 20:03:44.804926 11238 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9b82a92cc79047bbae080e7e7c21c821" candidate_uuid: "7f82ee574bdf426a894cadbfd3de9e20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20559cb549264b999ba8d5c6c61770da"
I20250625 20:03:44.805195 11105 raft_consensus.cc:3058] T 9b82a92cc79047bbae080e7e7c21c821 P 6b2899541b6e4e969255d6eade08f1b3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:03:44.805368 11238 raft_consensus.cc:3058] T 9b82a92cc79047bbae080e7e7c21c821 P 20559cb549264b999ba8d5c6c61770da [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:03:44.809286 11105 raft_consensus.cc:2466] T 9b82a92cc79047bbae080e7e7c21c821 P 6b2899541b6e4e969255d6eade08f1b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f82ee574bdf426a894cadbfd3de9e20 in term 1.
I20250625 20:03:44.809350 11238 raft_consensus.cc:2466] T 9b82a92cc79047bbae080e7e7c21c821 P 20559cb549264b999ba8d5c6c61770da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f82ee574bdf426a894cadbfd3de9e20 in term 1.
I20250625 20:03:44.810227 10905 leader_election.cc:304] T 9b82a92cc79047bbae080e7e7c21c821 P 7f82ee574bdf426a894cadbfd3de9e20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20559cb549264b999ba8d5c6c61770da, 7f82ee574bdf426a894cadbfd3de9e20; no voters:
I20250625 20:03:44.810853 11308 raft_consensus.cc:2802] T 9b82a92cc79047bbae080e7e7c21c821 P 7f82ee574bdf426a894cadbfd3de9e20 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:03:44.811201 11308 raft_consensus.cc:695] T 9b82a92cc79047bbae080e7e7c21c821 P 7f82ee574bdf426a894cadbfd3de9e20 [term 1 LEADER]: Becoming Leader. State: Replica: 7f82ee574bdf426a894cadbfd3de9e20, State: Running, Role: LEADER
I20250625 20:03:44.811866 11308 consensus_queue.cc:237] T 9b82a92cc79047bbae080e7e7c21c821 P 7f82ee574bdf426a894cadbfd3de9e20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f82ee574bdf426a894cadbfd3de9e20" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 39231 } } peers { permanent_uuid: "6b2899541b6e4e969255d6eade08f1b3" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 43729 } } peers { permanent_uuid: "20559cb549264b999ba8d5c6c61770da" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 46723 } }
I20250625 20:03:44.818359 10826 catalog_manager.cc:5582] T 9b82a92cc79047bbae080e7e7c21c821 P 7f82ee574bdf426a894cadbfd3de9e20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f82ee574bdf426a894cadbfd3de9e20 (127.9.30.65). New cstate: current_term: 1 leader_uuid: "7f82ee574bdf426a894cadbfd3de9e20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f82ee574bdf426a894cadbfd3de9e20" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 39231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b2899541b6e4e969255d6eade08f1b3" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 43729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20559cb549264b999ba8d5c6c61770da" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 46723 } health_report { overall_health: UNKNOWN } } }
I20250625 20:03:44.867086 11308 consensus_queue.cc:1035] T 320fe425de1d4ce195c7931e36dd4c58 P 7f82ee574bdf426a894cadbfd3de9e20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b2899541b6e4e969255d6eade08f1b3" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 43729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:03:44.894580 11315 consensus_queue.cc:1035] T 320fe425de1d4ce195c7931e36dd4c58 P 7f82ee574bdf426a894cadbfd3de9e20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20559cb549264b999ba8d5c6c61770da" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 46723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250625 20:03:45.153916 11323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:03:45.154536 11323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:03:45.185315 11323 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250625 20:03:45.220795 11315 consensus_queue.cc:1035] T 9b82a92cc79047bbae080e7e7c21c821 P 7f82ee574bdf426a894cadbfd3de9e20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20559cb549264b999ba8d5c6c61770da" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 46723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:03:45.234650 11315 consensus_queue.cc:1035] T 9b82a92cc79047bbae080e7e7c21c821 P 7f82ee574bdf426a894cadbfd3de9e20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b2899541b6e4e969255d6eade08f1b3" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 43729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250625 20:03:46.422468 11323 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.196s user 0.480s sys 0.706s
W20250625 20:03:46.422788 11323 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.197s user 0.480s sys 0.706s
W20250625 20:03:47.802568 11345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:03:47.803149 11345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:03:47.834767 11345 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250625 20:03:49.006942 11345 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.128s user 0.466s sys 0.659s
W20250625 20:03:49.007311 11345 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.128s user 0.466s sys 0.659s
W20250625 20:03:50.395947 11361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:03:50.396579 11361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:03:50.428928 11361 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250625 20:03:51.622160 11361 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.154s user 0.498s sys 0.653s
W20250625 20:03:51.622467 11361 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.154s user 0.498s sys 0.653s
W20250625 20:03:52.981097 11376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:03:52.981794 11376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:03:53.013510 11376 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250625 20:03:54.204340 11376 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.148s user 0.446s sys 0.701s
W20250625 20:03:54.204769 11376 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.148s user 0.446s sys 0.701s
I20250625 20:03:55.277922 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 10888
I20250625 20:03:55.302111 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 11021
W20250625 20:03:55.322865 10815 connection.cc:537] server connection from 127.9.30.66:36423 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250625 20:03:55.323280 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 11154
I20250625 20:03:55.343588 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 10796
2025-06-25T20:03:55Z chronyd exiting
[ OK ] AdminCliTest.TestDescribeTableColumnFlags (18556 ms)
[ RUN ] AdminCliTest.TestAuthzResetCacheNotAuthorized
I20250625 20:03:55.397688 9337 test_util.cc:276] Using random seed: -237522403
I20250625 20:03:55.401795 9337 ts_itest-base.cc:115] Starting cluster with:
I20250625 20:03:55.401974 9337 ts_itest-base.cc:116] --------------
I20250625 20:03:55.402133 9337 ts_itest-base.cc:117] 3 tablet servers
I20250625 20:03:55.402261 9337 ts_itest-base.cc:118] 3 replicas per TS
I20250625 20:03:55.402402 9337 ts_itest-base.cc:119] --------------
2025-06-25T20:03:55Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-25T20:03:55Z Disabled control of system clock
I20250625 20:03:55.437043 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.30.126:38121
--webserver_interface=127.9.30.126
--webserver_port=0
--builtin_ntp_servers=127.9.30.84:37641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.30.126:38121
--superuser_acl=no-such-user with env {}
W20250625 20:03:55.727007 11397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:03:55.727561 11397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:03:55.728045 11397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:03:55.759138 11397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 20:03:55.759430 11397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:03:55.759670 11397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 20:03:55.759866 11397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 20:03:55.795364 11397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:37641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.30.126:38121
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.30.126:38121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.9.30.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:03:55.796620 11397 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:03:55.798290 11397 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:03:55.813025 11404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:03:55.813033 11406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:03:55.815145 11397 server_base.cc:1048] running on GCE node
W20250625 20:03:55.813295 11403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:03:56.904794 11397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:03:56.907840 11397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:03:56.909308 11397 hybrid_clock.cc:648] HybridClock initialized: now 1750881836909258 us; error 59 us; skew 500 ppm
I20250625 20:03:56.910166 11397 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:03:56.916759 11397 webserver.cc:469] Webserver started at http://127.9.30.126:42835/ using document root <none> and password file <none>
I20250625 20:03:56.918046 11397 fs_manager.cc:362] Metadata directory not provided
I20250625 20:03:56.918365 11397 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:03:56.920298 11397 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 20:03:56.926254 11397 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "ef56abcd7c52483499aec591d8cf8776"
format_stamp: "Formatted at 2025-06-25 20:03:56 on dist-test-slave-jhxp"
I20250625 20:03:56.927414 11397 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "ef56abcd7c52483499aec591d8cf8776"
format_stamp: "Formatted at 2025-06-25 20:03:56 on dist-test-slave-jhxp"
I20250625 20:03:56.934749 11397 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20250625 20:03:56.940683 11413 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:03:56.941736 11397 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250625 20:03:56.942080 11397 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ef56abcd7c52483499aec591d8cf8776"
format_stamp: "Formatted at 2025-06-25 20:03:56 on dist-test-slave-jhxp"
I20250625 20:03:56.942425 11397 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:03:57.001984 11397 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:03:57.003472 11397 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:03:57.003949 11397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:03:57.079733 11397 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.126:38121
I20250625 20:03:57.079793 11464 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.126:38121 every 8 connection(s)
I20250625 20:03:57.082372 11397 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250625 20:03:57.086592 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 11397
I20250625 20:03:57.087004 9337 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250625 20:03:57.087522 11465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:03:57.112453 11465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef56abcd7c52483499aec591d8cf8776: Bootstrap starting.
I20250625 20:03:57.117909 11465 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef56abcd7c52483499aec591d8cf8776: Neither blocks nor log segments found. Creating new log.
I20250625 20:03:57.119701 11465 log.cc:826] T 00000000000000000000000000000000 P ef56abcd7c52483499aec591d8cf8776: Log is configured to *not* fsync() on all Append() calls
I20250625 20:03:57.124179 11465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef56abcd7c52483499aec591d8cf8776: No bootstrap required, opened a new log
I20250625 20:03:57.141460 11465 raft_consensus.cc:357] T 00000000000000000000000000000000 P ef56abcd7c52483499aec591d8cf8776 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef56abcd7c52483499aec591d8cf8776" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 38121 } }
I20250625 20:03:57.142122 11465 raft_consensus.cc:383] T 00000000000000000000000000000000 P ef56abcd7c52483499aec591d8cf8776 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:03:57.142390 11465 raft_consensus.cc:738] T 00000000000000000000000000000000 P ef56abcd7c52483499aec591d8cf8776 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef56abcd7c52483499aec591d8cf8776, State: Initialized, Role: FOLLOWER
I20250625 20:03:57.143054 11465 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef56abcd7c52483499aec591d8cf8776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef56abcd7c52483499aec591d8cf8776" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 38121 } }
I20250625 20:03:57.143550 11465 raft_consensus.cc:397] T 00000000000000000000000000000000 P ef56abcd7c52483499aec591d8cf8776 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 20:03:57.143922 11465 raft_consensus.cc:491] T 00000000000000000000000000000000 P ef56abcd7c52483499aec591d8cf8776 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 20:03:57.144291 11465 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ef56abcd7c52483499aec591d8cf8776 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:03:57.148519 11465 raft_consensus.cc:513] T 00000000000000000000000000000000 P ef56abcd7c52483499aec591d8cf8776 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef56abcd7c52483499aec591d8cf8776" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 38121 } }
I20250625 20:03:57.149267 11465 leader_election.cc:304] T 00000000000000000000000000000000 P ef56abcd7c52483499aec591d8cf8776 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef56abcd7c52483499aec591d8cf8776; no voters:
I20250625 20:03:57.150879 11465 leader_election.cc:290] T 00000000000000000000000000000000 P ef56abcd7c52483499aec591d8cf8776 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 20:03:57.151513 11470 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ef56abcd7c52483499aec591d8cf8776 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:03:57.153658 11470 raft_consensus.cc:695] T 00000000000000000000000000000000 P ef56abcd7c52483499aec591d8cf8776 [term 1 LEADER]: Becoming Leader. State: Replica: ef56abcd7c52483499aec591d8cf8776, State: Running, Role: LEADER
I20250625 20:03:57.154471 11470 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef56abcd7c52483499aec591d8cf8776 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef56abcd7c52483499aec591d8cf8776" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 38121 } }
I20250625 20:03:57.154835 11465 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef56abcd7c52483499aec591d8cf8776 [sys.catalog]: configured and running, proceeding with master startup.
I20250625 20:03:57.164390 11472 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef56abcd7c52483499aec591d8cf8776 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef56abcd7c52483499aec591d8cf8776. Latest consensus state: current_term: 1 leader_uuid: "ef56abcd7c52483499aec591d8cf8776" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef56abcd7c52483499aec591d8cf8776" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 38121 } } }
I20250625 20:03:57.165139 11472 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef56abcd7c52483499aec591d8cf8776 [sys.catalog]: This master's current role is: LEADER
I20250625 20:03:57.165388 11471 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef56abcd7c52483499aec591d8cf8776 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef56abcd7c52483499aec591d8cf8776" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef56abcd7c52483499aec591d8cf8776" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 38121 } } }
I20250625 20:03:57.165982 11471 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef56abcd7c52483499aec591d8cf8776 [sys.catalog]: This master's current role is: LEADER
I20250625 20:03:57.168398 11480 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 20:03:57.178517 11480 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 20:03:57.193252 11480 catalog_manager.cc:1349] Generated new cluster ID: 464a6b4d5b984d428bfabbcd5db28d73
I20250625 20:03:57.193568 11480 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 20:03:57.218775 11480 catalog_manager.cc:1372] Generated new certificate authority record
I20250625 20:03:57.220904 11480 catalog_manager.cc:1506] Loading token signing keys...
I20250625 20:03:57.238570 11480 catalog_manager.cc:5955] T 00000000000000000000000000000000 P ef56abcd7c52483499aec591d8cf8776: Generated new TSK 0
I20250625 20:03:57.239741 11480 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 20:03:57.261405 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.30.65:0
--local_ip_for_outbound_sockets=127.9.30.65
--webserver_interface=127.9.30.65
--webserver_port=0
--tserver_master_addrs=127.9.30.126:38121
--builtin_ntp_servers=127.9.30.84:37641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250625 20:03:57.563064 11489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:03:57.563583 11489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:03:57.564146 11489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:03:57.595650 11489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:03:57.596529 11489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.30.65
I20250625 20:03:57.632422 11489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:37641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.30.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.9.30.65
--webserver_port=0
--tserver_master_addrs=127.9.30.126:38121
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.30.65
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:03:57.633759 11489 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:03:57.635392 11489 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:03:57.652725 11496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:03:57.652822 11498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:03:57.654207 11489 server_base.cc:1048] running on GCE node
W20250625 20:03:57.654117 11495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:03:58.740545 11489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:03:58.743250 11489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:03:58.744757 11489 hybrid_clock.cc:648] HybridClock initialized: now 1750881838744666 us; error 93 us; skew 500 ppm
I20250625 20:03:58.745642 11489 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:03:58.753214 11489 webserver.cc:469] Webserver started at http://127.9.30.65:35525/ using document root <none> and password file <none>
I20250625 20:03:58.754190 11489 fs_manager.cc:362] Metadata directory not provided
I20250625 20:03:58.754402 11489 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:03:58.754881 11489 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 20:03:58.759667 11489 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "7066daf5ddb94807b8a499f10343e08d"
format_stamp: "Formatted at 2025-06-25 20:03:58 on dist-test-slave-jhxp"
I20250625 20:03:58.760847 11489 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "7066daf5ddb94807b8a499f10343e08d"
format_stamp: "Formatted at 2025-06-25 20:03:58 on dist-test-slave-jhxp"
I20250625 20:03:58.768690 11489 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250625 20:03:58.774829 11505 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:03:58.776007 11489 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250625 20:03:58.776335 11489 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "7066daf5ddb94807b8a499f10343e08d"
format_stamp: "Formatted at 2025-06-25 20:03:58 on dist-test-slave-jhxp"
I20250625 20:03:58.776685 11489 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:03:58.824370 11489 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:03:58.825872 11489 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:03:58.826303 11489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:03:58.828995 11489 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 20:03:58.833621 11489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 20:03:58.833839 11489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 20:03:58.834100 11489 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 20:03:58.834247 11489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 20:03:58.983564 11489 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.65:36467
I20250625 20:03:58.983705 11617 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.65:36467 every 8 connection(s)
I20250625 20:03:58.986099 11489 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250625 20:03:58.992787 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 11489
I20250625 20:03:58.993184 9337 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250625 20:03:58.999586 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.30.66:0
--local_ip_for_outbound_sockets=127.9.30.66
--webserver_interface=127.9.30.66
--webserver_port=0
--tserver_master_addrs=127.9.30.126:38121
--builtin_ntp_servers=127.9.30.84:37641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 20:03:59.008004 11618 heartbeater.cc:344] Connected to a master server at 127.9.30.126:38121
I20250625 20:03:59.008466 11618 heartbeater.cc:461] Registering TS with master...
I20250625 20:03:59.009536 11618 heartbeater.cc:507] Master 127.9.30.126:38121 requested a full tablet report, sending...
I20250625 20:03:59.012146 11430 ts_manager.cc:194] Registered new tserver with Master: 7066daf5ddb94807b8a499f10343e08d (127.9.30.65:36467)
I20250625 20:03:59.014120 11430 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.30.65:54891
W20250625 20:03:59.295239 11622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:03:59.295781 11622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:03:59.296262 11622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:03:59.327510 11622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:03:59.328406 11622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.30.66
I20250625 20:03:59.363438 11622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:37641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.30.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.9.30.66
--webserver_port=0
--tserver_master_addrs=127.9.30.126:38121
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.30.66
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:03:59.364782 11622 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:03:59.366436 11622 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:03:59.381448 11629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:04:00.018004 11618 heartbeater.cc:499] Master 127.9.30.126:38121 was elected leader, sending a full tablet report...
W20250625 20:03:59.382366 11628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:03:59.382759 11631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:04:00.450047 11630 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250625 20:04:00.450160 11622 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 20:04:00.453766 11622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:04:00.455989 11622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:04:00.457360 11622 hybrid_clock.cc:648] HybridClock initialized: now 1750881840457317 us; error 49 us; skew 500 ppm
I20250625 20:04:00.458168 11622 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:04:00.469111 11622 webserver.cc:469] Webserver started at http://127.9.30.66:44327/ using document root <none> and password file <none>
I20250625 20:04:00.470067 11622 fs_manager.cc:362] Metadata directory not provided
I20250625 20:04:00.470294 11622 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:04:00.470750 11622 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 20:04:00.475265 11622 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "ab32b4dc5f194b809f3e41151faae173"
format_stamp: "Formatted at 2025-06-25 20:04:00 on dist-test-slave-jhxp"
I20250625 20:04:00.476466 11622 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "ab32b4dc5f194b809f3e41151faae173"
format_stamp: "Formatted at 2025-06-25 20:04:00 on dist-test-slave-jhxp"
I20250625 20:04:00.483721 11622 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250625 20:04:00.489413 11638 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:04:00.490528 11622 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250625 20:04:00.490871 11622 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ab32b4dc5f194b809f3e41151faae173"
format_stamp: "Formatted at 2025-06-25 20:04:00 on dist-test-slave-jhxp"
I20250625 20:04:00.491217 11622 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:04:00.537604 11622 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:04:00.539054 11622 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:04:00.539502 11622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:04:00.541944 11622 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 20:04:00.546059 11622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 20:04:00.546298 11622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 20:04:00.546538 11622 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 20:04:00.546702 11622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 20:04:00.678579 11622 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.66:37267
I20250625 20:04:00.678666 11750 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.66:37267 every 8 connection(s)
I20250625 20:04:00.681236 11622 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250625 20:04:00.690014 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 11622
I20250625 20:04:00.690385 9337 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250625 20:04:00.696242 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.30.67:0
--local_ip_for_outbound_sockets=127.9.30.67
--webserver_interface=127.9.30.67
--webserver_port=0
--tserver_master_addrs=127.9.30.126:38121
--builtin_ntp_servers=127.9.30.84:37641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 20:04:00.702889 11751 heartbeater.cc:344] Connected to a master server at 127.9.30.126:38121
I20250625 20:04:00.703339 11751 heartbeater.cc:461] Registering TS with master...
I20250625 20:04:00.704635 11751 heartbeater.cc:507] Master 127.9.30.126:38121 requested a full tablet report, sending...
I20250625 20:04:00.707062 11430 ts_manager.cc:194] Registered new tserver with Master: ab32b4dc5f194b809f3e41151faae173 (127.9.30.66:37267)
I20250625 20:04:00.708398 11430 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.30.66:41979
W20250625 20:04:00.987191 11755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:04:00.987740 11755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:04:00.988250 11755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:04:01.022302 11755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:04:01.023590 11755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.30.67
I20250625 20:04:01.059248 11755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:37641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.30.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.9.30.67
--webserver_port=0
--tserver_master_addrs=127.9.30.126:38121
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.30.67
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:04:01.060608 11755 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:04:01.062319 11755 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:04:01.077745 11761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:04:01.711786 11751 heartbeater.cc:499] Master 127.9.30.126:38121 was elected leader, sending a full tablet report...
W20250625 20:04:01.078418 11762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:04:01.077797 11764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:04:02.160791 11763 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250625 20:04:02.160879 11755 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 20:04:02.164445 11755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:04:02.166664 11755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:04:02.168036 11755 hybrid_clock.cc:648] HybridClock initialized: now 1750881842167991 us; error 50 us; skew 500 ppm
I20250625 20:04:02.168934 11755 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:04:02.175045 11755 webserver.cc:469] Webserver started at http://127.9.30.67:40107/ using document root <none> and password file <none>
I20250625 20:04:02.176028 11755 fs_manager.cc:362] Metadata directory not provided
I20250625 20:04:02.176234 11755 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:04:02.176644 11755 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 20:04:02.181027 11755 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "ab58983fbb414a5f866ba696b82def63"
format_stamp: "Formatted at 2025-06-25 20:04:02 on dist-test-slave-jhxp"
I20250625 20:04:02.182117 11755 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "ab58983fbb414a5f866ba696b82def63"
format_stamp: "Formatted at 2025-06-25 20:04:02 on dist-test-slave-jhxp"
I20250625 20:04:02.189728 11755 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.001s sys 0.007s
I20250625 20:04:02.195446 11771 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:04:02.196594 11755 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250625 20:04:02.196924 11755 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ab58983fbb414a5f866ba696b82def63"
format_stamp: "Formatted at 2025-06-25 20:04:02 on dist-test-slave-jhxp"
I20250625 20:04:02.197270 11755 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:04:02.242645 11755 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:04:02.244194 11755 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:04:02.244650 11755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:04:02.247121 11755 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 20:04:02.251157 11755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 20:04:02.251384 11755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 20:04:02.251670 11755 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 20:04:02.251843 11755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 20:04:02.381098 11755 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.67:40783
I20250625 20:04:02.381208 11883 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.67:40783 every 8 connection(s)
I20250625 20:04:02.383666 11755 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250625 20:04:02.385776 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 11755
I20250625 20:04:02.386284 9337 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250625 20:04:02.403909 11884 heartbeater.cc:344] Connected to a master server at 127.9.30.126:38121
I20250625 20:04:02.404377 11884 heartbeater.cc:461] Registering TS with master...
I20250625 20:04:02.405483 11884 heartbeater.cc:507] Master 127.9.30.126:38121 requested a full tablet report, sending...
I20250625 20:04:02.407701 11429 ts_manager.cc:194] Registered new tserver with Master: ab58983fbb414a5f866ba696b82def63 (127.9.30.67:40783)
I20250625 20:04:02.409116 11429 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.30.67:48751
I20250625 20:04:02.422233 9337 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 20:04:02.452648 11429 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58428:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250625 20:04:02.471678 11429 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 20:04:02.527688 11686 tablet_service.cc:1468] Processing CreateTablet for tablet 77151e8fcf88401abd37f1470d456843 (DEFAULT_TABLE table=TestTable [id=379b2ca6b89144db9d67c535b53e6629]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:04:02.527945 11553 tablet_service.cc:1468] Processing CreateTablet for tablet 77151e8fcf88401abd37f1470d456843 (DEFAULT_TABLE table=TestTable [id=379b2ca6b89144db9d67c535b53e6629]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:04:02.529768 11686 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77151e8fcf88401abd37f1470d456843. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:04:02.529999 11553 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77151e8fcf88401abd37f1470d456843. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:04:02.530385 11819 tablet_service.cc:1468] Processing CreateTablet for tablet 77151e8fcf88401abd37f1470d456843 (DEFAULT_TABLE table=TestTable [id=379b2ca6b89144db9d67c535b53e6629]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:04:02.532320 11819 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77151e8fcf88401abd37f1470d456843. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:04:02.551856 11903 tablet_bootstrap.cc:492] T 77151e8fcf88401abd37f1470d456843 P ab32b4dc5f194b809f3e41151faae173: Bootstrap starting.
I20250625 20:04:02.558966 11903 tablet_bootstrap.cc:654] T 77151e8fcf88401abd37f1470d456843 P ab32b4dc5f194b809f3e41151faae173: Neither blocks nor log segments found. Creating new log.
I20250625 20:04:02.561264 11905 tablet_bootstrap.cc:492] T 77151e8fcf88401abd37f1470d456843 P ab58983fbb414a5f866ba696b82def63: Bootstrap starting.
I20250625 20:04:02.561859 11903 log.cc:826] T 77151e8fcf88401abd37f1470d456843 P ab32b4dc5f194b809f3e41151faae173: Log is configured to *not* fsync() on all Append() calls
I20250625 20:04:02.563584 11904 tablet_bootstrap.cc:492] T 77151e8fcf88401abd37f1470d456843 P 7066daf5ddb94807b8a499f10343e08d: Bootstrap starting.
I20250625 20:04:02.568549 11903 tablet_bootstrap.cc:492] T 77151e8fcf88401abd37f1470d456843 P ab32b4dc5f194b809f3e41151faae173: No bootstrap required, opened a new log
I20250625 20:04:02.568974 11905 tablet_bootstrap.cc:654] T 77151e8fcf88401abd37f1470d456843 P ab58983fbb414a5f866ba696b82def63: Neither blocks nor log segments found. Creating new log.
I20250625 20:04:02.569161 11903 ts_tablet_manager.cc:1397] T 77151e8fcf88401abd37f1470d456843 P ab32b4dc5f194b809f3e41151faae173: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.006s
I20250625 20:04:02.570798 11905 log.cc:826] T 77151e8fcf88401abd37f1470d456843 P ab58983fbb414a5f866ba696b82def63: Log is configured to *not* fsync() on all Append() calls
I20250625 20:04:02.571399 11904 tablet_bootstrap.cc:654] T 77151e8fcf88401abd37f1470d456843 P 7066daf5ddb94807b8a499f10343e08d: Neither blocks nor log segments found. Creating new log.
I20250625 20:04:02.573858 11904 log.cc:826] T 77151e8fcf88401abd37f1470d456843 P 7066daf5ddb94807b8a499f10343e08d: Log is configured to *not* fsync() on all Append() calls
I20250625 20:04:02.575989 11905 tablet_bootstrap.cc:492] T 77151e8fcf88401abd37f1470d456843 P ab58983fbb414a5f866ba696b82def63: No bootstrap required, opened a new log
I20250625 20:04:02.576632 11905 ts_tablet_manager.cc:1397] T 77151e8fcf88401abd37f1470d456843 P ab58983fbb414a5f866ba696b82def63: Time spent bootstrapping tablet: real 0.016s user 0.015s sys 0.000s
I20250625 20:04:02.579851 11904 tablet_bootstrap.cc:492] T 77151e8fcf88401abd37f1470d456843 P 7066daf5ddb94807b8a499f10343e08d: No bootstrap required, opened a new log
I20250625 20:04:02.580399 11904 ts_tablet_manager.cc:1397] T 77151e8fcf88401abd37f1470d456843 P 7066daf5ddb94807b8a499f10343e08d: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.003s
I20250625 20:04:02.595470 11905 raft_consensus.cc:357] T 77151e8fcf88401abd37f1470d456843 P ab58983fbb414a5f866ba696b82def63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7066daf5ddb94807b8a499f10343e08d" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 36467 } } peers { permanent_uuid: "ab32b4dc5f194b809f3e41151faae173" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 37267 } } peers { permanent_uuid: "ab58983fbb414a5f866ba696b82def63" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 40783 } }
I20250625 20:04:02.596228 11905 raft_consensus.cc:383] T 77151e8fcf88401abd37f1470d456843 P ab58983fbb414a5f866ba696b82def63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:04:02.595747 11903 raft_consensus.cc:357] T 77151e8fcf88401abd37f1470d456843 P ab32b4dc5f194b809f3e41151faae173 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7066daf5ddb94807b8a499f10343e08d" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 36467 } } peers { permanent_uuid: "ab32b4dc5f194b809f3e41151faae173" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 37267 } } peers { permanent_uuid: "ab58983fbb414a5f866ba696b82def63" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 40783 } }
I20250625 20:04:02.596506 11905 raft_consensus.cc:738] T 77151e8fcf88401abd37f1470d456843 P ab58983fbb414a5f866ba696b82def63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab58983fbb414a5f866ba696b82def63, State: Initialized, Role: FOLLOWER
I20250625 20:04:02.596639 11903 raft_consensus.cc:383] T 77151e8fcf88401abd37f1470d456843 P ab32b4dc5f194b809f3e41151faae173 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:04:02.596930 11903 raft_consensus.cc:738] T 77151e8fcf88401abd37f1470d456843 P ab32b4dc5f194b809f3e41151faae173 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab32b4dc5f194b809f3e41151faae173, State: Initialized, Role: FOLLOWER
I20250625 20:04:02.597362 11905 consensus_queue.cc:260] T 77151e8fcf88401abd37f1470d456843 P ab58983fbb414a5f866ba696b82def63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7066daf5ddb94807b8a499f10343e08d" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 36467 } } peers { permanent_uuid: "ab32b4dc5f194b809f3e41151faae173" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 37267 } } peers { permanent_uuid: "ab58983fbb414a5f866ba696b82def63" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 40783 } }
I20250625 20:04:02.597807 11903 consensus_queue.cc:260] T 77151e8fcf88401abd37f1470d456843 P ab32b4dc5f194b809f3e41151faae173 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7066daf5ddb94807b8a499f10343e08d" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 36467 } } peers { permanent_uuid: "ab32b4dc5f194b809f3e41151faae173" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 37267 } } peers { permanent_uuid: "ab58983fbb414a5f866ba696b82def63" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 40783 } }
I20250625 20:04:02.602581 11884 heartbeater.cc:499] Master 127.9.30.126:38121 was elected leader, sending a full tablet report...
I20250625 20:04:02.604025 11905 ts_tablet_manager.cc:1428] T 77151e8fcf88401abd37f1470d456843 P ab58983fbb414a5f866ba696b82def63: Time spent starting tablet: real 0.027s user 0.016s sys 0.011s
I20250625 20:04:02.607106 11903 ts_tablet_manager.cc:1428] T 77151e8fcf88401abd37f1470d456843 P ab32b4dc5f194b809f3e41151faae173: Time spent starting tablet: real 0.038s user 0.029s sys 0.008s
I20250625 20:04:02.608062 11904 raft_consensus.cc:357] T 77151e8fcf88401abd37f1470d456843 P 7066daf5ddb94807b8a499f10343e08d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7066daf5ddb94807b8a499f10343e08d" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 36467 } } peers { permanent_uuid: "ab32b4dc5f194b809f3e41151faae173" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 37267 } } peers { permanent_uuid: "ab58983fbb414a5f866ba696b82def63" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 40783 } }
I20250625 20:04:02.608906 11904 raft_consensus.cc:383] T 77151e8fcf88401abd37f1470d456843 P 7066daf5ddb94807b8a499f10343e08d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:04:02.609184 11904 raft_consensus.cc:738] T 77151e8fcf88401abd37f1470d456843 P 7066daf5ddb94807b8a499f10343e08d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7066daf5ddb94807b8a499f10343e08d, State: Initialized, Role: FOLLOWER
I20250625 20:04:02.609943 11904 consensus_queue.cc:260] T 77151e8fcf88401abd37f1470d456843 P 7066daf5ddb94807b8a499f10343e08d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7066daf5ddb94807b8a499f10343e08d" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 36467 } } peers { permanent_uuid: "ab32b4dc5f194b809f3e41151faae173" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 37267 } } peers { permanent_uuid: "ab58983fbb414a5f866ba696b82def63" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 40783 } }
I20250625 20:04:02.615518 11904 ts_tablet_manager.cc:1428] T 77151e8fcf88401abd37f1470d456843 P 7066daf5ddb94807b8a499f10343e08d: Time spent starting tablet: real 0.035s user 0.032s sys 0.000s
W20250625 20:04:02.638262 11885 tablet.cc:2378] T 77151e8fcf88401abd37f1470d456843 P ab58983fbb414a5f866ba696b82def63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 20:04:02.652052 11909 raft_consensus.cc:491] T 77151e8fcf88401abd37f1470d456843 P ab58983fbb414a5f866ba696b82def63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:04:02.652586 11909 raft_consensus.cc:513] T 77151e8fcf88401abd37f1470d456843 P ab58983fbb414a5f866ba696b82def63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7066daf5ddb94807b8a499f10343e08d" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 36467 } } peers { permanent_uuid: "ab32b4dc5f194b809f3e41151faae173" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 37267 } } peers { permanent_uuid: "ab58983fbb414a5f866ba696b82def63" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 40783 } }
I20250625 20:04:02.655174 11909 leader_election.cc:290] T 77151e8fcf88401abd37f1470d456843 P ab58983fbb414a5f866ba696b82def63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7066daf5ddb94807b8a499f10343e08d (127.9.30.65:36467), ab32b4dc5f194b809f3e41151faae173 (127.9.30.66:37267)
I20250625 20:04:02.665035 11573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "77151e8fcf88401abd37f1470d456843" candidate_uuid: "ab58983fbb414a5f866ba696b82def63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7066daf5ddb94807b8a499f10343e08d" is_pre_election: true
I20250625 20:04:02.665897 11573 raft_consensus.cc:2466] T 77151e8fcf88401abd37f1470d456843 P 7066daf5ddb94807b8a499f10343e08d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab58983fbb414a5f866ba696b82def63 in term 0.
I20250625 20:04:02.666983 11910 raft_consensus.cc:491] T 77151e8fcf88401abd37f1470d456843 P ab32b4dc5f194b809f3e41151faae173 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:04:02.667196 11706 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "77151e8fcf88401abd37f1470d456843" candidate_uuid: "ab58983fbb414a5f866ba696b82def63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab32b4dc5f194b809f3e41151faae173" is_pre_election: true
I20250625 20:04:02.667351 11775 leader_election.cc:304] T 77151e8fcf88401abd37f1470d456843 P ab58983fbb414a5f866ba696b82def63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7066daf5ddb94807b8a499f10343e08d, ab58983fbb414a5f866ba696b82def63; no voters:
I20250625 20:04:02.667506 11910 raft_consensus.cc:513] T 77151e8fcf88401abd37f1470d456843 P ab32b4dc5f194b809f3e41151faae173 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7066daf5ddb94807b8a499f10343e08d" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 36467 } } peers { permanent_uuid: "ab32b4dc5f194b809f3e41151faae173" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 37267 } } peers { permanent_uuid: "ab58983fbb414a5f866ba696b82def63" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 40783 } }
I20250625 20:04:02.668215 11909 raft_consensus.cc:2802] T 77151e8fcf88401abd37f1470d456843 P ab58983fbb414a5f866ba696b82def63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:04:02.668555 11909 raft_consensus.cc:491] T 77151e8fcf88401abd37f1470d456843 P ab58983fbb414a5f866ba696b82def63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:04:02.668741 11706 raft_consensus.cc:2466] T 77151e8fcf88401abd37f1470d456843 P ab32b4dc5f194b809f3e41151faae173 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab58983fbb414a5f866ba696b82def63 in term 0.
I20250625 20:04:02.668927 11909 raft_consensus.cc:3058] T 77151e8fcf88401abd37f1470d456843 P ab58983fbb414a5f866ba696b82def63 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:04:02.670325 11910 leader_election.cc:290] T 77151e8fcf88401abd37f1470d456843 P ab32b4dc5f194b809f3e41151faae173 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7066daf5ddb94807b8a499f10343e08d (127.9.30.65:36467), ab58983fbb414a5f866ba696b82def63 (127.9.30.67:40783)
I20250625 20:04:02.676187 11909 raft_consensus.cc:513] T 77151e8fcf88401abd37f1470d456843 P ab58983fbb414a5f866ba696b82def63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7066daf5ddb94807b8a499f10343e08d" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 36467 } } peers { permanent_uuid: "ab32b4dc5f194b809f3e41151faae173" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 37267 } } peers { permanent_uuid: "ab58983fbb414a5f866ba696b82def63" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 40783 } }
I20250625 20:04:02.679134 11573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "77151e8fcf88401abd37f1470d456843" candidate_uuid: "ab58983fbb414a5f866ba696b82def63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7066daf5ddb94807b8a499f10343e08d"
I20250625 20:04:02.679422 11706 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "77151e8fcf88401abd37f1470d456843" candidate_uuid: "ab58983fbb414a5f866ba696b82def63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab32b4dc5f194b809f3e41151faae173"
I20250625 20:04:02.679744 11573 raft_consensus.cc:3058] T 77151e8fcf88401abd37f1470d456843 P 7066daf5ddb94807b8a499f10343e08d [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:04:02.680040 11706 raft_consensus.cc:3058] T 77151e8fcf88401abd37f1470d456843 P ab32b4dc5f194b809f3e41151faae173 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:04:02.685078 11573 raft_consensus.cc:2466] T 77151e8fcf88401abd37f1470d456843 P 7066daf5ddb94807b8a499f10343e08d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab58983fbb414a5f866ba696b82def63 in term 1.
I20250625 20:04:02.686386 11775 leader_election.cc:304] T 77151e8fcf88401abd37f1470d456843 P ab58983fbb414a5f866ba696b82def63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7066daf5ddb94807b8a499f10343e08d, ab58983fbb414a5f866ba696b82def63; no voters:
I20250625 20:04:02.687806 11909 leader_election.cc:290] T 77151e8fcf88401abd37f1470d456843 P ab58983fbb414a5f866ba696b82def63 [CANDIDATE]: Term 1 election: Requested vote from peers 7066daf5ddb94807b8a499f10343e08d (127.9.30.65:36467), ab32b4dc5f194b809f3e41151faae173 (127.9.30.66:37267)
I20250625 20:04:02.688387 11909 raft_consensus.cc:2802] T 77151e8fcf88401abd37f1470d456843 P ab58983fbb414a5f866ba696b82def63 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:04:02.687479 11706 raft_consensus.cc:2466] T 77151e8fcf88401abd37f1470d456843 P ab32b4dc5f194b809f3e41151faae173 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab58983fbb414a5f866ba696b82def63 in term 1.
W20250625 20:04:02.688930 11752 tablet.cc:2378] T 77151e8fcf88401abd37f1470d456843 P ab32b4dc5f194b809f3e41151faae173: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 20:04:02.690330 11909 raft_consensus.cc:695] T 77151e8fcf88401abd37f1470d456843 P ab58983fbb414a5f866ba696b82def63 [term 1 LEADER]: Becoming Leader. State: Replica: ab58983fbb414a5f866ba696b82def63, State: Running, Role: LEADER
I20250625 20:04:02.691526 11909 consensus_queue.cc:237] T 77151e8fcf88401abd37f1470d456843 P ab58983fbb414a5f866ba696b82def63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7066daf5ddb94807b8a499f10343e08d" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 36467 } } peers { permanent_uuid: "ab32b4dc5f194b809f3e41151faae173" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 37267 } } peers { permanent_uuid: "ab58983fbb414a5f866ba696b82def63" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 40783 } }
I20250625 20:04:02.693264 11573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "77151e8fcf88401abd37f1470d456843" candidate_uuid: "ab32b4dc5f194b809f3e41151faae173" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7066daf5ddb94807b8a499f10343e08d" is_pre_election: true
I20250625 20:04:02.694636 11573 raft_consensus.cc:2391] T 77151e8fcf88401abd37f1470d456843 P 7066daf5ddb94807b8a499f10343e08d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab32b4dc5f194b809f3e41151faae173 in current term 1: Already voted for candidate ab58983fbb414a5f866ba696b82def63 in this term.
I20250625 20:04:02.698611 11839 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "77151e8fcf88401abd37f1470d456843" candidate_uuid: "ab32b4dc5f194b809f3e41151faae173" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab58983fbb414a5f866ba696b82def63" is_pre_election: true
I20250625 20:04:02.700800 11641 leader_election.cc:304] T 77151e8fcf88401abd37f1470d456843 P ab32b4dc5f194b809f3e41151faae173 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab32b4dc5f194b809f3e41151faae173; no voters: 7066daf5ddb94807b8a499f10343e08d, ab58983fbb414a5f866ba696b82def63
I20250625 20:04:02.701618 11910 raft_consensus.cc:2747] T 77151e8fcf88401abd37f1470d456843 P ab32b4dc5f194b809f3e41151faae173 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 20:04:02.707996 11430 catalog_manager.cc:5582] T 77151e8fcf88401abd37f1470d456843 P ab58983fbb414a5f866ba696b82def63 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab58983fbb414a5f866ba696b82def63 (127.9.30.67). New cstate: current_term: 1 leader_uuid: "ab58983fbb414a5f866ba696b82def63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7066daf5ddb94807b8a499f10343e08d" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 36467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab32b4dc5f194b809f3e41151faae173" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab58983fbb414a5f866ba696b82def63" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 40783 } health_report { overall_health: HEALTHY } } }
I20250625 20:04:02.726917 9337 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 20:04:02.730414 9337 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 7066daf5ddb94807b8a499f10343e08d to finish bootstrapping
I20250625 20:04:02.743242 9337 ts_itest-base.cc:246] Waiting for 1 tablets on tserver ab32b4dc5f194b809f3e41151faae173 to finish bootstrapping
W20250625 20:04:02.743929 11619 tablet.cc:2378] T 77151e8fcf88401abd37f1470d456843 P 7066daf5ddb94807b8a499f10343e08d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 20:04:02.754047 9337 ts_itest-base.cc:246] Waiting for 1 tablets on tserver ab58983fbb414a5f866ba696b82def63 to finish bootstrapping
I20250625 20:04:03.315629 11928 consensus_queue.cc:1035] T 77151e8fcf88401abd37f1470d456843 P ab58983fbb414a5f866ba696b82def63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7066daf5ddb94807b8a499f10343e08d" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 36467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250625 20:04:03.324802 11928 consensus_queue.cc:1035] T 77151e8fcf88401abd37f1470d456843 P ab58983fbb414a5f866ba696b82def63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab32b4dc5f194b809f3e41151faae173" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250625 20:04:04.378602 11428 server_base.cc:1130] Unauthorized access attempt to method kudu.master.MasterService.RefreshAuthzCache from {username='slave'} at 127.0.0.1:58444
W20250625 20:04:04.714995 11747 debug-util.cc:398] Leaking SignalData structure 0x7b08000b5de0 after lost signal to thread 11623
W20250625 20:04:04.715826 11747 debug-util.cc:398] Leaking SignalData structure 0x7b08000cbec0 after lost signal to thread 11750
I20250625 20:04:05.414244 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 11489
I20250625 20:04:05.437376 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 11622
I20250625 20:04:05.460661 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 11755
I20250625 20:04:05.480875 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 11397
2025-06-25T20:04:05Z chronyd exiting
[ OK ] AdminCliTest.TestAuthzResetCacheNotAuthorized (10139 ms)
[ RUN ] AdminCliTest.TestRebuildTables
I20250625 20:04:05.537434 9337 test_util.cc:276] Using random seed: -227382659
I20250625 20:04:05.541785 9337 ts_itest-base.cc:115] Starting cluster with:
I20250625 20:04:05.541980 9337 ts_itest-base.cc:116] --------------
I20250625 20:04:05.542143 9337 ts_itest-base.cc:117] 3 tablet servers
I20250625 20:04:05.542284 9337 ts_itest-base.cc:118] 3 replicas per TS
I20250625 20:04:05.542428 9337 ts_itest-base.cc:119] --------------
2025-06-25T20:04:05Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-25T20:04:05Z Disabled control of system clock
I20250625 20:04:05.577783 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.30.126:39493
--webserver_interface=127.9.30.126
--webserver_port=0
--builtin_ntp_servers=127.9.30.84:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.30.126:39493 with env {}
W20250625 20:04:05.872324 11953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:04:05.872881 11953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:04:05.873301 11953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:04:05.904644 11953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 20:04:05.904953 11953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:04:05.905155 11953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 20:04:05.905369 11953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 20:04:05.941448 11953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.30.126:39493
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.30.126:39493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.9.30.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:04:05.942741 11953 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:04:05.944458 11953 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:04:05.958024 11959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:04:05.958029 11960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:04:07.069599 11962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:04:07.072631 11961 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1116 milliseconds
W20250625 20:04:07.073508 11953 thread.cc:640] OpenStack (cloud detector) Time spent creating pthread: real 1.118s user 0.374s sys 0.742s
W20250625 20:04:07.073767 11953 thread.cc:606] OpenStack (cloud detector) Time spent starting thread: real 1.118s user 0.374s sys 0.742s
I20250625 20:04:07.073987 11953 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 20:04:07.075078 11953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:04:07.077805 11953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:04:07.079195 11953 hybrid_clock.cc:648] HybridClock initialized: now 1750881847079144 us; error 71 us; skew 500 ppm
I20250625 20:04:07.080090 11953 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:04:07.090586 11953 webserver.cc:469] Webserver started at http://127.9.30.126:43167/ using document root <none> and password file <none>
I20250625 20:04:07.091620 11953 fs_manager.cc:362] Metadata directory not provided
I20250625 20:04:07.091872 11953 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:04:07.092345 11953 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 20:04:07.098342 11953 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "4d3eec5ea99248df990c8b46a3384c6b"
format_stamp: "Formatted at 2025-06-25 20:04:07 on dist-test-slave-jhxp"
I20250625 20:04:07.099762 11953 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "4d3eec5ea99248df990c8b46a3384c6b"
format_stamp: "Formatted at 2025-06-25 20:04:07 on dist-test-slave-jhxp"
I20250625 20:04:07.108007 11953 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.002s sys 0.006s
I20250625 20:04:07.114107 11969 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:04:07.115404 11953 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20250625 20:04:07.115804 11953 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
uuid: "4d3eec5ea99248df990c8b46a3384c6b"
format_stamp: "Formatted at 2025-06-25 20:04:07 on dist-test-slave-jhxp"
I20250625 20:04:07.116168 11953 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:04:07.184608 11953 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:04:07.186156 11953 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:04:07.186611 11953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:04:07.255465 11953 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.126:39493
I20250625 20:04:07.255554 12020 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.126:39493 every 8 connection(s)
I20250625 20:04:07.258152 11953 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250625 20:04:07.258841 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 11953
I20250625 20:04:07.259231 9337 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250625 20:04:07.264904 12021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:04:07.287040 12021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b: Bootstrap starting.
I20250625 20:04:07.292933 12021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b: Neither blocks nor log segments found. Creating new log.
I20250625 20:04:07.294819 12021 log.cc:826] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b: Log is configured to *not* fsync() on all Append() calls
I20250625 20:04:07.299530 12021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b: No bootstrap required, opened a new log
I20250625 20:04:07.318143 12021 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3eec5ea99248df990c8b46a3384c6b" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 39493 } }
I20250625 20:04:07.318812 12021 raft_consensus.cc:383] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:04:07.319072 12021 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3eec5ea99248df990c8b46a3384c6b, State: Initialized, Role: FOLLOWER
I20250625 20:04:07.319782 12021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3eec5ea99248df990c8b46a3384c6b" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 39493 } }
I20250625 20:04:07.320286 12021 raft_consensus.cc:397] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 20:04:07.320546 12021 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 20:04:07.320840 12021 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:04:07.325711 12021 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3eec5ea99248df990c8b46a3384c6b" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 39493 } }
I20250625 20:04:07.326463 12021 leader_election.cc:304] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d3eec5ea99248df990c8b46a3384c6b; no voters:
I20250625 20:04:07.328128 12021 leader_election.cc:290] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 20:04:07.328876 12026 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:04:07.331624 12026 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [term 1 LEADER]: Becoming Leader. State: Replica: 4d3eec5ea99248df990c8b46a3384c6b, State: Running, Role: LEADER
I20250625 20:04:07.332610 12021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [sys.catalog]: configured and running, proceeding with master startup.
I20250625 20:04:07.332444 12026 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3eec5ea99248df990c8b46a3384c6b" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 39493 } }
I20250625 20:04:07.340066 12028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d3eec5ea99248df990c8b46a3384c6b. Latest consensus state: current_term: 1 leader_uuid: "4d3eec5ea99248df990c8b46a3384c6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3eec5ea99248df990c8b46a3384c6b" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 39493 } } }
I20250625 20:04:07.340391 12027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d3eec5ea99248df990c8b46a3384c6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3eec5ea99248df990c8b46a3384c6b" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 39493 } } }
I20250625 20:04:07.340971 12028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [sys.catalog]: This master's current role is: LEADER
I20250625 20:04:07.340971 12027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [sys.catalog]: This master's current role is: LEADER
I20250625 20:04:07.349962 12034 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 20:04:07.360047 12034 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 20:04:07.375663 12034 catalog_manager.cc:1349] Generated new cluster ID: de233e8fc4a34d7db5e31c956aad7d4b
I20250625 20:04:07.376004 12034 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 20:04:07.408631 12034 catalog_manager.cc:1372] Generated new certificate authority record
I20250625 20:04:07.410779 12034 catalog_manager.cc:1506] Loading token signing keys...
I20250625 20:04:07.430717 12034 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b: Generated new TSK 0
I20250625 20:04:07.432193 12034 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 20:04:07.457407 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.30.65:0
--local_ip_for_outbound_sockets=127.9.30.65
--webserver_interface=127.9.30.65
--webserver_port=0
--tserver_master_addrs=127.9.30.126:39493
--builtin_ntp_servers=127.9.30.84:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250625 20:04:07.760340 12045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:04:07.760874 12045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:04:07.761381 12045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:04:07.794842 12045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:04:07.795778 12045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.30.65
I20250625 20:04:07.831763 12045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.30.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.9.30.65
--webserver_port=0
--tserver_master_addrs=127.9.30.126:39493
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.30.65
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:04:07.833093 12045 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:04:07.834877 12045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:04:07.853413 12052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:04:07.854301 12054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:04:07.855453 12045 server_base.cc:1048] running on GCE node
W20250625 20:04:07.853410 12051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:04:08.963421 12045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:04:08.974547 12045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:04:08.976051 12045 hybrid_clock.cc:648] HybridClock initialized: now 1750881848975980 us; error 88 us; skew 500 ppm
I20250625 20:04:08.976935 12045 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:04:08.983835 12045 webserver.cc:469] Webserver started at http://127.9.30.65:33919/ using document root <none> and password file <none>
I20250625 20:04:08.984875 12045 fs_manager.cc:362] Metadata directory not provided
I20250625 20:04:08.985116 12045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:04:08.985606 12045 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 20:04:08.990104 12045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "44f691d035604bcab46e9120f6028166"
format_stamp: "Formatted at 2025-06-25 20:04:08 on dist-test-slave-jhxp"
I20250625 20:04:08.991262 12045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "44f691d035604bcab46e9120f6028166"
format_stamp: "Formatted at 2025-06-25 20:04:08 on dist-test-slave-jhxp"
I20250625 20:04:08.998891 12045 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250625 20:04:09.004935 12061 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:04:09.006079 12045 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.005s
I20250625 20:04:09.006412 12045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "44f691d035604bcab46e9120f6028166"
format_stamp: "Formatted at 2025-06-25 20:04:08 on dist-test-slave-jhxp"
I20250625 20:04:09.006772 12045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:04:09.062166 12045 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:04:09.063706 12045 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:04:09.064150 12045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:04:09.066915 12045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 20:04:09.071578 12045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 20:04:09.071831 12045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20250625 20:04:09.072113 12045 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 20:04:09.072261 12045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 20:04:09.215140 12045 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.65:44285
I20250625 20:04:09.215250 12173 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.65:44285 every 8 connection(s)
I20250625 20:04:09.217811 12045 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250625 20:04:09.220125 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 12045
I20250625 20:04:09.220639 9337 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250625 20:04:09.227437 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.30.66:0
--local_ip_for_outbound_sockets=127.9.30.66
--webserver_interface=127.9.30.66
--webserver_port=0
--tserver_master_addrs=127.9.30.126:39493
--builtin_ntp_servers=127.9.30.84:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 20:04:09.241568 12174 heartbeater.cc:344] Connected to a master server at 127.9.30.126:39493
I20250625 20:04:09.242058 12174 heartbeater.cc:461] Registering TS with master...
I20250625 20:04:09.243186 12174 heartbeater.cc:507] Master 127.9.30.126:39493 requested a full tablet report, sending...
I20250625 20:04:09.246205 11986 ts_manager.cc:194] Registered new tserver with Master: 44f691d035604bcab46e9120f6028166 (127.9.30.65:44285)
I20250625 20:04:09.249058 11986 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.30.65:43213
W20250625 20:04:09.537120 12178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:04:09.537616 12178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:04:09.538076 12178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:04:09.569914 12178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:04:09.570748 12178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.30.66
I20250625 20:04:09.605612 12178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.30.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.9.30.66
--webserver_port=0
--tserver_master_addrs=127.9.30.126:39493
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.30.66
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:04:09.606861 12178 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:04:09.608590 12178 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:04:09.625135 12184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:04:10.253201 12174 heartbeater.cc:499] Master 127.9.30.126:39493 was elected leader, sending a full tablet report...
W20250625 20:04:09.625429 12185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:04:09.627530 12178 server_base.cc:1048] running on GCE node
W20250625 20:04:09.626566 12187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:04:10.705257 12178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:04:10.707670 12178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:04:10.709084 12178 hybrid_clock.cc:648] HybridClock initialized: now 1750881850709037 us; error 52 us; skew 500 ppm
I20250625 20:04:10.709921 12178 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:04:10.716122 12178 webserver.cc:469] Webserver started at http://127.9.30.66:40759/ using document root <none> and password file <none>
I20250625 20:04:10.717152 12178 fs_manager.cc:362] Metadata directory not provided
I20250625 20:04:10.717384 12178 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:04:10.717860 12178 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 20:04:10.722348 12178 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "526f4e745a694c10a4403c8e8de7b98d"
format_stamp: "Formatted at 2025-06-25 20:04:10 on dist-test-slave-jhxp"
I20250625 20:04:10.723495 12178 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "526f4e745a694c10a4403c8e8de7b98d"
format_stamp: "Formatted at 2025-06-25 20:04:10 on dist-test-slave-jhxp"
I20250625 20:04:10.730917 12178 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20250625 20:04:10.736799 12194 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:04:10.737962 12178 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.005s
I20250625 20:04:10.738306 12178 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "526f4e745a694c10a4403c8e8de7b98d"
format_stamp: "Formatted at 2025-06-25 20:04:10 on dist-test-slave-jhxp"
I20250625 20:04:10.738663 12178 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:04:10.789358 12178 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:04:10.790840 12178 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:04:10.791281 12178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:04:10.793826 12178 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 20:04:10.798031 12178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 20:04:10.798254 12178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 20:04:10.798498 12178 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 20:04:10.798663 12178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 20:04:10.927762 12178 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.66:41265
I20250625 20:04:10.927867 12306 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.66:41265 every 8 connection(s)
I20250625 20:04:10.930284 12178 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250625 20:04:10.932367 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 12178
I20250625 20:04:10.932902 9337 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250625 20:04:10.939975 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.30.67:0
--local_ip_for_outbound_sockets=127.9.30.67
--webserver_interface=127.9.30.67
--webserver_port=0
--tserver_master_addrs=127.9.30.126:39493
--builtin_ntp_servers=127.9.30.84:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 20:04:10.953388 12307 heartbeater.cc:344] Connected to a master server at 127.9.30.126:39493
I20250625 20:04:10.953953 12307 heartbeater.cc:461] Registering TS with master...
I20250625 20:04:10.955242 12307 heartbeater.cc:507] Master 127.9.30.126:39493 requested a full tablet report, sending...
I20250625 20:04:10.957855 11986 ts_manager.cc:194] Registered new tserver with Master: 526f4e745a694c10a4403c8e8de7b98d (127.9.30.66:41265)
I20250625 20:04:10.959110 11986 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.30.66:54613
W20250625 20:04:11.242672 12311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:04:11.243181 12311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:04:11.243731 12311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:04:11.277557 12311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:04:11.278432 12311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.30.67
I20250625 20:04:11.313750 12311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.30.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.9.30.67
--webserver_port=0
--tserver_master_addrs=127.9.30.126:39493
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.30.67
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:04:11.315078 12311 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:04:11.316787 12311 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:04:11.333501 12318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:04:11.962762 12307 heartbeater.cc:499] Master 127.9.30.126:39493 was elected leader, sending a full tablet report...
W20250625 20:04:11.333539 12317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:04:11.335744 12311 server_base.cc:1048] running on GCE node
W20250625 20:04:11.333730 12320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:04:12.414583 12311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:04:12.417409 12311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:04:12.418908 12311 hybrid_clock.cc:648] HybridClock initialized: now 1750881852418873 us; error 63 us; skew 500 ppm
I20250625 20:04:12.419802 12311 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:04:12.438212 12311 webserver.cc:469] Webserver started at http://127.9.30.67:35311/ using document root <none> and password file <none>
I20250625 20:04:12.439127 12311 fs_manager.cc:362] Metadata directory not provided
I20250625 20:04:12.439357 12311 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:04:12.439867 12311 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 20:04:12.444511 12311 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "df7ba59f0fd749819107c7ef7cff68cd"
format_stamp: "Formatted at 2025-06-25 20:04:12 on dist-test-slave-jhxp"
I20250625 20:04:12.445750 12311 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "df7ba59f0fd749819107c7ef7cff68cd"
format_stamp: "Formatted at 2025-06-25 20:04:12 on dist-test-slave-jhxp"
I20250625 20:04:12.455319 12311 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.005s sys 0.004s
I20250625 20:04:12.461498 12327 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:04:12.462592 12311 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20250625 20:04:12.462925 12311 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "df7ba59f0fd749819107c7ef7cff68cd"
format_stamp: "Formatted at 2025-06-25 20:04:12 on dist-test-slave-jhxp"
I20250625 20:04:12.463260 12311 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:04:12.513842 12311 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:04:12.515372 12311 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:04:12.515858 12311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:04:12.518294 12311 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 20:04:12.522431 12311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 20:04:12.522675 12311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 20:04:12.522927 12311 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 20:04:12.523093 12311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 20:04:12.656260 12311 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.67:34021
I20250625 20:04:12.656371 12439 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.67:34021 every 8 connection(s)
I20250625 20:04:12.658864 12311 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250625 20:04:12.668931 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 12311
I20250625 20:04:12.669474 9337 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250625 20:04:12.681787 12440 heartbeater.cc:344] Connected to a master server at 127.9.30.126:39493
I20250625 20:04:12.682243 12440 heartbeater.cc:461] Registering TS with master...
I20250625 20:04:12.683265 12440 heartbeater.cc:507] Master 127.9.30.126:39493 requested a full tablet report, sending...
I20250625 20:04:12.685503 11986 ts_manager.cc:194] Registered new tserver with Master: df7ba59f0fd749819107c7ef7cff68cd (127.9.30.67:34021)
I20250625 20:04:12.686776 11986 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.30.67:53749
I20250625 20:04:12.690771 9337 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 20:04:12.723070 9337 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 20:04:12.723433 9337 test_util.cc:276] Using random seed: -220196651
I20250625 20:04:12.765506 11986 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43844:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:04:12.807305 12109 tablet_service.cc:1468] Processing CreateTablet for tablet bd510751b1cb4241a9f5541b94098e8c (DEFAULT_TABLE table=TestTable [id=b1883219b5d244bba8dee9a0514130bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:04:12.808934 12109 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd510751b1cb4241a9f5541b94098e8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:04:12.829370 12460 tablet_bootstrap.cc:492] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166: Bootstrap starting.
I20250625 20:04:12.835374 12460 tablet_bootstrap.cc:654] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166: Neither blocks nor log segments found. Creating new log.
I20250625 20:04:12.837182 12460 log.cc:826] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166: Log is configured to *not* fsync() on all Append() calls
I20250625 20:04:12.842016 12460 tablet_bootstrap.cc:492] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166: No bootstrap required, opened a new log
I20250625 20:04:12.842500 12460 ts_tablet_manager.cc:1397] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
I20250625 20:04:12.860535 12460 raft_consensus.cc:357] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } }
I20250625 20:04:12.861148 12460 raft_consensus.cc:383] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:04:12.861400 12460 raft_consensus.cc:738] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44f691d035604bcab46e9120f6028166, State: Initialized, Role: FOLLOWER
I20250625 20:04:12.862139 12460 consensus_queue.cc:260] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } }
I20250625 20:04:12.862680 12460 raft_consensus.cc:397] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 20:04:12.862993 12460 raft_consensus.cc:491] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 20:04:12.863317 12460 raft_consensus.cc:3058] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:04:12.867738 12460 raft_consensus.cc:513] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } }
I20250625 20:04:12.868543 12460 leader_election.cc:304] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44f691d035604bcab46e9120f6028166; no voters:
I20250625 20:04:12.870955 12460 leader_election.cc:290] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 20:04:12.871356 12462 raft_consensus.cc:2802] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:04:12.873762 12462 raft_consensus.cc:695] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [term 1 LEADER]: Becoming Leader. State: Replica: 44f691d035604bcab46e9120f6028166, State: Running, Role: LEADER
I20250625 20:04:12.874660 12462 consensus_queue.cc:237] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } }
I20250625 20:04:12.875959 12460 ts_tablet_manager.cc:1428] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166: Time spent starting tablet: real 0.033s user 0.032s sys 0.003s
I20250625 20:04:12.887938 11986 catalog_manager.cc:5582] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44f691d035604bcab46e9120f6028166 (127.9.30.65). New cstate: current_term: 1 leader_uuid: "44f691d035604bcab46e9120f6028166" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } health_report { overall_health: HEALTHY } } }
W20250625 20:04:13.004837 12452 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.9.30.126:39493, user_credentials={real_user=slave}} blocked reactor thread for 44850.8us
I20250625 20:04:13.108186 9337 test_util.cc:276] Using random seed: -219811910
I20250625 20:04:13.130180 11980 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43854:
name: "TestTable1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:04:13.159827 12375 tablet_service.cc:1468] Processing CreateTablet for tablet 15215d7e76d34b10b8b1f4f895db4204 (DEFAULT_TABLE table=TestTable1 [id=c3fa97ce44324402964bc935af3c07b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:04:13.161388 12375 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15215d7e76d34b10b8b1f4f895db4204. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:04:13.182677 12480 tablet_bootstrap.cc:492] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd: Bootstrap starting.
I20250625 20:04:13.188033 12480 tablet_bootstrap.cc:654] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd: Neither blocks nor log segments found. Creating new log.
I20250625 20:04:13.189785 12480 log.cc:826] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd: Log is configured to *not* fsync() on all Append() calls
I20250625 20:04:13.193984 12480 tablet_bootstrap.cc:492] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd: No bootstrap required, opened a new log
I20250625 20:04:13.194468 12480 ts_tablet_manager.cc:1397] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20250625 20:04:13.213236 12480 raft_consensus.cc:357] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } }
I20250625 20:04:13.214025 12480 raft_consensus.cc:383] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:04:13.214324 12480 raft_consensus.cc:738] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df7ba59f0fd749819107c7ef7cff68cd, State: Initialized, Role: FOLLOWER
I20250625 20:04:13.215217 12480 consensus_queue.cc:260] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } }
I20250625 20:04:13.215994 12480 raft_consensus.cc:397] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 20:04:13.216328 12480 raft_consensus.cc:491] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 20:04:13.216719 12480 raft_consensus.cc:3058] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:04:13.223249 12480 raft_consensus.cc:513] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } }
I20250625 20:04:13.224256 12480 leader_election.cc:304] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df7ba59f0fd749819107c7ef7cff68cd; no voters:
I20250625 20:04:13.226426 12480 leader_election.cc:290] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 20:04:13.226907 12482 raft_consensus.cc:2802] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:04:13.229166 12482 raft_consensus.cc:695] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [term 1 LEADER]: Becoming Leader. State: Replica: df7ba59f0fd749819107c7ef7cff68cd, State: Running, Role: LEADER
I20250625 20:04:13.229867 12440 heartbeater.cc:499] Master 127.9.30.126:39493 was elected leader, sending a full tablet report...
I20250625 20:04:13.230095 12482 consensus_queue.cc:237] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } }
I20250625 20:04:13.232421 12480 ts_tablet_manager.cc:1428] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd: Time spent starting tablet: real 0.038s user 0.031s sys 0.007s
I20250625 20:04:13.242502 11980 catalog_manager.cc:5582] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd reported cstate change: term changed from 0 to 1, leader changed from <none> to df7ba59f0fd749819107c7ef7cff68cd (127.9.30.67). New cstate: current_term: 1 leader_uuid: "df7ba59f0fd749819107c7ef7cff68cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } health_report { overall_health: HEALTHY } } }
I20250625 20:04:13.461042 9337 test_util.cc:276] Using random seed: -219459051
I20250625 20:04:13.482161 11979 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43868:
name: "TestTable2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 20:04:13.512306 12242 tablet_service.cc:1468] Processing CreateTablet for tablet 148d21f4b42e42d69b83bd9becad95e0 (DEFAULT_TABLE table=TestTable2 [id=988512eb9b3f4c649f1e7277df21a79b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:04:13.513938 12242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 148d21f4b42e42d69b83bd9becad95e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:04:13.533260 12501 tablet_bootstrap.cc:492] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d: Bootstrap starting.
I20250625 20:04:13.539784 12501 tablet_bootstrap.cc:654] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d: Neither blocks nor log segments found. Creating new log.
I20250625 20:04:13.541654 12501 log.cc:826] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d: Log is configured to *not* fsync() on all Append() calls
I20250625 20:04:13.546298 12501 tablet_bootstrap.cc:492] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d: No bootstrap required, opened a new log
I20250625 20:04:13.546808 12501 ts_tablet_manager.cc:1397] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20250625 20:04:13.564947 12501 raft_consensus.cc:357] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } }
I20250625 20:04:13.565536 12501 raft_consensus.cc:383] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:04:13.565804 12501 raft_consensus.cc:738] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 526f4e745a694c10a4403c8e8de7b98d, State: Initialized, Role: FOLLOWER
I20250625 20:04:13.566457 12501 consensus_queue.cc:260] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } }
I20250625 20:04:13.566972 12501 raft_consensus.cc:397] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 20:04:13.567239 12501 raft_consensus.cc:491] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 20:04:13.567528 12501 raft_consensus.cc:3058] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:04:13.571821 12501 raft_consensus.cc:513] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } }
I20250625 20:04:13.572543 12501 leader_election.cc:304] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 526f4e745a694c10a4403c8e8de7b98d; no voters:
I20250625 20:04:13.574396 12501 leader_election.cc:290] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 20:04:13.574733 12503 raft_consensus.cc:2802] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:04:13.577220 12503 raft_consensus.cc:695] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d [term 1 LEADER]: Becoming Leader. State: Replica: 526f4e745a694c10a4403c8e8de7b98d, State: Running, Role: LEADER
I20250625 20:04:13.578431 12501 ts_tablet_manager.cc:1428] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d: Time spent starting tablet: real 0.031s user 0.018s sys 0.015s
I20250625 20:04:13.578682 12503 consensus_queue.cc:237] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } }
I20250625 20:04:13.589593 11979 catalog_manager.cc:5582] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d reported cstate change: term changed from 0 to 1, leader changed from <none> to 526f4e745a694c10a4403c8e8de7b98d (127.9.30.66). New cstate: current_term: 1 leader_uuid: "526f4e745a694c10a4403c8e8de7b98d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } health_report { overall_health: HEALTHY } } }
I20250625 20:04:13.816931 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 11953
W20250625 20:04:13.903673 12174 heartbeater.cc:646] Failed to heartbeat to 127.9.30.126:39493 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.30.126:39493: connect: Connection refused (error 111)
W20250625 20:04:14.264062 12440 heartbeater.cc:646] Failed to heartbeat to 127.9.30.126:39493 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.30.126:39493: connect: Connection refused (error 111)
W20250625 20:04:14.606148 12307 heartbeater.cc:646] Failed to heartbeat to 127.9.30.126:39493 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.30.126:39493: connect: Connection refused (error 111)
I20250625 20:04:18.438766 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 12045
I20250625 20:04:18.457937 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 12178
I20250625 20:04:18.480571 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 12311
I20250625 20:04:18.501928 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.30.126:39493
--webserver_interface=127.9.30.126
--webserver_port=43167
--builtin_ntp_servers=127.9.30.84:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.30.126:39493 with env {}
W20250625 20:04:18.794777 12580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:04:18.795380 12580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:04:18.795868 12580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:04:18.827287 12580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 20:04:18.827631 12580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:04:18.827888 12580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 20:04:18.828125 12580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 20:04:18.863982 12580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.30.126:39493
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.30.126:39493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.9.30.126
--webserver_port=43167
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:04:18.865350 12580 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:04:18.867018 12580 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:04:18.880986 12586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:04:18.882030 12587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:04:18.882828 12580 server_base.cc:1048] running on GCE node
W20250625 20:04:18.883216 12589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:04:19.976413 12580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:04:19.979012 12580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:04:19.980379 12580 hybrid_clock.cc:648] HybridClock initialized: now 1750881859980341 us; error 58 us; skew 500 ppm
I20250625 20:04:19.981170 12580 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:04:19.991415 12580 webserver.cc:469] Webserver started at http://127.9.30.126:43167/ using document root <none> and password file <none>
I20250625 20:04:19.992444 12580 fs_manager.cc:362] Metadata directory not provided
I20250625 20:04:19.992671 12580 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:04:20.000661 12580 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.002s
I20250625 20:04:20.005362 12596 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:04:20.006469 12580 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250625 20:04:20.006803 12580 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
uuid: "4d3eec5ea99248df990c8b46a3384c6b"
format_stamp: "Formatted at 2025-06-25 20:04:07 on dist-test-slave-jhxp"
I20250625 20:04:20.008792 12580 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:04:20.059176 12580 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:04:20.060727 12580 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:04:20.061165 12580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:04:20.129679 12580 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.126:39493
I20250625 20:04:20.129747 12647 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.126:39493 every 8 connection(s)
I20250625 20:04:20.132602 12580 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250625 20:04:20.142347 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 12580
I20250625 20:04:20.142931 12648 sys_catalog.cc:263] Verifying existing consensus state
I20250625 20:04:20.143818 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.30.65:44285
--local_ip_for_outbound_sockets=127.9.30.65
--tserver_master_addrs=127.9.30.126:39493
--webserver_port=33919
--webserver_interface=127.9.30.65
--builtin_ntp_servers=127.9.30.84:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 20:04:20.147929 12648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b: Bootstrap starting.
I20250625 20:04:20.160291 12648 log.cc:826] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b: Log is configured to *not* fsync() on all Append() calls
I20250625 20:04:20.228297 12648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b: Bootstrap replayed 1/1 log segments. Stats: ops{read=19 overwritten=0 applied=19 ignored=0} inserts{seen=13 ignored=0} mutations{seen=11 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 20:04:20.229142 12648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b: Bootstrap complete.
I20250625 20:04:20.250234 12648 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3eec5ea99248df990c8b46a3384c6b" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 39493 } }
I20250625 20:04:20.252449 12648 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3eec5ea99248df990c8b46a3384c6b, State: Initialized, Role: FOLLOWER
I20250625 20:04:20.253204 12648 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 19, Last appended: 2.19, Last appended by leader: 19, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3eec5ea99248df990c8b46a3384c6b" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 39493 } }
I20250625 20:04:20.253744 12648 raft_consensus.cc:397] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 20:04:20.253997 12648 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 20:04:20.254292 12648 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [term 2 FOLLOWER]: Advancing to term 3
I20250625 20:04:20.260074 12648 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3eec5ea99248df990c8b46a3384c6b" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 39493 } }
I20250625 20:04:20.260845 12648 leader_election.cc:304] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d3eec5ea99248df990c8b46a3384c6b; no voters:
I20250625 20:04:20.262805 12648 leader_election.cc:290] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [CANDIDATE]: Term 3 election: Requested vote from peers
I20250625 20:04:20.263370 12652 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [term 3 FOLLOWER]: Leader election won for term 3
I20250625 20:04:20.267189 12648 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [sys.catalog]: configured and running, proceeding with master startup.
I20250625 20:04:20.271132 12652 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [term 3 LEADER]: Becoming Leader. State: Replica: 4d3eec5ea99248df990c8b46a3384c6b, State: Running, Role: LEADER
I20250625 20:04:20.272322 12652 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 2.19, Last appended by leader: 19, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3eec5ea99248df990c8b46a3384c6b" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 39493 } }
I20250625 20:04:20.284036 12654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d3eec5ea99248df990c8b46a3384c6b. Latest consensus state: current_term: 3 leader_uuid: "4d3eec5ea99248df990c8b46a3384c6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3eec5ea99248df990c8b46a3384c6b" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 39493 } } }
I20250625 20:04:20.284550 12653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "4d3eec5ea99248df990c8b46a3384c6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3eec5ea99248df990c8b46a3384c6b" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 39493 } } }
I20250625 20:04:20.285487 12654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [sys.catalog]: This master's current role is: LEADER
I20250625 20:04:20.285537 12653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [sys.catalog]: This master's current role is: LEADER
I20250625 20:04:20.300056 12661 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 20:04:20.312253 12661 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=988512eb9b3f4c649f1e7277df21a79b]
I20250625 20:04:20.313990 12661 catalog_manager.cc:671] Loaded metadata for table TestTable [id=aa2216c1b076490897ce03a0ba97ae25]
I20250625 20:04:20.315640 12661 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=db0e8ff6e0f74f769bed2e7f671276aa]
I20250625 20:04:20.323244 12661 tablet_loader.cc:96] loaded metadata for tablet 148d21f4b42e42d69b83bd9becad95e0 (table TestTable2 [id=988512eb9b3f4c649f1e7277df21a79b])
I20250625 20:04:20.324910 12661 tablet_loader.cc:96] loaded metadata for tablet 15215d7e76d34b10b8b1f4f895db4204 (table TestTable1 [id=db0e8ff6e0f74f769bed2e7f671276aa])
I20250625 20:04:20.326200 12661 tablet_loader.cc:96] loaded metadata for tablet bd510751b1cb4241a9f5541b94098e8c (table TestTable [id=aa2216c1b076490897ce03a0ba97ae25])
I20250625 20:04:20.327531 12661 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 20:04:20.332619 12661 catalog_manager.cc:1261] Loaded cluster ID: de233e8fc4a34d7db5e31c956aad7d4b
I20250625 20:04:20.333029 12661 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 20:04:20.341637 12661 catalog_manager.cc:1506] Loading token signing keys...
I20250625 20:04:20.346876 12661 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b: Loaded TSK: 0
I20250625 20:04:20.348547 12661 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250625 20:04:20.499362 12650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:04:20.499938 12650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:04:20.500455 12650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:04:20.533478 12650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:04:20.534392 12650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.30.65
I20250625 20:04:20.571048 12650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.30.65:44285
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.9.30.65
--webserver_port=33919
--tserver_master_addrs=127.9.30.126:39493
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.30.65
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:04:20.572455 12650 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:04:20.574226 12650 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:04:20.592056 12676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:04:20.592657 12675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:04:20.592047 12678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:04:21.677639 12677 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250625 20:04:21.677762 12650 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 20:04:21.682065 12650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:04:21.684801 12650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:04:21.686259 12650 hybrid_clock.cc:648] HybridClock initialized: now 1750881861686196 us; error 87 us; skew 500 ppm
I20250625 20:04:21.687112 12650 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:04:21.694195 12650 webserver.cc:469] Webserver started at http://127.9.30.65:33919/ using document root <none> and password file <none>
I20250625 20:04:21.695178 12650 fs_manager.cc:362] Metadata directory not provided
I20250625 20:04:21.695417 12650 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:04:21.703860 12650 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20250625 20:04:21.709039 12685 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:04:21.710201 12650 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250625 20:04:21.710548 12650 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "44f691d035604bcab46e9120f6028166"
format_stamp: "Formatted at 2025-06-25 20:04:08 on dist-test-slave-jhxp"
I20250625 20:04:21.712539 12650 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:04:21.778792 12650 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:04:21.780426 12650 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:04:21.780872 12650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:04:21.783819 12650 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 20:04:21.790081 12692 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250625 20:04:21.798008 12650 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250625 20:04:21.798259 12650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.002s
I20250625 20:04:21.798545 12650 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250625 20:04:21.803627 12650 ts_tablet_manager.cc:610] Registered 1 tablets
I20250625 20:04:21.803896 12650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.001s sys 0.005s
I20250625 20:04:21.804307 12692 tablet_bootstrap.cc:492] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166: Bootstrap starting.
I20250625 20:04:21.876245 12692 log.cc:826] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166: Log is configured to *not* fsync() on all Append() calls
I20250625 20:04:21.987221 12692 tablet_bootstrap.cc:492] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 20:04:21.988361 12692 tablet_bootstrap.cc:492] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166: Bootstrap complete.
I20250625 20:04:21.990209 12692 ts_tablet_manager.cc:1397] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166: Time spent bootstrapping tablet: real 0.186s user 0.142s sys 0.035s
I20250625 20:04:21.996721 12650 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.65:44285
I20250625 20:04:21.996855 12799 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.65:44285 every 8 connection(s)
I20250625 20:04:21.999451 12650 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250625 20:04:22.003962 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 12650
I20250625 20:04:22.005760 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.30.66:41265
--local_ip_for_outbound_sockets=127.9.30.66
--tserver_master_addrs=127.9.30.126:39493
--webserver_port=40759
--webserver_interface=127.9.30.66
--builtin_ntp_servers=127.9.30.84:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 20:04:22.025053 12692 raft_consensus.cc:357] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } }
I20250625 20:04:22.028115 12692 raft_consensus.cc:738] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44f691d035604bcab46e9120f6028166, State: Initialized, Role: FOLLOWER
I20250625 20:04:22.029059 12692 consensus_queue.cc:260] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } }
I20250625 20:04:22.029788 12692 raft_consensus.cc:397] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 20:04:22.030146 12692 raft_consensus.cc:491] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 20:04:22.030526 12692 raft_consensus.cc:3058] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [term 1 FOLLOWER]: Advancing to term 2
I20250625 20:04:22.038646 12692 raft_consensus.cc:513] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } }
I20250625 20:04:22.040457 12692 leader_election.cc:304] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44f691d035604bcab46e9120f6028166; no voters:
I20250625 20:04:22.048640 12805 raft_consensus.cc:2802] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [term 2 FOLLOWER]: Leader election won for term 2
I20250625 20:04:22.049168 12692 leader_election.cc:290] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250625 20:04:22.053998 12692 ts_tablet_manager.cc:1428] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166: Time spent starting tablet: real 0.063s user 0.038s sys 0.007s
I20250625 20:04:22.057592 12805 raft_consensus.cc:695] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [term 2 LEADER]: Becoming Leader. State: Replica: 44f691d035604bcab46e9120f6028166, State: Running, Role: LEADER
I20250625 20:04:22.058609 12805 consensus_queue.cc:237] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } }
I20250625 20:04:22.067368 12800 heartbeater.cc:344] Connected to a master server at 127.9.30.126:39493
I20250625 20:04:22.067939 12800 heartbeater.cc:461] Registering TS with master...
I20250625 20:04:22.069384 12800 heartbeater.cc:507] Master 127.9.30.126:39493 requested a full tablet report, sending...
I20250625 20:04:22.074692 12613 ts_manager.cc:194] Registered new tserver with Master: 44f691d035604bcab46e9120f6028166 (127.9.30.65:44285)
I20250625 20:04:22.079192 12613 catalog_manager.cc:5582] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 reported cstate change: term changed from 0 to 2, leader changed from <none> to 44f691d035604bcab46e9120f6028166 (127.9.30.65), VOTER 44f691d035604bcab46e9120f6028166 (127.9.30.65) added. New cstate: current_term: 2 leader_uuid: "44f691d035604bcab46e9120f6028166" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } health_report { overall_health: HEALTHY } } }
W20250625 20:04:22.130070 12613 catalog_manager.cc:5260] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bd510751b1cb4241a9f5541b94098e8c with cas_config_opid_index -1: no extra replica candidate found for tablet bd510751b1cb4241a9f5541b94098e8c (table TestTable [id=aa2216c1b076490897ce03a0ba97ae25]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (1) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
I20250625 20:04:22.132603 12613 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.30.65:44381
I20250625 20:04:22.137389 12800 heartbeater.cc:499] Master 127.9.30.126:39493 was elected leader, sending a full tablet report...
W20250625 20:04:22.373559 12804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:04:22.374091 12804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:04:22.374593 12804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:04:22.406745 12804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:04:22.407637 12804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.30.66
I20250625 20:04:22.442221 12804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.30.66:41265
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.9.30.66
--webserver_port=40759
--tserver_master_addrs=127.9.30.126:39493
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.30.66
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:04:22.443526 12804 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:04:22.445366 12804 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:04:22.461093 12820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:04:22.461158 12822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:04:22.461086 12819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:04:23.561792 12821 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250625 20:04:23.561908 12804 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 20:04:23.566066 12804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:04:23.568814 12804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:04:23.570290 12804 hybrid_clock.cc:648] HybridClock initialized: now 1750881863570236 us; error 76 us; skew 500 ppm
I20250625 20:04:23.571094 12804 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:04:23.577741 12804 webserver.cc:469] Webserver started at http://127.9.30.66:40759/ using document root <none> and password file <none>
I20250625 20:04:23.578693 12804 fs_manager.cc:362] Metadata directory not provided
I20250625 20:04:23.578910 12804 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:04:23.587173 12804 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20250625 20:04:23.592018 12829 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:04:23.593161 12804 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20250625 20:04:23.593487 12804 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "526f4e745a694c10a4403c8e8de7b98d"
format_stamp: "Formatted at 2025-06-25 20:04:10 on dist-test-slave-jhxp"
I20250625 20:04:23.595465 12804 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:04:23.649551 12804 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:04:23.651036 12804 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:04:23.651453 12804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:04:23.653944 12804 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 20:04:23.660127 12836 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250625 20:04:23.667675 12804 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250625 20:04:23.667910 12804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.002s
I20250625 20:04:23.668145 12804 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250625 20:04:23.672966 12804 ts_tablet_manager.cc:610] Registered 1 tablets
I20250625 20:04:23.673233 12804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.004s sys 0.000s
I20250625 20:04:23.673585 12836 tablet_bootstrap.cc:492] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d: Bootstrap starting.
I20250625 20:04:23.747212 12836 log.cc:826] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d: Log is configured to *not* fsync() on all Append() calls
I20250625 20:04:23.827773 12836 tablet_bootstrap.cc:492] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 20:04:23.828564 12836 tablet_bootstrap.cc:492] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d: Bootstrap complete.
I20250625 20:04:23.829711 12836 ts_tablet_manager.cc:1397] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d: Time spent bootstrapping tablet: real 0.157s user 0.113s sys 0.040s
I20250625 20:04:23.845556 12836 raft_consensus.cc:357] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } }
I20250625 20:04:23.847731 12836 raft_consensus.cc:738] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 526f4e745a694c10a4403c8e8de7b98d, State: Initialized, Role: FOLLOWER
I20250625 20:04:23.849037 12836 consensus_queue.cc:260] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } }
I20250625 20:04:23.849627 12804 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.66:41265
I20250625 20:04:23.849757 12943 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.66:41265 every 8 connection(s)
I20250625 20:04:23.849869 12836 raft_consensus.cc:397] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 20:04:23.850231 12836 raft_consensus.cc:491] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 20:04:23.850518 12836 raft_consensus.cc:3058] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d [term 1 FOLLOWER]: Advancing to term 2
I20250625 20:04:23.852337 12804 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250625 20:04:23.854781 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 12804
I20250625 20:04:23.856693 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.30.67:34021
--local_ip_for_outbound_sockets=127.9.30.67
--tserver_master_addrs=127.9.30.126:39493
--webserver_port=35311
--webserver_interface=127.9.30.67
--builtin_ntp_servers=127.9.30.84:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 20:04:23.857266 12836 raft_consensus.cc:513] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } }
I20250625 20:04:23.858191 12836 leader_election.cc:304] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 526f4e745a694c10a4403c8e8de7b98d; no voters:
I20250625 20:04:23.865621 12836 leader_election.cc:290] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d [CANDIDATE]: Term 2 election: Requested vote from peers
I20250625 20:04:23.866012 12948 raft_consensus.cc:2802] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d [term 2 FOLLOWER]: Leader election won for term 2
I20250625 20:04:23.869840 12948 raft_consensus.cc:695] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d [term 2 LEADER]: Becoming Leader. State: Replica: 526f4e745a694c10a4403c8e8de7b98d, State: Running, Role: LEADER
I20250625 20:04:23.870867 12948 consensus_queue.cc:237] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } }
I20250625 20:04:23.872043 12836 ts_tablet_manager.cc:1428] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d: Time spent starting tablet: real 0.042s user 0.026s sys 0.017s
I20250625 20:04:23.882433 12944 heartbeater.cc:344] Connected to a master server at 127.9.30.126:39493
I20250625 20:04:23.882856 12944 heartbeater.cc:461] Registering TS with master...
I20250625 20:04:23.883868 12944 heartbeater.cc:507] Master 127.9.30.126:39493 requested a full tablet report, sending...
I20250625 20:04:23.887111 12613 ts_manager.cc:194] Registered new tserver with Master: 526f4e745a694c10a4403c8e8de7b98d (127.9.30.66:41265)
I20250625 20:04:23.888123 12613 catalog_manager.cc:5582] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "526f4e745a694c10a4403c8e8de7b98d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } health_report { overall_health: HEALTHY } } }
I20250625 20:04:23.898406 12613 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.30.66:44889
I20250625 20:04:23.901647 12944 heartbeater.cc:499] Master 127.9.30.126:39493 was elected leader, sending a full tablet report...
W20250625 20:04:24.175112 12950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:04:24.175693 12950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:04:24.176215 12950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:04:24.208051 12950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:04:24.209096 12950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.30.67
I20250625 20:04:24.244490 12950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.30.67:34021
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.9.30.67
--webserver_port=35311
--tserver_master_addrs=127.9.30.126:39493
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.30.67
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:04:24.245833 12950 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:04:24.247531 12950 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:04:24.264570 12966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:04:24.447443 12755 consensus_queue.cc:237] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: NON_VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: true } }
I20250625 20:04:24.456290 12973 raft_consensus.cc:2953] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index -1 to 8, NON_VOTER 526f4e745a694c10a4403c8e8de7b98d (127.9.30.66) added. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: NON_VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: true } } }
I20250625 20:04:24.472157 12598 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bd510751b1cb4241a9f5541b94098e8c with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 8)
I20250625 20:04:24.473088 12613 catalog_manager.cc:5582] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 reported cstate change: config changed from index -1 to 8, NON_VOTER 526f4e745a694c10a4403c8e8de7b98d (127.9.30.66) added. New cstate: current_term: 2 leader_uuid: "44f691d035604bcab46e9120f6028166" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: NON_VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250625 20:04:24.485600 12688 consensus_peers.cc:487] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 -> Peer 526f4e745a694c10a4403c8e8de7b98d (127.9.30.66:41265): Couldn't send request to peer 526f4e745a694c10a4403c8e8de7b98d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd510751b1cb4241a9f5541b94098e8c. This is attempt 1: this message will repeat every 5th retry.
I20250625 20:04:24.990520 12981 ts_tablet_manager.cc:927] T bd510751b1cb4241a9f5541b94098e8c P 526f4e745a694c10a4403c8e8de7b98d: Initiating tablet copy from peer 44f691d035604bcab46e9120f6028166 (127.9.30.65:44285)
I20250625 20:04:24.992892 12981 tablet_copy_client.cc:323] T bd510751b1cb4241a9f5541b94098e8c P 526f4e745a694c10a4403c8e8de7b98d: tablet copy: Beginning tablet copy session from remote peer at address 127.9.30.65:44285
I20250625 20:04:25.012876 12775 tablet_copy_service.cc:140] P 44f691d035604bcab46e9120f6028166: Received BeginTabletCopySession request for tablet bd510751b1cb4241a9f5541b94098e8c from peer 526f4e745a694c10a4403c8e8de7b98d ({username='slave'} at 127.9.30.66:56169)
I20250625 20:04:25.013381 12775 tablet_copy_service.cc:161] P 44f691d035604bcab46e9120f6028166: Beginning new tablet copy session on tablet bd510751b1cb4241a9f5541b94098e8c from peer 526f4e745a694c10a4403c8e8de7b98d at {username='slave'} at 127.9.30.66:56169: session id = 526f4e745a694c10a4403c8e8de7b98d-bd510751b1cb4241a9f5541b94098e8c
I20250625 20:04:25.019403 12775 tablet_copy_source_session.cc:215] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166: Tablet Copy: opened 0 blocks and 1 log segments
I20250625 20:04:25.030448 12981 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd510751b1cb4241a9f5541b94098e8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:04:25.044288 12981 tablet_copy_client.cc:806] T bd510751b1cb4241a9f5541b94098e8c P 526f4e745a694c10a4403c8e8de7b98d: tablet copy: Starting download of 0 data blocks...
I20250625 20:04:25.044847 12981 tablet_copy_client.cc:670] T bd510751b1cb4241a9f5541b94098e8c P 526f4e745a694c10a4403c8e8de7b98d: tablet copy: Starting download of 1 WAL segments...
I20250625 20:04:25.048766 12981 tablet_copy_client.cc:538] T bd510751b1cb4241a9f5541b94098e8c P 526f4e745a694c10a4403c8e8de7b98d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250625 20:04:25.054462 12981 tablet_bootstrap.cc:492] T bd510751b1cb4241a9f5541b94098e8c P 526f4e745a694c10a4403c8e8de7b98d: Bootstrap starting.
I20250625 20:04:25.127554 12981 tablet_bootstrap.cc:492] T bd510751b1cb4241a9f5541b94098e8c P 526f4e745a694c10a4403c8e8de7b98d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 20:04:25.128304 12981 tablet_bootstrap.cc:492] T bd510751b1cb4241a9f5541b94098e8c P 526f4e745a694c10a4403c8e8de7b98d: Bootstrap complete.
I20250625 20:04:25.128779 12981 ts_tablet_manager.cc:1397] T bd510751b1cb4241a9f5541b94098e8c P 526f4e745a694c10a4403c8e8de7b98d: Time spent bootstrapping tablet: real 0.074s user 0.064s sys 0.012s
I20250625 20:04:25.130513 12981 raft_consensus.cc:357] T bd510751b1cb4241a9f5541b94098e8c P 526f4e745a694c10a4403c8e8de7b98d [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: NON_VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: true } }
I20250625 20:04:25.130957 12981 raft_consensus.cc:738] T bd510751b1cb4241a9f5541b94098e8c P 526f4e745a694c10a4403c8e8de7b98d [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 526f4e745a694c10a4403c8e8de7b98d, State: Initialized, Role: LEARNER
I20250625 20:04:25.131421 12981 consensus_queue.cc:260] T bd510751b1cb4241a9f5541b94098e8c P 526f4e745a694c10a4403c8e8de7b98d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: NON_VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: true } }
I20250625 20:04:25.134442 12981 ts_tablet_manager.cc:1428] T bd510751b1cb4241a9f5541b94098e8c P 526f4e745a694c10a4403c8e8de7b98d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250625 20:04:25.136567 12775 tablet_copy_service.cc:342] P 44f691d035604bcab46e9120f6028166: Request end of tablet copy session 526f4e745a694c10a4403c8e8de7b98d-bd510751b1cb4241a9f5541b94098e8c received from {username='slave'} at 127.9.30.66:56169
I20250625 20:04:25.137040 12775 tablet_copy_service.cc:434] P 44f691d035604bcab46e9120f6028166: ending tablet copy session 526f4e745a694c10a4403c8e8de7b98d-bd510751b1cb4241a9f5541b94098e8c on tablet bd510751b1cb4241a9f5541b94098e8c with peer 526f4e745a694c10a4403c8e8de7b98d
W20250625 20:04:24.264914 12964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:04:24.266614 12950 server_base.cc:1048] running on GCE node
W20250625 20:04:24.265200 12963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:04:25.386111 12950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:04:25.388422 12950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:04:25.389842 12950 hybrid_clock.cc:648] HybridClock initialized: now 1750881865389789 us; error 68 us; skew 500 ppm
I20250625 20:04:25.390678 12950 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:04:25.399727 12950 webserver.cc:469] Webserver started at http://127.9.30.67:35311/ using document root <none> and password file <none>
I20250625 20:04:25.400729 12950 fs_manager.cc:362] Metadata directory not provided
I20250625 20:04:25.400979 12950 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:04:25.409358 12950 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20250625 20:04:25.414027 12991 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:04:25.415130 12950 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250625 20:04:25.415473 12950 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "df7ba59f0fd749819107c7ef7cff68cd"
format_stamp: "Formatted at 2025-06-25 20:04:12 on dist-test-slave-jhxp"
I20250625 20:04:25.417497 12950 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:04:25.522454 12950 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:04:25.524269 12950 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:04:25.524760 12950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:04:25.527284 12950 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 20:04:25.533017 12998 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250625 20:04:25.540910 12950 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250625 20:04:25.541217 12950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.002s sys 0.000s
I20250625 20:04:25.541507 12950 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250625 20:04:25.546629 12950 ts_tablet_manager.cc:610] Registered 1 tablets
I20250625 20:04:25.546847 12950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.003s sys 0.000s
I20250625 20:04:25.547260 12998 tablet_bootstrap.cc:492] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd: Bootstrap starting.
I20250625 20:04:25.567162 12899 raft_consensus.cc:1215] T bd510751b1cb4241a9f5541b94098e8c P 526f4e745a694c10a4403c8e8de7b98d [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20250625 20:04:25.611919 12998 log.cc:826] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd: Log is configured to *not* fsync() on all Append() calls
I20250625 20:04:25.689303 12998 tablet_bootstrap.cc:492] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 20:04:25.690533 12998 tablet_bootstrap.cc:492] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd: Bootstrap complete.
I20250625 20:04:25.692215 12998 ts_tablet_manager.cc:1397] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd: Time spent bootstrapping tablet: real 0.145s user 0.110s sys 0.031s
I20250625 20:04:25.707837 12998 raft_consensus.cc:357] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } }
I20250625 20:04:25.710472 12998 raft_consensus.cc:738] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: df7ba59f0fd749819107c7ef7cff68cd, State: Initialized, Role: FOLLOWER
I20250625 20:04:25.711463 12998 consensus_queue.cc:260] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } }
I20250625 20:04:25.712189 12998 raft_consensus.cc:397] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 20:04:25.712558 12998 raft_consensus.cc:491] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 20:04:25.712929 12998 raft_consensus.cc:3058] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [term 1 FOLLOWER]: Advancing to term 2
I20250625 20:04:25.719573 12998 raft_consensus.cc:513] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } }
I20250625 20:04:25.720335 12998 leader_election.cc:304] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df7ba59f0fd749819107c7ef7cff68cd; no voters:
I20250625 20:04:25.722545 12998 leader_election.cc:290] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [CANDIDATE]: Term 2 election: Requested vote from peers
I20250625 20:04:25.722932 13102 raft_consensus.cc:2802] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [term 2 FOLLOWER]: Leader election won for term 2
I20250625 20:04:25.726476 12998 ts_tablet_manager.cc:1428] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd: Time spent starting tablet: real 0.034s user 0.022s sys 0.010s
I20250625 20:04:25.726547 13102 raft_consensus.cc:695] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [term 2 LEADER]: Becoming Leader. State: Replica: df7ba59f0fd749819107c7ef7cff68cd, State: Running, Role: LEADER
I20250625 20:04:25.727670 13102 consensus_queue.cc:237] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } }
I20250625 20:04:25.737581 12950 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.67:34021
I20250625 20:04:25.738111 13114 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.67:34021 every 8 connection(s)
I20250625 20:04:25.740146 12950 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250625 20:04:25.743180 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 12950
I20250625 20:04:25.759929 13115 heartbeater.cc:344] Connected to a master server at 127.9.30.126:39493
I20250625 20:04:25.760356 13115 heartbeater.cc:461] Registering TS with master...
I20250625 20:04:25.761443 13115 heartbeater.cc:507] Master 127.9.30.126:39493 requested a full tablet report, sending...
I20250625 20:04:25.764780 12613 ts_manager.cc:194] Registered new tserver with Master: df7ba59f0fd749819107c7ef7cff68cd (127.9.30.67:34021)
I20250625 20:04:25.765756 12613 catalog_manager.cc:5582] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd reported cstate change: term changed from 0 to 2, leader changed from <none> to df7ba59f0fd749819107c7ef7cff68cd (127.9.30.67), VOTER df7ba59f0fd749819107c7ef7cff68cd (127.9.30.67) added. New cstate: current_term: 2 leader_uuid: "df7ba59f0fd749819107c7ef7cff68cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } health_report { overall_health: HEALTHY } } }
I20250625 20:04:25.777422 9337 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 20:04:25.778673 12613 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.30.67:45495
I20250625 20:04:25.783233 9337 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 20:04:25.783468 13115 heartbeater.cc:499] Master 127.9.30.126:39493 was elected leader, sending a full tablet report...
W20250625 20:04:25.788283 9337 ts_itest-base.cc:209] found only 2 out of 3 replicas of tablet bd510751b1cb4241a9f5541b94098e8c: tablet_id: "bd510751b1cb4241a9f5541b94098e8c" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER } interned_replicas { ts_info_idx: 1 role: LEARNER }
I20250625 20:04:25.798063 13065 consensus_queue.cc:237] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } } peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: NON_VOTER last_known_addr { host: "127.9.30.65" port: 44285 } attrs { promote: true } }
I20250625 20:04:25.801542 13105 raft_consensus.cc:2953] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index -1 to 8, NON_VOTER 44f691d035604bcab46e9120f6028166 (127.9.30.65) added. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } } peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: NON_VOTER last_known_addr { host: "127.9.30.65" port: 44285 } attrs { promote: true } } }
I20250625 20:04:25.810648 12600 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 15215d7e76d34b10b8b1f4f895db4204 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250625 20:04:25.812923 12993 consensus_peers.cc:487] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd -> Peer 44f691d035604bcab46e9120f6028166 (127.9.30.65:44285): Couldn't send request to peer 44f691d035604bcab46e9120f6028166. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15215d7e76d34b10b8b1f4f895db4204. This is attempt 1: this message will repeat every 5th retry.
I20250625 20:04:25.813648 12611 catalog_manager.cc:5582] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd reported cstate change: config changed from index -1 to 8, NON_VOTER 44f691d035604bcab46e9120f6028166 (127.9.30.65) added. New cstate: current_term: 2 leader_uuid: "df7ba59f0fd749819107c7ef7cff68cd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: NON_VOTER last_known_addr { host: "127.9.30.65" port: 44285 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250625 20:04:25.823045 13065 consensus_queue.cc:237] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } } peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: NON_VOTER last_known_addr { host: "127.9.30.65" port: 44285 } attrs { promote: true } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: NON_VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: true } }
I20250625 20:04:25.826211 13102 raft_consensus.cc:2953] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 526f4e745a694c10a4403c8e8de7b98d (127.9.30.66) added. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } } peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: NON_VOTER last_known_addr { host: "127.9.30.65" port: 44285 } attrs { promote: true } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: NON_VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: true } } }
W20250625 20:04:25.828018 12993 consensus_peers.cc:487] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd -> Peer 44f691d035604bcab46e9120f6028166 (127.9.30.65:44285): Couldn't send request to peer 44f691d035604bcab46e9120f6028166. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15215d7e76d34b10b8b1f4f895db4204. This is attempt 1: this message will repeat every 5th retry.
I20250625 20:04:25.833137 12600 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 15215d7e76d34b10b8b1f4f895db4204 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250625 20:04:25.837333 12994 consensus_peers.cc:487] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd -> Peer 526f4e745a694c10a4403c8e8de7b98d (127.9.30.66:41265): Couldn't send request to peer 526f4e745a694c10a4403c8e8de7b98d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15215d7e76d34b10b8b1f4f895db4204. This is attempt 1: this message will repeat every 5th retry.
I20250625 20:04:25.836920 12611 catalog_manager.cc:5582] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd reported cstate change: config changed from index 8 to 9, NON_VOTER 526f4e745a694c10a4403c8e8de7b98d (127.9.30.66) added. New cstate: current_term: 2 leader_uuid: "df7ba59f0fd749819107c7ef7cff68cd" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: NON_VOTER last_known_addr { host: "127.9.30.65" port: 44285 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: NON_VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250625 20:04:25.968672 13007 raft_consensus.cc:1062] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166: attempting to promote NON_VOTER 526f4e745a694c10a4403c8e8de7b98d to VOTER
I20250625 20:04:25.970130 13007 consensus_queue.cc:237] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } }
I20250625 20:04:25.975235 12899 raft_consensus.cc:1273] T bd510751b1cb4241a9f5541b94098e8c P 526f4e745a694c10a4403c8e8de7b98d [term 2 LEARNER]: Refusing update from remote peer 44f691d035604bcab46e9120f6028166: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20250625 20:04:25.976541 13008 consensus_queue.cc:1035] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250625 20:04:25.982996 13007 raft_consensus.cc:2953] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 526f4e745a694c10a4403c8e8de7b98d (127.9.30.66) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } } }
I20250625 20:04:25.984624 12899 raft_consensus.cc:2953] T bd510751b1cb4241a9f5541b94098e8c P 526f4e745a694c10a4403c8e8de7b98d [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 526f4e745a694c10a4403c8e8de7b98d (127.9.30.66) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } } }
I20250625 20:04:25.994940 12611 catalog_manager.cc:5582] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 reported cstate change: config changed from index 8 to 9, 526f4e745a694c10a4403c8e8de7b98d (127.9.30.66) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "44f691d035604bcab46e9120f6028166" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250625 20:04:26.006629 12755 consensus_queue.cc:237] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } } peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: NON_VOTER last_known_addr { host: "127.9.30.67" port: 34021 } attrs { promote: true } }
I20250625 20:04:26.012503 12899 raft_consensus.cc:1273] T bd510751b1cb4241a9f5541b94098e8c P 526f4e745a694c10a4403c8e8de7b98d [term 2 FOLLOWER]: Refusing update from remote peer 44f691d035604bcab46e9120f6028166: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250625 20:04:26.013625 13008 consensus_queue.cc:1035] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250625 20:04:26.020987 13005 raft_consensus.cc:2953] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, NON_VOTER df7ba59f0fd749819107c7ef7cff68cd (127.9.30.67) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } } peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: NON_VOTER last_known_addr { host: "127.9.30.67" port: 34021 } attrs { promote: true } } }
W20250625 20:04:26.022588 12689 consensus_peers.cc:487] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 -> Peer df7ba59f0fd749819107c7ef7cff68cd (127.9.30.67:34021): Couldn't send request to peer df7ba59f0fd749819107c7ef7cff68cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd510751b1cb4241a9f5541b94098e8c. This is attempt 1: this message will repeat every 5th retry.
I20250625 20:04:26.022619 12899 raft_consensus.cc:2953] T bd510751b1cb4241a9f5541b94098e8c P 526f4e745a694c10a4403c8e8de7b98d [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, NON_VOTER df7ba59f0fd749819107c7ef7cff68cd (127.9.30.67) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } } peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: NON_VOTER last_known_addr { host: "127.9.30.67" port: 34021 } attrs { promote: true } } }
I20250625 20:04:26.031632 12598 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bd510751b1cb4241a9f5541b94098e8c with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250625 20:04:26.034220 12611 catalog_manager.cc:5582] T bd510751b1cb4241a9f5541b94098e8c P 526f4e745a694c10a4403c8e8de7b98d reported cstate change: config changed from index 9 to 10, NON_VOTER df7ba59f0fd749819107c7ef7cff68cd (127.9.30.67) added. New cstate: current_term: 2 leader_uuid: "44f691d035604bcab46e9120f6028166" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } } peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: NON_VOTER last_known_addr { host: "127.9.30.67" port: 34021 } attrs { promote: true } } }
I20250625 20:04:26.272974 13132 ts_tablet_manager.cc:927] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166: Initiating tablet copy from peer df7ba59f0fd749819107c7ef7cff68cd (127.9.30.67:34021)
I20250625 20:04:26.275162 13132 tablet_copy_client.cc:323] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166: tablet copy: Beginning tablet copy session from remote peer at address 127.9.30.67:34021
I20250625 20:04:26.276741 13085 tablet_copy_service.cc:140] P df7ba59f0fd749819107c7ef7cff68cd: Received BeginTabletCopySession request for tablet 15215d7e76d34b10b8b1f4f895db4204 from peer 44f691d035604bcab46e9120f6028166 ({username='slave'} at 127.9.30.65:50855)
I20250625 20:04:26.277209 13085 tablet_copy_service.cc:161] P df7ba59f0fd749819107c7ef7cff68cd: Beginning new tablet copy session on tablet 15215d7e76d34b10b8b1f4f895db4204 from peer 44f691d035604bcab46e9120f6028166 at {username='slave'} at 127.9.30.65:50855: session id = 44f691d035604bcab46e9120f6028166-15215d7e76d34b10b8b1f4f895db4204
I20250625 20:04:26.281713 13085 tablet_copy_source_session.cc:215] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd: Tablet Copy: opened 0 blocks and 1 log segments
I20250625 20:04:26.285179 13132 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15215d7e76d34b10b8b1f4f895db4204. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:04:26.295583 13132 tablet_copy_client.cc:806] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166: tablet copy: Starting download of 0 data blocks...
I20250625 20:04:26.296186 13132 tablet_copy_client.cc:670] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166: tablet copy: Starting download of 1 WAL segments...
I20250625 20:04:26.300101 13132 tablet_copy_client.cc:538] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250625 20:04:26.301828 13134 ts_tablet_manager.cc:927] T 15215d7e76d34b10b8b1f4f895db4204 P 526f4e745a694c10a4403c8e8de7b98d: Initiating tablet copy from peer df7ba59f0fd749819107c7ef7cff68cd (127.9.30.67:34021)
I20250625 20:04:26.303315 13134 tablet_copy_client.cc:323] T 15215d7e76d34b10b8b1f4f895db4204 P 526f4e745a694c10a4403c8e8de7b98d: tablet copy: Beginning tablet copy session from remote peer at address 127.9.30.67:34021
I20250625 20:04:26.308089 13132 tablet_bootstrap.cc:492] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166: Bootstrap starting.
I20250625 20:04:26.314167 13085 tablet_copy_service.cc:140] P df7ba59f0fd749819107c7ef7cff68cd: Received BeginTabletCopySession request for tablet 15215d7e76d34b10b8b1f4f895db4204 from peer 526f4e745a694c10a4403c8e8de7b98d ({username='slave'} at 127.9.30.66:48681)
I20250625 20:04:26.314580 13085 tablet_copy_service.cc:161] P df7ba59f0fd749819107c7ef7cff68cd: Beginning new tablet copy session on tablet 15215d7e76d34b10b8b1f4f895db4204 from peer 526f4e745a694c10a4403c8e8de7b98d at {username='slave'} at 127.9.30.66:48681: session id = 526f4e745a694c10a4403c8e8de7b98d-15215d7e76d34b10b8b1f4f895db4204
I20250625 20:04:26.318995 13085 tablet_copy_source_session.cc:215] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd: Tablet Copy: opened 0 blocks and 1 log segments
I20250625 20:04:26.321597 13134 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15215d7e76d34b10b8b1f4f895db4204. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:04:26.331847 13134 tablet_copy_client.cc:806] T 15215d7e76d34b10b8b1f4f895db4204 P 526f4e745a694c10a4403c8e8de7b98d: tablet copy: Starting download of 0 data blocks...
I20250625 20:04:26.332366 13134 tablet_copy_client.cc:670] T 15215d7e76d34b10b8b1f4f895db4204 P 526f4e745a694c10a4403c8e8de7b98d: tablet copy: Starting download of 1 WAL segments...
I20250625 20:04:26.336318 13134 tablet_copy_client.cc:538] T 15215d7e76d34b10b8b1f4f895db4204 P 526f4e745a694c10a4403c8e8de7b98d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250625 20:04:26.342110 13134 tablet_bootstrap.cc:492] T 15215d7e76d34b10b8b1f4f895db4204 P 526f4e745a694c10a4403c8e8de7b98d: Bootstrap starting.
I20250625 20:04:26.385016 13132 tablet_bootstrap.cc:492] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 20:04:26.385684 13132 tablet_bootstrap.cc:492] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166: Bootstrap complete.
I20250625 20:04:26.386144 13132 ts_tablet_manager.cc:1397] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166: Time spent bootstrapping tablet: real 0.078s user 0.067s sys 0.009s
I20250625 20:04:26.387873 13132 raft_consensus.cc:357] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } } peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: NON_VOTER last_known_addr { host: "127.9.30.65" port: 44285 } attrs { promote: true } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: NON_VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: true } }
I20250625 20:04:26.388355 13132 raft_consensus.cc:738] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 44f691d035604bcab46e9120f6028166, State: Initialized, Role: LEARNER
I20250625 20:04:26.388828 13132 consensus_queue.cc:260] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } } peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: NON_VOTER last_known_addr { host: "127.9.30.65" port: 44285 } attrs { promote: true } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: NON_VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: true } }
I20250625 20:04:26.390450 13132 ts_tablet_manager.cc:1428] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20250625 20:04:26.392326 13085 tablet_copy_service.cc:342] P df7ba59f0fd749819107c7ef7cff68cd: Request end of tablet copy session 44f691d035604bcab46e9120f6028166-15215d7e76d34b10b8b1f4f895db4204 received from {username='slave'} at 127.9.30.65:50855
I20250625 20:04:26.392704 13085 tablet_copy_service.cc:434] P df7ba59f0fd749819107c7ef7cff68cd: ending tablet copy session 44f691d035604bcab46e9120f6028166-15215d7e76d34b10b8b1f4f895db4204 on tablet 15215d7e76d34b10b8b1f4f895db4204 with peer 44f691d035604bcab46e9120f6028166
I20250625 20:04:26.419025 13134 tablet_bootstrap.cc:492] T 15215d7e76d34b10b8b1f4f895db4204 P 526f4e745a694c10a4403c8e8de7b98d: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 20:04:26.419710 13134 tablet_bootstrap.cc:492] T 15215d7e76d34b10b8b1f4f895db4204 P 526f4e745a694c10a4403c8e8de7b98d: Bootstrap complete.
I20250625 20:04:26.420158 13134 ts_tablet_manager.cc:1397] T 15215d7e76d34b10b8b1f4f895db4204 P 526f4e745a694c10a4403c8e8de7b98d: Time spent bootstrapping tablet: real 0.078s user 0.063s sys 0.016s
I20250625 20:04:26.421687 13134 raft_consensus.cc:357] T 15215d7e76d34b10b8b1f4f895db4204 P 526f4e745a694c10a4403c8e8de7b98d [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } } peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: NON_VOTER last_known_addr { host: "127.9.30.65" port: 44285 } attrs { promote: true } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: NON_VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: true } }
I20250625 20:04:26.422139 13134 raft_consensus.cc:738] T 15215d7e76d34b10b8b1f4f895db4204 P 526f4e745a694c10a4403c8e8de7b98d [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 526f4e745a694c10a4403c8e8de7b98d, State: Initialized, Role: LEARNER
I20250625 20:04:26.422551 13134 consensus_queue.cc:260] T 15215d7e76d34b10b8b1f4f895db4204 P 526f4e745a694c10a4403c8e8de7b98d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } } peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: NON_VOTER last_known_addr { host: "127.9.30.65" port: 44285 } attrs { promote: true } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: NON_VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: true } }
I20250625 20:04:26.424036 13134 ts_tablet_manager.cc:1428] T 15215d7e76d34b10b8b1f4f895db4204 P 526f4e745a694c10a4403c8e8de7b98d: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250625 20:04:26.425572 13085 tablet_copy_service.cc:342] P df7ba59f0fd749819107c7ef7cff68cd: Request end of tablet copy session 526f4e745a694c10a4403c8e8de7b98d-15215d7e76d34b10b8b1f4f895db4204 received from {username='slave'} at 127.9.30.66:48681
I20250625 20:04:26.425902 13085 tablet_copy_service.cc:434] P df7ba59f0fd749819107c7ef7cff68cd: ending tablet copy session 526f4e745a694c10a4403c8e8de7b98d-15215d7e76d34b10b8b1f4f895db4204 on tablet 15215d7e76d34b10b8b1f4f895db4204 with peer 526f4e745a694c10a4403c8e8de7b98d
I20250625 20:04:26.476620 13139 ts_tablet_manager.cc:927] T bd510751b1cb4241a9f5541b94098e8c P df7ba59f0fd749819107c7ef7cff68cd: Initiating tablet copy from peer 44f691d035604bcab46e9120f6028166 (127.9.30.65:44285)
I20250625 20:04:26.478410 13139 tablet_copy_client.cc:323] T bd510751b1cb4241a9f5541b94098e8c P df7ba59f0fd749819107c7ef7cff68cd: tablet copy: Beginning tablet copy session from remote peer at address 127.9.30.65:44285
I20250625 20:04:26.479895 12775 tablet_copy_service.cc:140] P 44f691d035604bcab46e9120f6028166: Received BeginTabletCopySession request for tablet bd510751b1cb4241a9f5541b94098e8c from peer df7ba59f0fd749819107c7ef7cff68cd ({username='slave'} at 127.9.30.67:49081)
I20250625 20:04:26.480253 12775 tablet_copy_service.cc:161] P 44f691d035604bcab46e9120f6028166: Beginning new tablet copy session on tablet bd510751b1cb4241a9f5541b94098e8c from peer df7ba59f0fd749819107c7ef7cff68cd at {username='slave'} at 127.9.30.67:49081: session id = df7ba59f0fd749819107c7ef7cff68cd-bd510751b1cb4241a9f5541b94098e8c
I20250625 20:04:26.484398 12775 tablet_copy_source_session.cc:215] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166: Tablet Copy: opened 0 blocks and 1 log segments
I20250625 20:04:26.487666 13139 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd510751b1cb4241a9f5541b94098e8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:04:26.500813 13139 tablet_copy_client.cc:806] T bd510751b1cb4241a9f5541b94098e8c P df7ba59f0fd749819107c7ef7cff68cd: tablet copy: Starting download of 0 data blocks...
I20250625 20:04:26.501439 13139 tablet_copy_client.cc:670] T bd510751b1cb4241a9f5541b94098e8c P df7ba59f0fd749819107c7ef7cff68cd: tablet copy: Starting download of 1 WAL segments...
I20250625 20:04:26.505303 13139 tablet_copy_client.cc:538] T bd510751b1cb4241a9f5541b94098e8c P df7ba59f0fd749819107c7ef7cff68cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250625 20:04:26.511178 13139 tablet_bootstrap.cc:492] T bd510751b1cb4241a9f5541b94098e8c P df7ba59f0fd749819107c7ef7cff68cd: Bootstrap starting.
I20250625 20:04:26.585464 13139 tablet_bootstrap.cc:492] T bd510751b1cb4241a9f5541b94098e8c P df7ba59f0fd749819107c7ef7cff68cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 20:04:26.586081 13139 tablet_bootstrap.cc:492] T bd510751b1cb4241a9f5541b94098e8c P df7ba59f0fd749819107c7ef7cff68cd: Bootstrap complete.
I20250625 20:04:26.586531 13139 ts_tablet_manager.cc:1397] T bd510751b1cb4241a9f5541b94098e8c P df7ba59f0fd749819107c7ef7cff68cd: Time spent bootstrapping tablet: real 0.076s user 0.060s sys 0.016s
I20250625 20:04:26.588233 13139 raft_consensus.cc:357] T bd510751b1cb4241a9f5541b94098e8c P df7ba59f0fd749819107c7ef7cff68cd [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } } peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: NON_VOTER last_known_addr { host: "127.9.30.67" port: 34021 } attrs { promote: true } }
I20250625 20:04:26.588704 13139 raft_consensus.cc:738] T bd510751b1cb4241a9f5541b94098e8c P df7ba59f0fd749819107c7ef7cff68cd [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: df7ba59f0fd749819107c7ef7cff68cd, State: Initialized, Role: LEARNER
I20250625 20:04:26.589162 13139 consensus_queue.cc:260] T bd510751b1cb4241a9f5541b94098e8c P df7ba59f0fd749819107c7ef7cff68cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } } peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: NON_VOTER last_known_addr { host: "127.9.30.67" port: 34021 } attrs { promote: true } }
I20250625 20:04:26.590754 13139 ts_tablet_manager.cc:1428] T bd510751b1cb4241a9f5541b94098e8c P df7ba59f0fd749819107c7ef7cff68cd: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250625 20:04:26.592585 12775 tablet_copy_service.cc:342] P 44f691d035604bcab46e9120f6028166: Request end of tablet copy session df7ba59f0fd749819107c7ef7cff68cd-bd510751b1cb4241a9f5541b94098e8c received from {username='slave'} at 127.9.30.67:49081
I20250625 20:04:26.592931 12775 tablet_copy_service.cc:434] P 44f691d035604bcab46e9120f6028166: ending tablet copy session df7ba59f0fd749819107c7ef7cff68cd-bd510751b1cb4241a9f5541b94098e8c on tablet bd510751b1cb4241a9f5541b94098e8c with peer df7ba59f0fd749819107c7ef7cff68cd
I20250625 20:04:26.706518 12599 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bd510751b1cb4241a9f5541b94098e8c with cas_config_opid_index 8: aborting the task: latest config opid_index 10; task opid_index 8
I20250625 20:04:26.737906 12899 raft_consensus.cc:1215] T 15215d7e76d34b10b8b1f4f895db4204 P 526f4e745a694c10a4403c8e8de7b98d [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20250625 20:04:26.792814 9337 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 44f691d035604bcab46e9120f6028166 to finish bootstrapping
I20250625 20:04:26.806034 9337 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 526f4e745a694c10a4403c8e8de7b98d to finish bootstrapping
I20250625 20:04:26.818742 9337 ts_itest-base.cc:246] Waiting for 1 tablets on tserver df7ba59f0fd749819107c7ef7cff68cd to finish bootstrapping
I20250625 20:04:26.827399 12755 raft_consensus.cc:1215] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20250625 20:04:26.972314 13065 raft_consensus.cc:1215] T bd510751b1cb4241a9f5541b94098e8c P df7ba59f0fd749819107c7ef7cff68cd [term 2 LEARNER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20250625 20:04:27.129163 12735 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250625 20:04:27.137476 12878 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250625 20:04:27.138829 13045 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250625 20:04:27.255702 13143 raft_consensus.cc:1062] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd: attempting to promote NON_VOTER 526f4e745a694c10a4403c8e8de7b98d to VOTER
I20250625 20:04:27.257925 13143 consensus_queue.cc:237] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } } peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: NON_VOTER last_known_addr { host: "127.9.30.65" port: 44285 } attrs { promote: true } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } }
I20250625 20:04:27.273914 12899 raft_consensus.cc:1273] T 15215d7e76d34b10b8b1f4f895db4204 P 526f4e745a694c10a4403c8e8de7b98d [term 2 LEARNER]: Refusing update from remote peer df7ba59f0fd749819107c7ef7cff68cd: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250625 20:04:27.275650 13102 consensus_queue.cc:1035] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250625 20:04:27.285331 12755 raft_consensus.cc:1273] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166 [term 2 LEARNER]: Refusing update from remote peer df7ba59f0fd749819107c7ef7cff68cd: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250625 20:04:27.287374 13102 consensus_queue.cc:1035] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: NON_VOTER last_known_addr { host: "127.9.30.65" port: 44285 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250625 20:04:27.295334 13142 raft_consensus.cc:2953] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 526f4e745a694c10a4403c8e8de7b98d (127.9.30.66) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } } peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: NON_VOTER last_known_addr { host: "127.9.30.65" port: 44285 } attrs { promote: true } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } } }
I20250625 20:04:27.304433 12899 raft_consensus.cc:2953] T 15215d7e76d34b10b8b1f4f895db4204 P 526f4e745a694c10a4403c8e8de7b98d [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 526f4e745a694c10a4403c8e8de7b98d (127.9.30.66) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } } peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: NON_VOTER last_known_addr { host: "127.9.30.65" port: 44285 } attrs { promote: true } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } } }
I20250625 20:04:27.315397 12612 catalog_manager.cc:5582] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd reported cstate change: config changed from index 9 to 10, 526f4e745a694c10a4403c8e8de7b98d (127.9.30.66) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "df7ba59f0fd749819107c7ef7cff68cd" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: NON_VOTER last_known_addr { host: "127.9.30.65" port: 44285 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250625 20:04:27.333976 13143 raft_consensus.cc:1062] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd: attempting to promote NON_VOTER 44f691d035604bcab46e9120f6028166 to VOTER
I20250625 20:04:27.335815 13143 consensus_queue.cc:237] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } } peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } attrs { promote: false } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } }
I20250625 20:04:27.335712 12755 raft_consensus.cc:2953] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166 [term 2 LEARNER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 526f4e745a694c10a4403c8e8de7b98d (127.9.30.66) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } } peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: NON_VOTER last_known_addr { host: "127.9.30.65" port: 44285 } attrs { promote: true } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } } }
I20250625 20:04:27.342240 12899 raft_consensus.cc:1273] T 15215d7e76d34b10b8b1f4f895db4204 P 526f4e745a694c10a4403c8e8de7b98d [term 2 FOLLOWER]: Refusing update from remote peer df7ba59f0fd749819107c7ef7cff68cd: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250625 20:04:27.343853 13102 consensus_queue.cc:1035] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250625 20:04:27.351197 13142 raft_consensus.cc:2953] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 44f691d035604bcab46e9120f6028166 (127.9.30.65) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } } peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } attrs { promote: false } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } } }
I20250625 20:04:27.352934 12899 raft_consensus.cc:2953] T 15215d7e76d34b10b8b1f4f895db4204 P 526f4e745a694c10a4403c8e8de7b98d [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 44f691d035604bcab46e9120f6028166 (127.9.30.65) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } } peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } attrs { promote: false } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } } }
I20250625 20:04:27.363399 12755 raft_consensus.cc:1273] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166 [term 2 LEARNER]: Refusing update from remote peer df7ba59f0fd749819107c7ef7cff68cd: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250625 20:04:27.364883 13143 consensus_queue.cc:1035] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250625 20:04:27.369242 12754 raft_consensus.cc:2953] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 44f691d035604bcab46e9120f6028166 (127.9.30.65) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } } peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } attrs { promote: false } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } } }
I20250625 20:04:27.396983 12611 catalog_manager.cc:5582] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd reported cstate change: config changed from index 10 to 11, 44f691d035604bcab46e9120f6028166 (127.9.30.65) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "df7ba59f0fd749819107c7ef7cff68cd" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
4d3eec5ea99248df990c8b46a3384c6b | 127.9.30.126:39493 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.9.30.84:44869 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
44f691d035604bcab46e9120f6028166 | 127.9.30.65:44285 | HEALTHY | <none> | 1 | 0
526f4e745a694c10a4403c8e8de7b98d | 127.9.30.66:41265 | HEALTHY | <none> | 1 | 0
df7ba59f0fd749819107c7ef7cff68cd | 127.9.30.67:34021 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.9.30.65 | experimental | 127.9.30.65:44285
local_ip_for_outbound_sockets | 127.9.30.66 | experimental | 127.9.30.66:41265
local_ip_for_outbound_sockets | 127.9.30.67 | experimental | 127.9.30.67:34021
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.9.30.65:44285
server_dump_info_path | /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.9.30.66:41265
server_dump_info_path | /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.9.30.67:34021
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.9.30.84:44869 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable1 | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable2 | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 3
Tablets | 3
Replicas | 7
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250625 20:04:27.418984 9337 log_verifier.cc:126] Checking tablet 148d21f4b42e42d69b83bd9becad95e0
I20250625 20:04:27.432679 13155 raft_consensus.cc:1062] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166: attempting to promote NON_VOTER df7ba59f0fd749819107c7ef7cff68cd to VOTER
I20250625 20:04:27.434746 13155 consensus_queue.cc:237] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } } peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } attrs { promote: false } }
I20250625 20:04:27.440613 12899 raft_consensus.cc:1273] T bd510751b1cb4241a9f5541b94098e8c P 526f4e745a694c10a4403c8e8de7b98d [term 2 FOLLOWER]: Refusing update from remote peer 44f691d035604bcab46e9120f6028166: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250625 20:04:27.440649 13065 raft_consensus.cc:1273] T bd510751b1cb4241a9f5541b94098e8c P df7ba59f0fd749819107c7ef7cff68cd [term 2 LEARNER]: Refusing update from remote peer 44f691d035604bcab46e9120f6028166: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250625 20:04:27.442060 13005 consensus_queue.cc:1035] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250625 20:04:27.442936 13155 consensus_queue.cc:1035] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250625 20:04:27.450645 13155 raft_consensus.cc:2953] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, df7ba59f0fd749819107c7ef7cff68cd (127.9.30.67) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } } peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } attrs { promote: false } } }
I20250625 20:04:27.453024 12899 raft_consensus.cc:2953] T bd510751b1cb4241a9f5541b94098e8c P 526f4e745a694c10a4403c8e8de7b98d [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, df7ba59f0fd749819107c7ef7cff68cd (127.9.30.67) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } } peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } attrs { promote: false } } }
I20250625 20:04:27.459810 9337 log_verifier.cc:177] Verified matching terms for 8 ops in tablet 148d21f4b42e42d69b83bd9becad95e0
I20250625 20:04:27.460103 9337 log_verifier.cc:126] Checking tablet 15215d7e76d34b10b8b1f4f895db4204
I20250625 20:04:27.465351 12613 catalog_manager.cc:5582] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 reported cstate change: config changed from index 10 to 11, df7ba59f0fd749819107c7ef7cff68cd (127.9.30.67) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "44f691d035604bcab46e9120f6028166" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250625 20:04:27.466580 13065 raft_consensus.cc:2953] T bd510751b1cb4241a9f5541b94098e8c P df7ba59f0fd749819107c7ef7cff68cd [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, df7ba59f0fd749819107c7ef7cff68cd (127.9.30.67) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } } peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } attrs { promote: false } } }
I20250625 20:04:27.559481 9337 log_verifier.cc:177] Verified matching terms for 11 ops in tablet 15215d7e76d34b10b8b1f4f895db4204
I20250625 20:04:27.559810 9337 log_verifier.cc:126] Checking tablet bd510751b1cb4241a9f5541b94098e8c
I20250625 20:04:27.647898 9337 log_verifier.cc:177] Verified matching terms for 11 ops in tablet bd510751b1cb4241a9f5541b94098e8c
I20250625 20:04:27.648335 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 12580
I20250625 20:04:27.675051 9337 minidump.cc:252] Setting minidump size limit to 20M
I20250625 20:04:27.676623 9337 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:04:27.678217 9337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:04:27.689713 13188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:04:27.689934 13189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:04:27.867098 13191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:04:27.867746 9337 server_base.cc:1048] running on GCE node
I20250625 20:04:27.868834 9337 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250625 20:04:27.869067 9337 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250625 20:04:27.869239 9337 hybrid_clock.cc:648] HybridClock initialized: now 1750881867869220 us; error 0 us; skew 500 ppm
I20250625 20:04:27.869925 9337 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:04:27.873369 9337 webserver.cc:469] Webserver started at http://0.0.0.0:40591/ using document root <none> and password file <none>
I20250625 20:04:27.874464 9337 fs_manager.cc:362] Metadata directory not provided
I20250625 20:04:27.874738 9337 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:04:27.881032 9337 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.002s sys 0.002s
I20250625 20:04:27.885144 13196 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:04:27.886252 9337 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20250625 20:04:27.886595 9337 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
uuid: "4d3eec5ea99248df990c8b46a3384c6b"
format_stamp: "Formatted at 2025-06-25 20:04:07 on dist-test-slave-jhxp"
I20250625 20:04:27.888332 9337 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:04:27.904327 9337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:04:27.905783 9337 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:04:27.906224 9337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:04:27.914952 9337 sys_catalog.cc:263] Verifying existing consensus state
W20250625 20:04:27.918632 9337 sys_catalog.cc:243] For a single master config, on-disk Raft master: 127.9.30.126:39493 exists but no master address supplied!
I20250625 20:04:27.920732 9337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b: Bootstrap starting.
W20250625 20:04:27.933305 12796 debug-util.cc:398] Leaking SignalData structure 0x7b08000b3920 after lost signal to thread 12670
W20250625 20:04:27.956919 13115 heartbeater.cc:646] Failed to heartbeat to 127.9.30.126:39493 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.30.126:39493: connect: Connection refused (error 111)
I20250625 20:04:27.968245 9337 log.cc:826] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b: Log is configured to *not* fsync() on all Append() calls
I20250625 20:04:28.035727 9337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b: Bootstrap replayed 1/1 log segments. Stats: ops{read=31 overwritten=0 applied=31 ignored=0} inserts{seen=13 ignored=0} mutations{seen=22 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 20:04:28.036489 9337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b: Bootstrap complete.
I20250625 20:04:28.049835 9337 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3eec5ea99248df990c8b46a3384c6b" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 39493 } }
I20250625 20:04:28.050541 9337 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3eec5ea99248df990c8b46a3384c6b, State: Initialized, Role: FOLLOWER
I20250625 20:04:28.051301 9337 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3eec5ea99248df990c8b46a3384c6b" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 39493 } }
I20250625 20:04:28.051836 9337 raft_consensus.cc:397] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [term 3 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 20:04:28.052081 9337 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [term 3 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 20:04:28.052386 9337 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [term 3 FOLLOWER]: Advancing to term 4
I20250625 20:04:28.058049 9337 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3eec5ea99248df990c8b46a3384c6b" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 39493 } }
I20250625 20:04:28.058825 9337 leader_election.cc:304] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d3eec5ea99248df990c8b46a3384c6b; no voters:
I20250625 20:04:28.060166 9337 leader_election.cc:290] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [CANDIDATE]: Term 4 election: Requested vote from peers
I20250625 20:04:28.060441 13206 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [term 4 FOLLOWER]: Leader election won for term 4
I20250625 20:04:28.064129 13206 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [term 4 LEADER]: Becoming Leader. State: Replica: 4d3eec5ea99248df990c8b46a3384c6b, State: Running, Role: LEADER
I20250625 20:04:28.064957 13206 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 4, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3eec5ea99248df990c8b46a3384c6b" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 39493 } }
I20250625 20:04:28.071332 13208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d3eec5ea99248df990c8b46a3384c6b. Latest consensus state: current_term: 4 leader_uuid: "4d3eec5ea99248df990c8b46a3384c6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3eec5ea99248df990c8b46a3384c6b" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 39493 } } }
I20250625 20:04:28.072080 13208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [sys.catalog]: This master's current role is: LEADER
I20250625 20:04:28.072196 13207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 leader_uuid: "4d3eec5ea99248df990c8b46a3384c6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3eec5ea99248df990c8b46a3384c6b" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 39493 } } }
I20250625 20:04:28.072719 13207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [sys.catalog]: This master's current role is: LEADER
I20250625 20:04:28.101133 9337 tablet_replica.cc:331] stopping tablet replica
I20250625 20:04:28.101804 9337 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [term 4 LEADER]: Raft consensus shutting down.
I20250625 20:04:28.102324 9337 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [term 4 FOLLOWER]: Raft consensus is shut down!
I20250625 20:04:28.104760 9337 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250625 20:04:28.105280 9337 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250625 20:04:28.129556 9337 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
W20250625 20:04:28.495863 12944 heartbeater.cc:646] Failed to heartbeat to 127.9.30.126:39493 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.30.126:39493: connect: Connection refused (error 111)
W20250625 20:04:28.497490 12800 heartbeater.cc:646] Failed to heartbeat to 127.9.30.126:39493 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.30.126:39493: connect: Connection refused (error 111)
I20250625 20:04:33.754174 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 12650
I20250625 20:04:33.778409 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 12804
I20250625 20:04:33.802896 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 12950
I20250625 20:04:33.827544 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.30.126:39493
--webserver_interface=127.9.30.126
--webserver_port=43167
--builtin_ntp_servers=127.9.30.84:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.30.126:39493 with env {}
W20250625 20:04:34.127923 13280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:04:34.128556 13280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:04:34.129030 13280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:04:34.160661 13280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 20:04:34.160969 13280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:04:34.161350 13280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 20:04:34.161732 13280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 20:04:34.197683 13280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.30.126:39493
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.30.126:39493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.9.30.126
--webserver_port=43167
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:04:34.199047 13280 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:04:34.200825 13280 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:04:34.217129 13287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:04:34.217118 13286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:04:34.217375 13289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:04:34.219805 13280 server_base.cc:1048] running on GCE node
I20250625 20:04:35.313323 13280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:04:35.316437 13280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:04:35.317902 13280 hybrid_clock.cc:648] HybridClock initialized: now 1750881875317861 us; error 58 us; skew 500 ppm
I20250625 20:04:35.318709 13280 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:04:35.325147 13280 webserver.cc:469] Webserver started at http://127.9.30.126:43167/ using document root <none> and password file <none>
I20250625 20:04:35.326122 13280 fs_manager.cc:362] Metadata directory not provided
I20250625 20:04:35.326324 13280 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:04:35.334349 13280 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20250625 20:04:35.339129 13297 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:04:35.340292 13280 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250625 20:04:35.340597 13280 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
uuid: "4d3eec5ea99248df990c8b46a3384c6b"
format_stamp: "Formatted at 2025-06-25 20:04:07 on dist-test-slave-jhxp"
I20250625 20:04:35.342545 13280 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:04:35.408747 13280 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:04:35.410212 13280 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:04:35.410621 13280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:04:35.480558 13280 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.126:39493
I20250625 20:04:35.480626 13348 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.126:39493 every 8 connection(s)
I20250625 20:04:35.483393 13280 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250625 20:04:35.486905 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 13280
I20250625 20:04:35.489259 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.30.65:44285
--local_ip_for_outbound_sockets=127.9.30.65
--tserver_master_addrs=127.9.30.126:39493
--webserver_port=33919
--webserver_interface=127.9.30.65
--builtin_ntp_servers=127.9.30.84:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 20:04:35.497303 13349 sys_catalog.cc:263] Verifying existing consensus state
I20250625 20:04:35.503626 13349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b: Bootstrap starting.
I20250625 20:04:35.513995 13349 log.cc:826] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b: Log is configured to *not* fsync() on all Append() calls
I20250625 20:04:35.599583 13349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b: Bootstrap replayed 1/1 log segments. Stats: ops{read=35 overwritten=0 applied=35 ignored=0} inserts{seen=15 ignored=0} mutations{seen=24 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 20:04:35.600435 13349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b: Bootstrap complete.
I20250625 20:04:35.620213 13349 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3eec5ea99248df990c8b46a3384c6b" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 39493 } }
I20250625 20:04:35.622282 13349 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3eec5ea99248df990c8b46a3384c6b, State: Initialized, Role: FOLLOWER
I20250625 20:04:35.623147 13349 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 35, Last appended: 5.35, Last appended by leader: 35, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3eec5ea99248df990c8b46a3384c6b" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 39493 } }
I20250625 20:04:35.623683 13349 raft_consensus.cc:397] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [term 5 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 20:04:35.623920 13349 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [term 5 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 20:04:35.624212 13349 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [term 5 FOLLOWER]: Advancing to term 6
I20250625 20:04:35.629655 13349 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3eec5ea99248df990c8b46a3384c6b" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 39493 } }
I20250625 20:04:35.630373 13349 leader_election.cc:304] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d3eec5ea99248df990c8b46a3384c6b; no voters:
I20250625 20:04:35.632422 13349 leader_election.cc:290] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [CANDIDATE]: Term 6 election: Requested vote from peers
I20250625 20:04:35.632930 13353 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [term 6 FOLLOWER]: Leader election won for term 6
I20250625 20:04:35.636085 13353 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [term 6 LEADER]: Becoming Leader. State: Replica: 4d3eec5ea99248df990c8b46a3384c6b, State: Running, Role: LEADER
I20250625 20:04:35.637391 13349 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [sys.catalog]: configured and running, proceeding with master startup.
I20250625 20:04:35.637063 13353 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 5.35, Last appended by leader: 35, Current term: 6, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3eec5ea99248df990c8b46a3384c6b" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 39493 } }
I20250625 20:04:35.648896 13355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d3eec5ea99248df990c8b46a3384c6b. Latest consensus state: current_term: 6 leader_uuid: "4d3eec5ea99248df990c8b46a3384c6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3eec5ea99248df990c8b46a3384c6b" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 39493 } } }
I20250625 20:04:35.648589 13354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 leader_uuid: "4d3eec5ea99248df990c8b46a3384c6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3eec5ea99248df990c8b46a3384c6b" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 39493 } } }
I20250625 20:04:35.650311 13355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [sys.catalog]: This master's current role is: LEADER
I20250625 20:04:35.650703 13354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b [sys.catalog]: This master's current role is: LEADER
I20250625 20:04:35.656949 13362 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 20:04:35.675240 13362 catalog_manager.cc:671] Loaded metadata for table TestTable [id=24b010942db747f3a40e5b9065b4859e]
I20250625 20:04:35.677264 13362 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=988512eb9b3f4c649f1e7277df21a79b]
I20250625 20:04:35.679100 13362 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=db0e8ff6e0f74f769bed2e7f671276aa]
I20250625 20:04:35.688033 13362 tablet_loader.cc:96] loaded metadata for tablet 148d21f4b42e42d69b83bd9becad95e0 (table TestTable2 [id=988512eb9b3f4c649f1e7277df21a79b])
I20250625 20:04:35.689770 13362 tablet_loader.cc:96] loaded metadata for tablet 15215d7e76d34b10b8b1f4f895db4204 (table TestTable1 [id=db0e8ff6e0f74f769bed2e7f671276aa])
I20250625 20:04:35.691186 13362 tablet_loader.cc:96] loaded metadata for tablet bd510751b1cb4241a9f5541b94098e8c (table TestTable [id=24b010942db747f3a40e5b9065b4859e])
I20250625 20:04:35.692723 13362 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 20:04:35.698994 13362 catalog_manager.cc:1261] Loaded cluster ID: de233e8fc4a34d7db5e31c956aad7d4b
I20250625 20:04:35.699455 13362 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 20:04:35.709827 13362 catalog_manager.cc:1506] Loading token signing keys...
I20250625 20:04:35.715973 13362 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 4d3eec5ea99248df990c8b46a3384c6b: Loaded TSK: 0
I20250625 20:04:35.717738 13362 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250625 20:04:35.837203 13351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:04:35.837738 13351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:04:35.838261 13351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:04:35.871347 13351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:04:35.872291 13351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.30.65
I20250625 20:04:35.908742 13351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.30.65:44285
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.9.30.65
--webserver_port=33919
--tserver_master_addrs=127.9.30.126:39493
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.30.65
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:04:35.910100 13351 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:04:35.911921 13351 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:04:35.929590 13376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:04:35.929600 13377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:04:35.931908 13379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:04:37.029733 13378 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1101 milliseconds
I20250625 20:04:37.030334 13351 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 20:04:37.034359 13351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:04:37.037222 13351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:04:37.038805 13351 hybrid_clock.cc:648] HybridClock initialized: now 1750881877038746 us; error 56 us; skew 500 ppm
I20250625 20:04:37.040100 13351 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:04:37.051405 13351 webserver.cc:469] Webserver started at http://127.9.30.65:33919/ using document root <none> and password file <none>
I20250625 20:04:37.052758 13351 fs_manager.cc:362] Metadata directory not provided
I20250625 20:04:37.053076 13351 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:04:37.064100 13351 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.006s sys 0.001s
I20250625 20:04:37.070163 13386 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:04:37.071455 13351 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.004s
I20250625 20:04:37.071913 13351 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "44f691d035604bcab46e9120f6028166"
format_stamp: "Formatted at 2025-06-25 20:04:08 on dist-test-slave-jhxp"
I20250625 20:04:37.074800 13351 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:04:37.135205 13351 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:04:37.137254 13351 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:04:37.137852 13351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:04:37.141281 13351 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 20:04:37.149080 13393 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250625 20:04:37.161350 13351 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250625 20:04:37.161727 13351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.000s sys 0.003s
I20250625 20:04:37.162091 13351 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250625 20:04:37.171197 13393 tablet_bootstrap.cc:492] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166: Bootstrap starting.
I20250625 20:04:37.176005 13351 ts_tablet_manager.cc:610] Registered 2 tablets
I20250625 20:04:37.176328 13351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.014s user 0.006s sys 0.008s
I20250625 20:04:37.232769 13393 log.cc:826] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166: Log is configured to *not* fsync() on all Append() calls
I20250625 20:04:37.343417 13393 tablet_bootstrap.cc:492] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 20:04:37.344473 13393 tablet_bootstrap.cc:492] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166: Bootstrap complete.
I20250625 20:04:37.346195 13393 ts_tablet_manager.cc:1397] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166: Time spent bootstrapping tablet: real 0.176s user 0.128s sys 0.040s
I20250625 20:04:37.363441 13393 raft_consensus.cc:357] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } } peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } attrs { promote: false } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } }
I20250625 20:04:37.366524 13393 raft_consensus.cc:738] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44f691d035604bcab46e9120f6028166, State: Initialized, Role: FOLLOWER
I20250625 20:04:37.367486 13393 consensus_queue.cc:260] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } } peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } attrs { promote: false } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } }
I20250625 20:04:37.372208 13393 ts_tablet_manager.cc:1428] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166: Time spent starting tablet: real 0.026s user 0.028s sys 0.000s
I20250625 20:04:37.373124 13393 tablet_bootstrap.cc:492] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166: Bootstrap starting.
I20250625 20:04:37.374671 13351 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.65:44285
I20250625 20:04:37.374857 13501 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.65:44285 every 8 connection(s)
I20250625 20:04:37.378324 13351 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250625 20:04:37.388962 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 13351
I20250625 20:04:37.390830 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.30.66:41265
--local_ip_for_outbound_sockets=127.9.30.66
--tserver_master_addrs=127.9.30.126:39493
--webserver_port=40759
--webserver_interface=127.9.30.66
--builtin_ntp_servers=127.9.30.84:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 20:04:37.425874 13502 heartbeater.cc:344] Connected to a master server at 127.9.30.126:39493
I20250625 20:04:37.426441 13502 heartbeater.cc:461] Registering TS with master...
I20250625 20:04:37.427788 13502 heartbeater.cc:507] Master 127.9.30.126:39493 requested a full tablet report, sending...
I20250625 20:04:37.433539 13314 ts_manager.cc:194] Registered new tserver with Master: 44f691d035604bcab46e9120f6028166 (127.9.30.65:44285)
I20250625 20:04:37.438469 13314 catalog_manager.cc:5582] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 reported cstate change: config changed from index -1 to 11, term changed from 0 to 2, VOTER 44f691d035604bcab46e9120f6028166 (127.9.30.65) added, VOTER 526f4e745a694c10a4403c8e8de7b98d (127.9.30.66) added, VOTER df7ba59f0fd749819107c7ef7cff68cd (127.9.30.67) added. New cstate: current_term: 2 committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } } peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } attrs { promote: false } } }
I20250625 20:04:37.518539 13314 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.30.65:34269
I20250625 20:04:37.534652 13393 tablet_bootstrap.cc:492] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 20:04:37.535559 13393 tablet_bootstrap.cc:492] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166: Bootstrap complete.
I20250625 20:04:37.537042 13393 ts_tablet_manager.cc:1397] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166: Time spent bootstrapping tablet: real 0.164s user 0.126s sys 0.020s
I20250625 20:04:37.539186 13393 raft_consensus.cc:357] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } } peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } attrs { promote: false } }
I20250625 20:04:37.539860 13393 raft_consensus.cc:738] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44f691d035604bcab46e9120f6028166, State: Initialized, Role: FOLLOWER
I20250625 20:04:37.540463 13393 consensus_queue.cc:260] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } } peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } attrs { promote: false } }
I20250625 20:04:37.541716 13502 heartbeater.cc:499] Master 127.9.30.126:39493 was elected leader, sending a full tablet report...
I20250625 20:04:37.542346 13393 ts_tablet_manager.cc:1428] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20250625 20:04:37.780275 13506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:04:37.780790 13506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:04:37.781304 13506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:04:37.813372 13506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:04:37.814250 13506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.30.66
I20250625 20:04:37.850492 13506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.30.66:41265
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.9.30.66
--webserver_port=40759
--tserver_master_addrs=127.9.30.126:39493
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.30.66
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:04:37.851941 13506 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:04:37.853727 13506 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:04:37.869853 13517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:04:38.733727 13523 raft_consensus.cc:491] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:04:38.734248 13523 raft_consensus.cc:513] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } } peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } attrs { promote: false } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } }
I20250625 20:04:38.736776 13523 leader_election.cc:290] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df7ba59f0fd749819107c7ef7cff68cd (127.9.30.67:34021), 526f4e745a694c10a4403c8e8de7b98d (127.9.30.66:41265)
W20250625 20:04:38.740655 13390 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.30.67:34021: connect: Connection refused (error 111)
W20250625 20:04:38.745142 13389 leader_election.cc:336] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 526f4e745a694c10a4403c8e8de7b98d (127.9.30.66:41265): Network error: Client connection negotiation failed: client connection to 127.9.30.66:41265: connect: Connection refused (error 111)
W20250625 20:04:38.745783 13390 leader_election.cc:336] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer df7ba59f0fd749819107c7ef7cff68cd (127.9.30.67:34021): Network error: Client connection negotiation failed: client connection to 127.9.30.67:34021: connect: Connection refused (error 111)
I20250625 20:04:38.746106 13390 leader_election.cc:304] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44f691d035604bcab46e9120f6028166; no voters: 526f4e745a694c10a4403c8e8de7b98d, df7ba59f0fd749819107c7ef7cff68cd
I20250625 20:04:38.746740 13523 raft_consensus.cc:2747] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250625 20:04:38.862346 13523 raft_consensus.cc:491] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:04:38.862860 13523 raft_consensus.cc:513] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } } peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } attrs { promote: false } }
I20250625 20:04:38.864782 13523 leader_election.cc:290] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 526f4e745a694c10a4403c8e8de7b98d (127.9.30.66:41265), df7ba59f0fd749819107c7ef7cff68cd (127.9.30.67:34021)
W20250625 20:04:38.868319 13390 leader_election.cc:336] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer df7ba59f0fd749819107c7ef7cff68cd (127.9.30.67:34021): Network error: Client connection negotiation failed: client connection to 127.9.30.67:34021: connect: Connection refused (error 111)
W20250625 20:04:38.868865 13389 leader_election.cc:336] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 526f4e745a694c10a4403c8e8de7b98d (127.9.30.66:41265): Network error: Client connection negotiation failed: client connection to 127.9.30.66:41265: connect: Connection refused (error 111)
I20250625 20:04:38.869213 13389 leader_election.cc:304] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44f691d035604bcab46e9120f6028166; no voters: 526f4e745a694c10a4403c8e8de7b98d, df7ba59f0fd749819107c7ef7cff68cd
I20250625 20:04:38.869797 13523 raft_consensus.cc:2747] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250625 20:04:37.870198 13518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:04:37.871150 13520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:04:38.949648 13519 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250625 20:04:38.949714 13506 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 20:04:38.954049 13506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:04:38.956709 13506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:04:38.958149 13506 hybrid_clock.cc:648] HybridClock initialized: now 1750881878958104 us; error 64 us; skew 500 ppm
I20250625 20:04:38.958966 13506 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:04:38.965282 13506 webserver.cc:469] Webserver started at http://127.9.30.66:40759/ using document root <none> and password file <none>
I20250625 20:04:38.966252 13506 fs_manager.cc:362] Metadata directory not provided
I20250625 20:04:38.966485 13506 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:04:38.974637 13506 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20250625 20:04:38.979589 13531 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:04:38.980791 13506 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250625 20:04:38.981140 13506 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "526f4e745a694c10a4403c8e8de7b98d"
format_stamp: "Formatted at 2025-06-25 20:04:10 on dist-test-slave-jhxp"
I20250625 20:04:38.983181 13506 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:04:39.030858 13506 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:04:39.032405 13506 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:04:39.032845 13506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:04:39.035323 13506 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 20:04:39.041638 13538 ts_tablet_manager.cc:542] Loading tablet metadata (0/3 complete)
I20250625 20:04:39.058060 13506 ts_tablet_manager.cc:579] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20250625 20:04:39.058337 13506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.018s user 0.000s sys 0.002s
I20250625 20:04:39.058606 13506 ts_tablet_manager.cc:594] Registering tablets (0/3 complete)
I20250625 20:04:39.064110 13538 tablet_bootstrap.cc:492] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d: Bootstrap starting.
I20250625 20:04:39.070410 13506 ts_tablet_manager.cc:610] Registered 3 tablets
I20250625 20:04:39.070649 13506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.012s user 0.009s sys 0.001s
I20250625 20:04:39.116253 13538 log.cc:826] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d: Log is configured to *not* fsync() on all Append() calls
I20250625 20:04:39.213193 13538 tablet_bootstrap.cc:492] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 20:04:39.214010 13538 tablet_bootstrap.cc:492] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d: Bootstrap complete.
I20250625 20:04:39.215299 13538 ts_tablet_manager.cc:1397] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d: Time spent bootstrapping tablet: real 0.152s user 0.120s sys 0.027s
I20250625 20:04:39.232602 13538 raft_consensus.cc:357] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } }
I20250625 20:04:39.235069 13538 raft_consensus.cc:738] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 526f4e745a694c10a4403c8e8de7b98d, State: Initialized, Role: FOLLOWER
I20250625 20:04:39.236047 13538 consensus_queue.cc:260] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } }
I20250625 20:04:39.236589 13538 raft_consensus.cc:397] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 20:04:39.236948 13538 raft_consensus.cc:491] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 20:04:39.237329 13538 raft_consensus.cc:3058] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d [term 2 FOLLOWER]: Advancing to term 3
I20250625 20:04:39.239180 13506 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.66:41265
I20250625 20:04:39.239398 13645 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.66:41265 every 8 connection(s)
I20250625 20:04:39.242049 13506 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250625 20:04:39.243984 13538 raft_consensus.cc:513] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } }
I20250625 20:04:39.244306 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 13506
I20250625 20:04:39.244985 13538 leader_election.cc:304] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 526f4e745a694c10a4403c8e8de7b98d; no voters:
I20250625 20:04:39.246215 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.30.67:34021
--local_ip_for_outbound_sockets=127.9.30.67
--tserver_master_addrs=127.9.30.126:39493
--webserver_port=35311
--webserver_interface=127.9.30.67
--builtin_ntp_servers=127.9.30.84:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 20:04:39.253825 13538 leader_election.cc:290] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d [CANDIDATE]: Term 3 election: Requested vote from peers
I20250625 20:04:39.254045 13648 raft_consensus.cc:2802] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d [term 3 FOLLOWER]: Leader election won for term 3
I20250625 20:04:39.261011 13538 ts_tablet_manager.cc:1428] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d: Time spent starting tablet: real 0.045s user 0.036s sys 0.009s
I20250625 20:04:39.262003 13538 tablet_bootstrap.cc:492] T 15215d7e76d34b10b8b1f4f895db4204 P 526f4e745a694c10a4403c8e8de7b98d: Bootstrap starting.
I20250625 20:04:39.268767 13648 raft_consensus.cc:695] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d [term 3 LEADER]: Becoming Leader. State: Replica: 526f4e745a694c10a4403c8e8de7b98d, State: Running, Role: LEADER
I20250625 20:04:39.269827 13648 consensus_queue.cc:237] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } }
I20250625 20:04:39.283851 13646 heartbeater.cc:344] Connected to a master server at 127.9.30.126:39493
I20250625 20:04:39.284624 13646 heartbeater.cc:461] Registering TS with master...
I20250625 20:04:39.286458 13646 heartbeater.cc:507] Master 127.9.30.126:39493 requested a full tablet report, sending...
I20250625 20:04:39.303999 13314 ts_manager.cc:194] Registered new tserver with Master: 526f4e745a694c10a4403c8e8de7b98d (127.9.30.66:41265)
I20250625 20:04:39.306529 13314 catalog_manager.cc:5582] T 148d21f4b42e42d69b83bd9becad95e0 P 526f4e745a694c10a4403c8e8de7b98d reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "526f4e745a694c10a4403c8e8de7b98d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } health_report { overall_health: HEALTHY } } }
I20250625 20:04:39.322532 13314 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.30.66:48607
I20250625 20:04:39.326704 13646 heartbeater.cc:499] Master 127.9.30.126:39493 was elected leader, sending a full tablet report...
I20250625 20:04:39.378619 13538 tablet_bootstrap.cc:492] T 15215d7e76d34b10b8b1f4f895db4204 P 526f4e745a694c10a4403c8e8de7b98d: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 20:04:39.379616 13538 tablet_bootstrap.cc:492] T 15215d7e76d34b10b8b1f4f895db4204 P 526f4e745a694c10a4403c8e8de7b98d: Bootstrap complete.
I20250625 20:04:39.381132 13538 ts_tablet_manager.cc:1397] T 15215d7e76d34b10b8b1f4f895db4204 P 526f4e745a694c10a4403c8e8de7b98d: Time spent bootstrapping tablet: real 0.119s user 0.100s sys 0.015s
I20250625 20:04:39.383411 13538 raft_consensus.cc:357] T 15215d7e76d34b10b8b1f4f895db4204 P 526f4e745a694c10a4403c8e8de7b98d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } } peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } attrs { promote: false } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } }
I20250625 20:04:39.384233 13538 raft_consensus.cc:738] T 15215d7e76d34b10b8b1f4f895db4204 P 526f4e745a694c10a4403c8e8de7b98d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 526f4e745a694c10a4403c8e8de7b98d, State: Initialized, Role: FOLLOWER
I20250625 20:04:39.384896 13538 consensus_queue.cc:260] T 15215d7e76d34b10b8b1f4f895db4204 P 526f4e745a694c10a4403c8e8de7b98d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } } peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } attrs { promote: false } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } }
I20250625 20:04:39.386989 13538 ts_tablet_manager.cc:1428] T 15215d7e76d34b10b8b1f4f895db4204 P 526f4e745a694c10a4403c8e8de7b98d: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250625 20:04:39.387750 13538 tablet_bootstrap.cc:492] T bd510751b1cb4241a9f5541b94098e8c P 526f4e745a694c10a4403c8e8de7b98d: Bootstrap starting.
I20250625 20:04:39.480446 13538 tablet_bootstrap.cc:492] T bd510751b1cb4241a9f5541b94098e8c P 526f4e745a694c10a4403c8e8de7b98d: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 20:04:39.481122 13538 tablet_bootstrap.cc:492] T bd510751b1cb4241a9f5541b94098e8c P 526f4e745a694c10a4403c8e8de7b98d: Bootstrap complete.
I20250625 20:04:39.482088 13538 ts_tablet_manager.cc:1397] T bd510751b1cb4241a9f5541b94098e8c P 526f4e745a694c10a4403c8e8de7b98d: Time spent bootstrapping tablet: real 0.095s user 0.084s sys 0.008s
I20250625 20:04:39.483682 13538 raft_consensus.cc:357] T bd510751b1cb4241a9f5541b94098e8c P 526f4e745a694c10a4403c8e8de7b98d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } } peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } attrs { promote: false } }
I20250625 20:04:39.484174 13538 raft_consensus.cc:738] T bd510751b1cb4241a9f5541b94098e8c P 526f4e745a694c10a4403c8e8de7b98d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 526f4e745a694c10a4403c8e8de7b98d, State: Initialized, Role: FOLLOWER
I20250625 20:04:39.484699 13538 consensus_queue.cc:260] T bd510751b1cb4241a9f5541b94098e8c P 526f4e745a694c10a4403c8e8de7b98d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } } peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } attrs { promote: false } }
I20250625 20:04:39.486073 13538 ts_tablet_manager.cc:1428] T bd510751b1cb4241a9f5541b94098e8c P 526f4e745a694c10a4403c8e8de7b98d: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20250625 20:04:39.604090 13653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:04:39.604594 13653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:04:39.605089 13653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:04:39.637646 13653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:04:39.638511 13653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.30.67
I20250625 20:04:39.674721 13653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.30.67:34021
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.9.30.67
--webserver_port=35311
--tserver_master_addrs=127.9.30.126:39493
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.30.67
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:04:39.676148 13653 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:04:39.677829 13653 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:04:39.693898 13667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:04:40.417491 13673 raft_consensus.cc:491] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:04:40.417876 13673 raft_consensus.cc:513] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } } peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } attrs { promote: false } }
I20250625 20:04:40.419530 13673 leader_election.cc:290] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 526f4e745a694c10a4403c8e8de7b98d (127.9.30.66:41265), df7ba59f0fd749819107c7ef7cff68cd (127.9.30.67:34021)
W20250625 20:04:40.437834 13390 leader_election.cc:336] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer df7ba59f0fd749819107c7ef7cff68cd (127.9.30.67:34021): Network error: Client connection negotiation failed: client connection to 127.9.30.67:34021: connect: Connection refused (error 111)
I20250625 20:04:40.445588 13601 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd510751b1cb4241a9f5541b94098e8c" candidate_uuid: "44f691d035604bcab46e9120f6028166" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "526f4e745a694c10a4403c8e8de7b98d" is_pre_election: true
I20250625 20:04:40.446287 13601 raft_consensus.cc:2466] T bd510751b1cb4241a9f5541b94098e8c P 526f4e745a694c10a4403c8e8de7b98d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44f691d035604bcab46e9120f6028166 in term 2.
I20250625 20:04:40.447587 13389 leader_election.cc:304] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44f691d035604bcab46e9120f6028166, 526f4e745a694c10a4403c8e8de7b98d; no voters: df7ba59f0fd749819107c7ef7cff68cd
I20250625 20:04:40.448266 13673 raft_consensus.cc:2802] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250625 20:04:40.448575 13673 raft_consensus.cc:491] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:04:40.448827 13673 raft_consensus.cc:3058] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [term 2 FOLLOWER]: Advancing to term 3
I20250625 20:04:40.454989 13673 raft_consensus.cc:513] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } } peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } attrs { promote: false } }
I20250625 20:04:40.456542 13673 leader_election.cc:290] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [CANDIDATE]: Term 3 election: Requested vote from peers 526f4e745a694c10a4403c8e8de7b98d (127.9.30.66:41265), df7ba59f0fd749819107c7ef7cff68cd (127.9.30.67:34021)
I20250625 20:04:40.457340 13601 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd510751b1cb4241a9f5541b94098e8c" candidate_uuid: "44f691d035604bcab46e9120f6028166" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "526f4e745a694c10a4403c8e8de7b98d"
I20250625 20:04:40.457893 13601 raft_consensus.cc:3058] T bd510751b1cb4241a9f5541b94098e8c P 526f4e745a694c10a4403c8e8de7b98d [term 2 FOLLOWER]: Advancing to term 3
W20250625 20:04:40.461071 13390 leader_election.cc:336] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer df7ba59f0fd749819107c7ef7cff68cd (127.9.30.67:34021): Network error: Client connection negotiation failed: client connection to 127.9.30.67:34021: connect: Connection refused (error 111)
I20250625 20:04:40.463141 13601 raft_consensus.cc:2466] T bd510751b1cb4241a9f5541b94098e8c P 526f4e745a694c10a4403c8e8de7b98d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f691d035604bcab46e9120f6028166 in term 3.
I20250625 20:04:40.464241 13389 leader_election.cc:304] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44f691d035604bcab46e9120f6028166, 526f4e745a694c10a4403c8e8de7b98d; no voters: df7ba59f0fd749819107c7ef7cff68cd
I20250625 20:04:40.465175 13673 raft_consensus.cc:2802] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [term 3 FOLLOWER]: Leader election won for term 3
I20250625 20:04:40.466781 13673 raft_consensus.cc:695] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [term 3 LEADER]: Becoming Leader. State: Replica: 44f691d035604bcab46e9120f6028166, State: Running, Role: LEADER
I20250625 20:04:40.467730 13673 consensus_queue.cc:237] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } } peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } attrs { promote: false } }
I20250625 20:04:40.482052 13314 catalog_manager.cc:5582] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 reported cstate change: term changed from 2 to 3, leader changed from <none> to 44f691d035604bcab46e9120f6028166 (127.9.30.65). New cstate: current_term: 3 leader_uuid: "44f691d035604bcab46e9120f6028166" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250625 20:04:39.694278 13670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:04:39.696449 13653 server_base.cc:1048] running on GCE node
W20250625 20:04:39.694880 13668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:04:40.792439 13653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:04:40.795120 13653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:04:40.796584 13653 hybrid_clock.cc:648] HybridClock initialized: now 1750881880796546 us; error 58 us; skew 500 ppm
I20250625 20:04:40.797420 13653 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:04:40.804013 13653 webserver.cc:469] Webserver started at http://127.9.30.67:35311/ using document root <none> and password file <none>
I20250625 20:04:40.804971 13653 fs_manager.cc:362] Metadata directory not provided
I20250625 20:04:40.805207 13653 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:04:40.813158 13653 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20250625 20:04:40.818019 13688 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:04:40.818012 13673 raft_consensus.cc:491] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:04:40.818454 13673 raft_consensus.cc:513] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } } peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } attrs { promote: false } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } }
I20250625 20:04:40.819365 13653 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250625 20:04:40.820415 13673 leader_election.cc:290] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df7ba59f0fd749819107c7ef7cff68cd (127.9.30.67:34021), 526f4e745a694c10a4403c8e8de7b98d (127.9.30.66:41265)
I20250625 20:04:40.821439 13601 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15215d7e76d34b10b8b1f4f895db4204" candidate_uuid: "44f691d035604bcab46e9120f6028166" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "526f4e745a694c10a4403c8e8de7b98d" is_pre_election: true
I20250625 20:04:40.821956 13601 raft_consensus.cc:2466] T 15215d7e76d34b10b8b1f4f895db4204 P 526f4e745a694c10a4403c8e8de7b98d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44f691d035604bcab46e9120f6028166 in term 2.
I20250625 20:04:40.819811 13653 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "df7ba59f0fd749819107c7ef7cff68cd"
format_stamp: "Formatted at 2025-06-25 20:04:12 on dist-test-slave-jhxp"
I20250625 20:04:40.822885 13389 leader_election.cc:304] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44f691d035604bcab46e9120f6028166, 526f4e745a694c10a4403c8e8de7b98d; no voters:
I20250625 20:04:40.822657 13653 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:04:40.823709 13673 raft_consensus.cc:2802] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250625 20:04:40.824065 13673 raft_consensus.cc:491] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250625 20:04:40.824172 13390 leader_election.cc:336] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer df7ba59f0fd749819107c7ef7cff68cd (127.9.30.67:34021): Network error: Client connection negotiation failed: client connection to 127.9.30.67:34021: connect: Connection refused (error 111)
I20250625 20:04:40.824402 13673 raft_consensus.cc:3058] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166 [term 2 FOLLOWER]: Advancing to term 3
I20250625 20:04:40.830636 13673 raft_consensus.cc:513] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } } peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } attrs { promote: false } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } }
I20250625 20:04:40.832571 13673 leader_election.cc:290] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166 [CANDIDATE]: Term 3 election: Requested vote from peers df7ba59f0fd749819107c7ef7cff68cd (127.9.30.67:34021), 526f4e745a694c10a4403c8e8de7b98d (127.9.30.66:41265)
I20250625 20:04:40.833982 13601 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15215d7e76d34b10b8b1f4f895db4204" candidate_uuid: "44f691d035604bcab46e9120f6028166" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "526f4e745a694c10a4403c8e8de7b98d"
I20250625 20:04:40.834582 13601 raft_consensus.cc:3058] T 15215d7e76d34b10b8b1f4f895db4204 P 526f4e745a694c10a4403c8e8de7b98d [term 2 FOLLOWER]: Advancing to term 3
W20250625 20:04:40.836422 13390 leader_election.cc:336] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer df7ba59f0fd749819107c7ef7cff68cd (127.9.30.67:34021): Network error: Client connection negotiation failed: client connection to 127.9.30.67:34021: connect: Connection refused (error 111)
I20250625 20:04:40.839247 13601 raft_consensus.cc:2466] T 15215d7e76d34b10b8b1f4f895db4204 P 526f4e745a694c10a4403c8e8de7b98d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f691d035604bcab46e9120f6028166 in term 3.
I20250625 20:04:40.840274 13389 leader_election.cc:304] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44f691d035604bcab46e9120f6028166, 526f4e745a694c10a4403c8e8de7b98d; no voters: df7ba59f0fd749819107c7ef7cff68cd
I20250625 20:04:40.841037 13673 raft_consensus.cc:2802] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166 [term 3 FOLLOWER]: Leader election won for term 3
I20250625 20:04:40.841377 13673 raft_consensus.cc:695] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166 [term 3 LEADER]: Becoming Leader. State: Replica: 44f691d035604bcab46e9120f6028166, State: Running, Role: LEADER
I20250625 20:04:40.842100 13673 consensus_queue.cc:237] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } } peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } attrs { promote: false } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } }
I20250625 20:04:40.849864 13313 catalog_manager.cc:5582] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166 reported cstate change: term changed from 2 to 3, leader changed from df7ba59f0fd749819107c7ef7cff68cd (127.9.30.67) to 44f691d035604bcab46e9120f6028166 (127.9.30.65). New cstate: current_term: 3 leader_uuid: "44f691d035604bcab46e9120f6028166" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250625 20:04:40.875768 13653 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
W20250625 20:04:40.877441 13390 consensus_peers.cc:487] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 -> Peer df7ba59f0fd749819107c7ef7cff68cd (127.9.30.67:34021): Couldn't send request to peer df7ba59f0fd749819107c7ef7cff68cd. Status: Network error: Client connection negotiation failed: client connection to 127.9.30.67:34021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250625 20:04:40.877614 13653 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:04:40.878093 13653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:04:40.880625 13653 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 20:04:40.886406 13696 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250625 20:04:40.898380 13653 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250625 20:04:40.898650 13653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.001s sys 0.001s
I20250625 20:04:40.898924 13653 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250625 20:04:40.904727 13696 tablet_bootstrap.cc:492] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd: Bootstrap starting.
I20250625 20:04:40.907644 13653 ts_tablet_manager.cc:610] Registered 2 tablets
I20250625 20:04:40.907896 13653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.009s user 0.009s sys 0.000s
I20250625 20:04:40.914215 13601 raft_consensus.cc:1273] T bd510751b1cb4241a9f5541b94098e8c P 526f4e745a694c10a4403c8e8de7b98d [term 3 FOLLOWER]: Refusing update from remote peer 44f691d035604bcab46e9120f6028166: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20250625 20:04:40.916002 13673 consensus_queue.cc:1035] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20250625 20:04:40.974515 13456 consensus_queue.cc:237] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } }
I20250625 20:04:40.975498 13696 log.cc:826] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd: Log is configured to *not* fsync() on all Append() calls
I20250625 20:04:40.979970 13601 raft_consensus.cc:1273] T bd510751b1cb4241a9f5541b94098e8c P 526f4e745a694c10a4403c8e8de7b98d [term 3 FOLLOWER]: Refusing update from remote peer 44f691d035604bcab46e9120f6028166: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20250625 20:04:40.981696 13673 consensus_queue.cc:1035] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250625 20:04:40.989761 13678 raft_consensus.cc:2953] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [term 3 LEADER]: Committing config change with OpId 3.13: config changed from index 11 to 13, VOTER df7ba59f0fd749819107c7ef7cff68cd (127.9.30.67) evicted. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } } }
I20250625 20:04:40.991920 13601 raft_consensus.cc:2953] T bd510751b1cb4241a9f5541b94098e8c P 526f4e745a694c10a4403c8e8de7b98d [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 11 to 13, VOTER df7ba59f0fd749819107c7ef7cff68cd (127.9.30.67) evicted. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } } }
I20250625 20:04:41.001245 13299 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet bd510751b1cb4241a9f5541b94098e8c with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250625 20:04:41.006690 13313 catalog_manager.cc:5582] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 reported cstate change: config changed from index 11 to 13, VOTER df7ba59f0fd749819107c7ef7cff68cd (127.9.30.67) evicted. New cstate: current_term: 3 leader_uuid: "44f691d035604bcab46e9120f6028166" committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250625 20:04:41.021637 13313 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet bd510751b1cb4241a9f5541b94098e8c on TS df7ba59f0fd749819107c7ef7cff68cd: Not found: failed to reset TS proxy: Could not find TS for UUID df7ba59f0fd749819107c7ef7cff68cd
I20250625 20:04:41.025411 13456 consensus_queue.cc:237] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 11, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } }
I20250625 20:04:41.028155 13678 raft_consensus.cc:2953] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 13 to 14, VOTER 526f4e745a694c10a4403c8e8de7b98d (127.9.30.66) evicted. New config: { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } } }
I20250625 20:04:41.037979 13299 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet bd510751b1cb4241a9f5541b94098e8c with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250625 20:04:41.041780 13313 catalog_manager.cc:5582] T bd510751b1cb4241a9f5541b94098e8c P 44f691d035604bcab46e9120f6028166 reported cstate change: config changed from index 13 to 14, VOTER 526f4e745a694c10a4403c8e8de7b98d (127.9.30.66) evicted. New cstate: current_term: 3 leader_uuid: "44f691d035604bcab46e9120f6028166" committed_config { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } health_report { overall_health: HEALTHY } } }
W20250625 20:04:41.076102 13299 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet bd510751b1cb4241a9f5541b94098e8c on TS df7ba59f0fd749819107c7ef7cff68cd failed: Not found: failed to reset TS proxy: Could not find TS for UUID df7ba59f0fd749819107c7ef7cff68cd
I20250625 20:04:41.081804 13581 tablet_service.cc:1515] Processing DeleteTablet for tablet bd510751b1cb4241a9f5541b94098e8c with delete_type TABLET_DATA_TOMBSTONED (TS 526f4e745a694c10a4403c8e8de7b98d not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:56640
I20250625 20:04:41.084867 13756 tablet_replica.cc:331] stopping tablet replica
I20250625 20:04:41.085863 13756 raft_consensus.cc:2241] T bd510751b1cb4241a9f5541b94098e8c P 526f4e745a694c10a4403c8e8de7b98d [term 3 FOLLOWER]: Raft consensus shutting down.
I20250625 20:04:41.086639 13756 raft_consensus.cc:2270] T bd510751b1cb4241a9f5541b94098e8c P 526f4e745a694c10a4403c8e8de7b98d [term 3 FOLLOWER]: Raft consensus is shut down!
I20250625 20:04:41.090581 13756 ts_tablet_manager.cc:1905] T bd510751b1cb4241a9f5541b94098e8c P 526f4e745a694c10a4403c8e8de7b98d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250625 20:04:41.097939 13696 tablet_bootstrap.cc:492] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 20:04:41.098939 13696 tablet_bootstrap.cc:492] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd: Bootstrap complete.
I20250625 20:04:41.100518 13696 ts_tablet_manager.cc:1397] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd: Time spent bootstrapping tablet: real 0.196s user 0.138s sys 0.042s
I20250625 20:04:41.108567 13756 ts_tablet_manager.cc:1918] T bd510751b1cb4241a9f5541b94098e8c P 526f4e745a694c10a4403c8e8de7b98d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.13
I20250625 20:04:41.109257 13756 log.cc:1199] T bd510751b1cb4241a9f5541b94098e8c P 526f4e745a694c10a4403c8e8de7b98d: Deleting WAL directory at /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal/wals/bd510751b1cb4241a9f5541b94098e8c
I20250625 20:04:41.111426 13300 catalog_manager.cc:4928] TS 526f4e745a694c10a4403c8e8de7b98d (127.9.30.66:41265): tablet bd510751b1cb4241a9f5541b94098e8c (table TestTable [id=24b010942db747f3a40e5b9065b4859e]) successfully deleted
I20250625 20:04:41.120074 13696 raft_consensus.cc:357] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } } peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } attrs { promote: false } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } }
I20250625 20:04:41.122400 13696 raft_consensus.cc:738] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: df7ba59f0fd749819107c7ef7cff68cd, State: Initialized, Role: FOLLOWER
I20250625 20:04:41.123273 13696 consensus_queue.cc:260] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } } peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } attrs { promote: false } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } }
I20250625 20:04:41.128150 13696 ts_tablet_manager.cc:1428] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd: Time spent starting tablet: real 0.027s user 0.027s sys 0.001s
I20250625 20:04:41.129123 13696 tablet_bootstrap.cc:492] T bd510751b1cb4241a9f5541b94098e8c P df7ba59f0fd749819107c7ef7cff68cd: Bootstrap starting.
I20250625 20:04:41.177186 13653 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.67:34021
I20250625 20:04:41.177449 13812 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.67:34021 every 8 connection(s)
I20250625 20:04:41.180788 13653 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250625 20:04:41.183320 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 13653
I20250625 20:04:41.230738 13813 heartbeater.cc:344] Connected to a master server at 127.9.30.126:39493
I20250625 20:04:41.231339 13813 heartbeater.cc:461] Registering TS with master...
I20250625 20:04:41.232724 13813 heartbeater.cc:507] Master 127.9.30.126:39493 requested a full tablet report, sending...
I20250625 20:04:41.237754 13313 ts_manager.cc:194] Registered new tserver with Master: df7ba59f0fd749819107c7ef7cff68cd (127.9.30.67:34021)
I20250625 20:04:41.240810 9337 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 20:04:41.248229 9337 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 20:04:41.249892 13313 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.30.67:51073
W20250625 20:04:41.255457 9337 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bd510751b1cb4241a9f5541b94098e8c: tablet_id: "bd510751b1cb4241a9f5541b94098e8c" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
I20250625 20:04:41.285992 13696 tablet_bootstrap.cc:492] T bd510751b1cb4241a9f5541b94098e8c P df7ba59f0fd749819107c7ef7cff68cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 20:04:41.286685 13696 tablet_bootstrap.cc:492] T bd510751b1cb4241a9f5541b94098e8c P df7ba59f0fd749819107c7ef7cff68cd: Bootstrap complete.
I20250625 20:04:41.287737 13696 ts_tablet_manager.cc:1397] T bd510751b1cb4241a9f5541b94098e8c P df7ba59f0fd749819107c7ef7cff68cd: Time spent bootstrapping tablet: real 0.159s user 0.119s sys 0.035s
I20250625 20:04:41.289338 13696 raft_consensus.cc:357] T bd510751b1cb4241a9f5541b94098e8c P df7ba59f0fd749819107c7ef7cff68cd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } } peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } attrs { promote: false } }
I20250625 20:04:41.289815 13696 raft_consensus.cc:738] T bd510751b1cb4241a9f5541b94098e8c P df7ba59f0fd749819107c7ef7cff68cd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: df7ba59f0fd749819107c7ef7cff68cd, State: Initialized, Role: FOLLOWER
I20250625 20:04:41.290342 13696 consensus_queue.cc:260] T bd510751b1cb4241a9f5541b94098e8c P df7ba59f0fd749819107c7ef7cff68cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "44f691d035604bcab46e9120f6028166" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 44285 } } peers { permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false } } peers { permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 } attrs { promote: false } }
I20250625 20:04:41.291328 13813 heartbeater.cc:499] Master 127.9.30.126:39493 was elected leader, sending a full tablet report...
I20250625 20:04:41.291905 13696 ts_tablet_manager.cc:1428] T bd510751b1cb4241a9f5541b94098e8c P df7ba59f0fd749819107c7ef7cff68cd: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250625 20:04:41.312443 13767 raft_consensus.cc:3058] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [term 2 FOLLOWER]: Advancing to term 3
I20250625 20:04:41.319495 13767 raft_consensus.cc:1273] T 15215d7e76d34b10b8b1f4f895db4204 P df7ba59f0fd749819107c7ef7cff68cd [term 3 FOLLOWER]: Refusing update from remote peer 44f691d035604bcab46e9120f6028166: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20250625 20:04:41.321255 13678 consensus_queue.cc:1035] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df7ba59f0fd749819107c7ef7cff68cd" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 34021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250625 20:04:41.331696 13601 raft_consensus.cc:1273] T 15215d7e76d34b10b8b1f4f895db4204 P 526f4e745a694c10a4403c8e8de7b98d [term 3 FOLLOWER]: Refusing update from remote peer 44f691d035604bcab46e9120f6028166: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20250625 20:04:41.334019 13678 consensus_queue.cc:1035] T 15215d7e76d34b10b8b1f4f895db4204 P 44f691d035604bcab46e9120f6028166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "526f4e745a694c10a4403c8e8de7b98d" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 41265 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250625 20:04:41.405331 13746 tablet_service.cc:1515] Processing DeleteTablet for tablet bd510751b1cb4241a9f5541b94098e8c with delete_type TABLET_DATA_TOMBSTONED (TS df7ba59f0fd749819107c7ef7cff68cd not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:60876
I20250625 20:04:41.410271 13825 tablet_replica.cc:331] stopping tablet replica
I20250625 20:04:41.410948 13825 raft_consensus.cc:2241] T bd510751b1cb4241a9f5541b94098e8c P df7ba59f0fd749819107c7ef7cff68cd [term 2 FOLLOWER]: Raft consensus shutting down.
I20250625 20:04:41.411407 13825 raft_consensus.cc:2270] T bd510751b1cb4241a9f5541b94098e8c P df7ba59f0fd749819107c7ef7cff68cd [term 2 FOLLOWER]: Raft consensus is shut down!
I20250625 20:04:41.413828 13825 ts_tablet_manager.cc:1905] T bd510751b1cb4241a9f5541b94098e8c P df7ba59f0fd749819107c7ef7cff68cd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250625 20:04:41.424549 13825 ts_tablet_manager.cc:1918] T bd510751b1cb4241a9f5541b94098e8c P df7ba59f0fd749819107c7ef7cff68cd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.11
I20250625 20:04:41.424927 13825 log.cc:1199] T bd510751b1cb4241a9f5541b94098e8c P df7ba59f0fd749819107c7ef7cff68cd: Deleting WAL directory at /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal/wals/bd510751b1cb4241a9f5541b94098e8c
I20250625 20:04:41.426322 13301 catalog_manager.cc:4928] TS df7ba59f0fd749819107c7ef7cff68cd (127.9.30.67:34021): tablet bd510751b1cb4241a9f5541b94098e8c (table TestTable [id=24b010942db747f3a40e5b9065b4859e]) successfully deleted
W20250625 20:04:42.259688 9337 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bd510751b1cb4241a9f5541b94098e8c: tablet_id: "bd510751b1cb4241a9f5541b94098e8c" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250625 20:04:43.263299 9337 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bd510751b1cb4241a9f5541b94098e8c: tablet_id: "bd510751b1cb4241a9f5541b94098e8c" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250625 20:04:44.267493 9337 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bd510751b1cb4241a9f5541b94098e8c: tablet_id: "bd510751b1cb4241a9f5541b94098e8c" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250625 20:04:45.271528 9337 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bd510751b1cb4241a9f5541b94098e8c: tablet_id: "bd510751b1cb4241a9f5541b94098e8c" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250625 20:04:46.275475 9337 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bd510751b1cb4241a9f5541b94098e8c: tablet_id: "bd510751b1cb4241a9f5541b94098e8c" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250625 20:04:47.278781 9337 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bd510751b1cb4241a9f5541b94098e8c: tablet_id: "bd510751b1cb4241a9f5541b94098e8c" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250625 20:04:48.282203 9337 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bd510751b1cb4241a9f5541b94098e8c: tablet_id: "bd510751b1cb4241a9f5541b94098e8c" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250625 20:04:49.285604 9337 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bd510751b1cb4241a9f5541b94098e8c: tablet_id: "bd510751b1cb4241a9f5541b94098e8c" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250625 20:04:50.289041 9337 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bd510751b1cb4241a9f5541b94098e8c: tablet_id: "bd510751b1cb4241a9f5541b94098e8c" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250625 20:04:51.292433 9337 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bd510751b1cb4241a9f5541b94098e8c: tablet_id: "bd510751b1cb4241a9f5541b94098e8c" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250625 20:04:52.295764 9337 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bd510751b1cb4241a9f5541b94098e8c: tablet_id: "bd510751b1cb4241a9f5541b94098e8c" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250625 20:04:53.299235 9337 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bd510751b1cb4241a9f5541b94098e8c: tablet_id: "bd510751b1cb4241a9f5541b94098e8c" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250625 20:04:54.303495 9337 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bd510751b1cb4241a9f5541b94098e8c: tablet_id: "bd510751b1cb4241a9f5541b94098e8c" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250625 20:04:55.307585 9337 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bd510751b1cb4241a9f5541b94098e8c: tablet_id: "bd510751b1cb4241a9f5541b94098e8c" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250625 20:04:56.311331 9337 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bd510751b1cb4241a9f5541b94098e8c: tablet_id: "bd510751b1cb4241a9f5541b94098e8c" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250625 20:04:57.314873 9337 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bd510751b1cb4241a9f5541b94098e8c: tablet_id: "bd510751b1cb4241a9f5541b94098e8c" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250625 20:04:58.318395 9337 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bd510751b1cb4241a9f5541b94098e8c: tablet_id: "bd510751b1cb4241a9f5541b94098e8c" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250625 20:04:59.321731 9337 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bd510751b1cb4241a9f5541b94098e8c: tablet_id: "bd510751b1cb4241a9f5541b94098e8c" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250625 20:05:00.325089 9337 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet bd510751b1cb4241a9f5541b94098e8c: tablet_id: "bd510751b1cb4241a9f5541b94098e8c" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/kudu-admin-test.cc:3914: Failure
Failed
Bad status: Not found: not all replicas of tablets comprising table TestTable are registered yet
I20250625 20:05:01.328502 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 13351
I20250625 20:05:01.351477 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 13506
I20250625 20:05:01.372975 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 13653
I20250625 20:05:01.396289 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 13280
2025-06-25T20:05:01Z chronyd exiting
I20250625 20:05:01.443161 9337 test_util.cc:183] -----------------------------------------------
I20250625 20:05:01.443395 9337 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750881784340240-9337-0
[ FAILED ] AdminCliTest.TestRebuildTables (55909 ms)
[----------] 5 tests from AdminCliTest (117033 ms total)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest
[ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4
I20250625 20:05:01.447329 9337 test_util.cc:276] Using random seed: -171472763
I20250625 20:05:01.451710 9337 ts_itest-base.cc:115] Starting cluster with:
I20250625 20:05:01.451900 9337 ts_itest-base.cc:116] --------------
I20250625 20:05:01.452049 9337 ts_itest-base.cc:117] 5 tablet servers
I20250625 20:05:01.452183 9337 ts_itest-base.cc:118] 3 replicas per TS
I20250625 20:05:01.452324 9337 ts_itest-base.cc:119] --------------
2025-06-25T20:05:01Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-25T20:05:01Z Disabled control of system clock
I20250625 20:05:01.494120 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.30.126:43057
--webserver_interface=127.9.30.126
--webserver_port=0
--builtin_ntp_servers=127.9.30.84:43703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.30.126:43057
--raft_prepare_replacement_before_eviction=true with env {}
W20250625 20:05:01.792534 13851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:05:01.793134 13851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:05:01.793607 13851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:05:01.824846 13851 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250625 20:05:01.825246 13851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 20:05:01.825506 13851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:05:01.825744 13851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 20:05:01.825973 13851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 20:05:01.861462 13851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:43703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.30.126:43057
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.30.126:43057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.9.30.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:05:01.862841 13851 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:05:01.864591 13851 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:05:01.878352 13857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:05:01.878357 13858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:05:03.001143 13860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:05:03.003531 13859 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1122 milliseconds
I20250625 20:05:03.003660 13851 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 20:05:03.004868 13851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:05:03.007431 13851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:05:03.008860 13851 hybrid_clock.cc:648] HybridClock initialized: now 1750881903008832 us; error 38 us; skew 500 ppm
I20250625 20:05:03.009675 13851 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:05:03.019575 13851 webserver.cc:469] Webserver started at http://127.9.30.126:43403/ using document root <none> and password file <none>
I20250625 20:05:03.020913 13851 fs_manager.cc:362] Metadata directory not provided
I20250625 20:05:03.021205 13851 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:05:03.021842 13851 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 20:05:03.026906 13851 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "75210cdedfae4718bb4cfdf49914c7be"
format_stamp: "Formatted at 2025-06-25 20:05:03 on dist-test-slave-jhxp"
I20250625 20:05:03.028118 13851 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "75210cdedfae4718bb4cfdf49914c7be"
format_stamp: "Formatted at 2025-06-25 20:05:03 on dist-test-slave-jhxp"
I20250625 20:05:03.035395 13851 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250625 20:05:03.041088 13867 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:05:03.042176 13851 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.005s
I20250625 20:05:03.042539 13851 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
uuid: "75210cdedfae4718bb4cfdf49914c7be"
format_stamp: "Formatted at 2025-06-25 20:05:03 on dist-test-slave-jhxp"
I20250625 20:05:03.042883 13851 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:05:03.095423 13851 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:05:03.096961 13851 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:05:03.097421 13851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:05:03.167354 13851 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.126:43057
I20250625 20:05:03.167431 13918 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.126:43057 every 8 connection(s)
I20250625 20:05:03.170109 13851 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250625 20:05:03.175284 13919 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:05:03.177691 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 13851
I20250625 20:05:03.178217 9337 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250625 20:05:03.197508 13919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75210cdedfae4718bb4cfdf49914c7be: Bootstrap starting.
I20250625 20:05:03.203845 13919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75210cdedfae4718bb4cfdf49914c7be: Neither blocks nor log segments found. Creating new log.
I20250625 20:05:03.205750 13919 log.cc:826] T 00000000000000000000000000000000 P 75210cdedfae4718bb4cfdf49914c7be: Log is configured to *not* fsync() on all Append() calls
I20250625 20:05:03.210354 13919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75210cdedfae4718bb4cfdf49914c7be: No bootstrap required, opened a new log
I20250625 20:05:03.228163 13919 raft_consensus.cc:357] T 00000000000000000000000000000000 P 75210cdedfae4718bb4cfdf49914c7be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75210cdedfae4718bb4cfdf49914c7be" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 43057 } }
I20250625 20:05:03.228853 13919 raft_consensus.cc:383] T 00000000000000000000000000000000 P 75210cdedfae4718bb4cfdf49914c7be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:05:03.229085 13919 raft_consensus.cc:738] T 00000000000000000000000000000000 P 75210cdedfae4718bb4cfdf49914c7be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75210cdedfae4718bb4cfdf49914c7be, State: Initialized, Role: FOLLOWER
I20250625 20:05:03.229735 13919 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75210cdedfae4718bb4cfdf49914c7be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75210cdedfae4718bb4cfdf49914c7be" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 43057 } }
I20250625 20:05:03.230278 13919 raft_consensus.cc:397] T 00000000000000000000000000000000 P 75210cdedfae4718bb4cfdf49914c7be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 20:05:03.230545 13919 raft_consensus.cc:491] T 00000000000000000000000000000000 P 75210cdedfae4718bb4cfdf49914c7be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 20:05:03.230860 13919 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 75210cdedfae4718bb4cfdf49914c7be [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:05:03.234975 13919 raft_consensus.cc:513] T 00000000000000000000000000000000 P 75210cdedfae4718bb4cfdf49914c7be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75210cdedfae4718bb4cfdf49914c7be" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 43057 } }
I20250625 20:05:03.235780 13919 leader_election.cc:304] T 00000000000000000000000000000000 P 75210cdedfae4718bb4cfdf49914c7be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75210cdedfae4718bb4cfdf49914c7be; no voters:
I20250625 20:05:03.237695 13919 leader_election.cc:290] T 00000000000000000000000000000000 P 75210cdedfae4718bb4cfdf49914c7be [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 20:05:03.238498 13924 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 75210cdedfae4718bb4cfdf49914c7be [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:05:03.240584 13924 raft_consensus.cc:695] T 00000000000000000000000000000000 P 75210cdedfae4718bb4cfdf49914c7be [term 1 LEADER]: Becoming Leader. State: Replica: 75210cdedfae4718bb4cfdf49914c7be, State: Running, Role: LEADER
I20250625 20:05:03.241304 13924 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75210cdedfae4718bb4cfdf49914c7be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75210cdedfae4718bb4cfdf49914c7be" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 43057 } }
I20250625 20:05:03.241695 13919 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75210cdedfae4718bb4cfdf49914c7be [sys.catalog]: configured and running, proceeding with master startup.
I20250625 20:05:03.253309 13926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75210cdedfae4718bb4cfdf49914c7be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75210cdedfae4718bb4cfdf49914c7be. Latest consensus state: current_term: 1 leader_uuid: "75210cdedfae4718bb4cfdf49914c7be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75210cdedfae4718bb4cfdf49914c7be" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 43057 } } }
I20250625 20:05:03.254081 13926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75210cdedfae4718bb4cfdf49914c7be [sys.catalog]: This master's current role is: LEADER
I20250625 20:05:03.254541 13925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75210cdedfae4718bb4cfdf49914c7be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75210cdedfae4718bb4cfdf49914c7be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75210cdedfae4718bb4cfdf49914c7be" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 43057 } } }
I20250625 20:05:03.255198 13925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75210cdedfae4718bb4cfdf49914c7be [sys.catalog]: This master's current role is: LEADER
I20250625 20:05:03.257079 13934 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 20:05:03.269196 13934 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 20:05:03.286199 13934 catalog_manager.cc:1349] Generated new cluster ID: 45f17c31b5ee427f8dc85a5c7dd616ac
I20250625 20:05:03.286461 13934 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 20:05:03.319017 13934 catalog_manager.cc:1372] Generated new certificate authority record
I20250625 20:05:03.320559 13934 catalog_manager.cc:1506] Loading token signing keys...
I20250625 20:05:03.334506 13934 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 75210cdedfae4718bb4cfdf49914c7be: Generated new TSK 0
I20250625 20:05:03.335511 13934 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 20:05:03.344841 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.30.65:0
--local_ip_for_outbound_sockets=127.9.30.65
--webserver_interface=127.9.30.65
--webserver_port=0
--tserver_master_addrs=127.9.30.126:43057
--builtin_ntp_servers=127.9.30.84:43703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250625 20:05:03.671136 13943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:05:03.671691 13943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:05:03.672202 13943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:05:03.704245 13943 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250625 20:05:03.704660 13943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:05:03.705457 13943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.30.65
I20250625 20:05:03.740852 13943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:43703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.30.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.9.30.65
--webserver_port=0
--tserver_master_addrs=127.9.30.126:43057
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.30.65
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:05:03.742177 13943 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:05:03.744000 13943 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:05:03.761485 13949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:05:03.762851 13952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:05:03.763185 13950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:05:03.763294 13943 server_base.cc:1048] running on GCE node
I20250625 20:05:04.872689 13943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:05:04.875447 13943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:05:04.876957 13943 hybrid_clock.cc:648] HybridClock initialized: now 1750881904876880 us; error 81 us; skew 500 ppm
I20250625 20:05:04.878053 13943 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:05:04.885854 13943 webserver.cc:469] Webserver started at http://127.9.30.65:42651/ using document root <none> and password file <none>
I20250625 20:05:04.887153 13943 fs_manager.cc:362] Metadata directory not provided
I20250625 20:05:04.887442 13943 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:05:04.888068 13943 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 20:05:04.896234 13943 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "f6e300f0d75644c291f12e50a8228f55"
format_stamp: "Formatted at 2025-06-25 20:05:04 on dist-test-slave-jhxp"
I20250625 20:05:04.897843 13943 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "f6e300f0d75644c291f12e50a8228f55"
format_stamp: "Formatted at 2025-06-25 20:05:04 on dist-test-slave-jhxp"
I20250625 20:05:04.907361 13943 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.006s sys 0.005s
I20250625 20:05:04.914811 13960 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:05:04.916072 13943 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250625 20:05:04.916498 13943 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f6e300f0d75644c291f12e50a8228f55"
format_stamp: "Formatted at 2025-06-25 20:05:04 on dist-test-slave-jhxp"
I20250625 20:05:04.916935 13943 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:05:04.982651 13943 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:05:04.984220 13943 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:05:04.984675 13943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:05:04.987272 13943 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 20:05:04.991513 13943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 20:05:04.991798 13943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250625 20:05:04.992038 13943 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 20:05:04.992197 13943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 20:05:05.142027 13943 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.65:41333
I20250625 20:05:05.142089 14072 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.65:41333 every 8 connection(s)
I20250625 20:05:05.145519 13943 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250625 20:05:05.155671 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 13943
I20250625 20:05:05.156436 9337 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250625 20:05:05.164390 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.30.66:0
--local_ip_for_outbound_sockets=127.9.30.66
--webserver_interface=127.9.30.66
--webserver_port=0
--tserver_master_addrs=127.9.30.126:43057
--builtin_ntp_servers=127.9.30.84:43703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250625 20:05:05.169804 14073 heartbeater.cc:344] Connected to a master server at 127.9.30.126:43057
I20250625 20:05:05.170339 14073 heartbeater.cc:461] Registering TS with master...
I20250625 20:05:05.171653 14073 heartbeater.cc:507] Master 127.9.30.126:43057 requested a full tablet report, sending...
I20250625 20:05:05.174764 13884 ts_manager.cc:194] Registered new tserver with Master: f6e300f0d75644c291f12e50a8228f55 (127.9.30.65:41333)
I20250625 20:05:05.177685 13884 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.30.65:48911
W20250625 20:05:05.463966 14077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:05:05.464483 14077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:05:05.464985 14077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:05:05.496690 14077 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250625 20:05:05.497102 14077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:05:05.497892 14077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.30.66
I20250625 20:05:05.532995 14077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:43703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.30.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.9.30.66
--webserver_port=0
--tserver_master_addrs=127.9.30.126:43057
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.30.66
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:05:05.534338 14077 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:05:05.536039 14077 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:05:05.553017 14084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:05:06.182498 14073 heartbeater.cc:499] Master 127.9.30.126:43057 was elected leader, sending a full tablet report...
W20250625 20:05:05.553025 14083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:05:05.553067 14086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:05:06.619637 14085 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250625 20:05:06.619761 14077 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 20:05:06.623358 14077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:05:06.625599 14077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:05:06.626962 14077 hybrid_clock.cc:648] HybridClock initialized: now 1750881906626921 us; error 68 us; skew 500 ppm
I20250625 20:05:06.627818 14077 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:05:06.634403 14077 webserver.cc:469] Webserver started at http://127.9.30.66:33281/ using document root <none> and password file <none>
I20250625 20:05:06.635349 14077 fs_manager.cc:362] Metadata directory not provided
I20250625 20:05:06.635576 14077 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:05:06.636065 14077 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 20:05:06.641142 14077 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "20448e59f60a4266ab32666d1731cfcf"
format_stamp: "Formatted at 2025-06-25 20:05:06 on dist-test-slave-jhxp"
I20250625 20:05:06.642403 14077 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "20448e59f60a4266ab32666d1731cfcf"
format_stamp: "Formatted at 2025-06-25 20:05:06 on dist-test-slave-jhxp"
I20250625 20:05:06.649806 14077 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20250625 20:05:06.655949 14093 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:05:06.657114 14077 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250625 20:05:06.657449 14077 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "20448e59f60a4266ab32666d1731cfcf"
format_stamp: "Formatted at 2025-06-25 20:05:06 on dist-test-slave-jhxp"
I20250625 20:05:06.657805 14077 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:05:06.705714 14077 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:05:06.707214 14077 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:05:06.707711 14077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:05:06.710194 14077 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 20:05:06.714311 14077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 20:05:06.714551 14077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 20:05:06.714800 14077 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 20:05:06.714960 14077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 20:05:06.846709 14077 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.66:38899
I20250625 20:05:06.846824 14205 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.66:38899 every 8 connection(s)
I20250625 20:05:06.849392 14077 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250625 20:05:06.859818 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 14077
I20250625 20:05:06.860234 9337 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250625 20:05:06.866458 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.30.67:0
--local_ip_for_outbound_sockets=127.9.30.67
--webserver_interface=127.9.30.67
--webserver_port=0
--tserver_master_addrs=127.9.30.126:43057
--builtin_ntp_servers=127.9.30.84:43703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250625 20:05:06.872205 14206 heartbeater.cc:344] Connected to a master server at 127.9.30.126:43057
I20250625 20:05:06.872694 14206 heartbeater.cc:461] Registering TS with master...
I20250625 20:05:06.873786 14206 heartbeater.cc:507] Master 127.9.30.126:43057 requested a full tablet report, sending...
I20250625 20:05:06.876152 13884 ts_manager.cc:194] Registered new tserver with Master: 20448e59f60a4266ab32666d1731cfcf (127.9.30.66:38899)
I20250625 20:05:06.877488 13884 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.30.66:60325
W20250625 20:05:07.176091 14210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:05:07.176620 14210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:05:07.177120 14210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:05:07.209370 14210 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250625 20:05:07.209810 14210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:05:07.210681 14210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.30.67
I20250625 20:05:07.246052 14210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:43703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.30.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.9.30.67
--webserver_port=0
--tserver_master_addrs=127.9.30.126:43057
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.30.67
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:05:07.247543 14210 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:05:07.249495 14210 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:05:07.265498 14216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:05:07.880971 14206 heartbeater.cc:499] Master 127.9.30.126:43057 was elected leader, sending a full tablet report...
W20250625 20:05:07.265547 14217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:05:07.267634 14210 server_base.cc:1048] running on GCE node
W20250625 20:05:07.267205 14219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:05:08.343245 14210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:05:08.345832 14210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:05:08.347221 14210 hybrid_clock.cc:648] HybridClock initialized: now 1750881908347179 us; error 57 us; skew 500 ppm
I20250625 20:05:08.348093 14210 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:05:08.359344 14210 webserver.cc:469] Webserver started at http://127.9.30.67:37463/ using document root <none> and password file <none>
I20250625 20:05:08.360342 14210 fs_manager.cc:362] Metadata directory not provided
I20250625 20:05:08.360563 14210 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:05:08.360987 14210 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 20:05:08.365543 14210 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "e60cb07b06cb4f6c8e895c28a13eee04"
format_stamp: "Formatted at 2025-06-25 20:05:08 on dist-test-slave-jhxp"
I20250625 20:05:08.366703 14210 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "e60cb07b06cb4f6c8e895c28a13eee04"
format_stamp: "Formatted at 2025-06-25 20:05:08 on dist-test-slave-jhxp"
I20250625 20:05:08.374380 14210 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250625 20:05:08.380275 14226 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:05:08.381363 14210 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250625 20:05:08.381717 14210 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e60cb07b06cb4f6c8e895c28a13eee04"
format_stamp: "Formatted at 2025-06-25 20:05:08 on dist-test-slave-jhxp"
I20250625 20:05:08.382037 14210 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:05:08.437916 14210 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:05:08.439342 14210 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:05:08.439805 14210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:05:08.442243 14210 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 20:05:08.446396 14210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 20:05:08.446617 14210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 20:05:08.446887 14210 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 20:05:08.447041 14210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 20:05:08.577790 14210 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.67:42977
I20250625 20:05:08.577906 14338 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.67:42977 every 8 connection(s)
I20250625 20:05:08.580372 14210 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250625 20:05:08.589685 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 14210
I20250625 20:05:08.590231 9337 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250625 20:05:08.597301 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.30.68:0
--local_ip_for_outbound_sockets=127.9.30.68
--webserver_interface=127.9.30.68
--webserver_port=0
--tserver_master_addrs=127.9.30.126:43057
--builtin_ntp_servers=127.9.30.84:43703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250625 20:05:08.603374 14339 heartbeater.cc:344] Connected to a master server at 127.9.30.126:43057
I20250625 20:05:08.603935 14339 heartbeater.cc:461] Registering TS with master...
I20250625 20:05:08.605199 14339 heartbeater.cc:507] Master 127.9.30.126:43057 requested a full tablet report, sending...
I20250625 20:05:08.607833 13884 ts_manager.cc:194] Registered new tserver with Master: e60cb07b06cb4f6c8e895c28a13eee04 (127.9.30.67:42977)
I20250625 20:05:08.609469 13884 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.30.67:35571
W20250625 20:05:08.897723 14343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:05:08.898237 14343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:05:08.898725 14343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:05:08.931097 14343 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250625 20:05:08.931497 14343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:05:08.932263 14343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.30.68
I20250625 20:05:08.967646 14343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:43703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.30.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.9.30.68
--webserver_port=0
--tserver_master_addrs=127.9.30.126:43057
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.30.68
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:05:08.968900 14343 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:05:08.970533 14343 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:05:08.985555 14349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:05:09.612789 14339 heartbeater.cc:499] Master 127.9.30.126:43057 was elected leader, sending a full tablet report...
W20250625 20:05:08.990648 14350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:05:08.992048 14343 server_base.cc:1048] running on GCE node
W20250625 20:05:08.992161 14352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:05:10.070237 14343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:05:10.072932 14343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:05:10.074381 14343 hybrid_clock.cc:648] HybridClock initialized: now 1750881910074334 us; error 61 us; skew 500 ppm
I20250625 20:05:10.075212 14343 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:05:10.081979 14343 webserver.cc:469] Webserver started at http://127.9.30.68:43907/ using document root <none> and password file <none>
I20250625 20:05:10.082937 14343 fs_manager.cc:362] Metadata directory not provided
I20250625 20:05:10.083160 14343 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:05:10.083683 14343 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 20:05:10.090051 14343 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "aec86bddcc9e49a682530f61c3146ef2"
format_stamp: "Formatted at 2025-06-25 20:05:10 on dist-test-slave-jhxp"
I20250625 20:05:10.091224 14343 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "aec86bddcc9e49a682530f61c3146ef2"
format_stamp: "Formatted at 2025-06-25 20:05:10 on dist-test-slave-jhxp"
I20250625 20:05:10.098789 14343 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20250625 20:05:10.104593 14359 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:05:10.105723 14343 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250625 20:05:10.106069 14343 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "aec86bddcc9e49a682530f61c3146ef2"
format_stamp: "Formatted at 2025-06-25 20:05:10 on dist-test-slave-jhxp"
I20250625 20:05:10.106400 14343 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:05:10.161221 14343 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:05:10.162765 14343 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:05:10.163194 14343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:05:10.166112 14343 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 20:05:10.170539 14343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 20:05:10.170776 14343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 20:05:10.171016 14343 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 20:05:10.171168 14343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 20:05:10.306164 14343 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.68:34641
I20250625 20:05:10.306269 14471 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.68:34641 every 8 connection(s)
I20250625 20:05:10.308797 14343 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250625 20:05:10.311553 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 14343
I20250625 20:05:10.312078 9337 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250625 20:05:10.319180 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.30.69:0
--local_ip_for_outbound_sockets=127.9.30.69
--webserver_interface=127.9.30.69
--webserver_port=0
--tserver_master_addrs=127.9.30.126:43057
--builtin_ntp_servers=127.9.30.84:43703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250625 20:05:10.331024 14472 heartbeater.cc:344] Connected to a master server at 127.9.30.126:43057
I20250625 20:05:10.331530 14472 heartbeater.cc:461] Registering TS with master...
I20250625 20:05:10.332617 14472 heartbeater.cc:507] Master 127.9.30.126:43057 requested a full tablet report, sending...
I20250625 20:05:10.334977 13884 ts_manager.cc:194] Registered new tserver with Master: aec86bddcc9e49a682530f61c3146ef2 (127.9.30.68:34641)
I20250625 20:05:10.336262 13884 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.30.68:37197
W20250625 20:05:10.621297 14476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:05:10.621764 14476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:05:10.622212 14476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:05:10.654840 14476 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250625 20:05:10.655365 14476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:05:10.656237 14476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.30.69
I20250625 20:05:10.691390 14476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:43703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.30.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.9.30.69
--webserver_port=0
--tserver_master_addrs=127.9.30.126:43057
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.30.69
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:05:10.692781 14476 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:05:10.694520 14476 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:05:10.710150 14482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:05:11.339799 14472 heartbeater.cc:499] Master 127.9.30.126:43057 was elected leader, sending a full tablet report...
W20250625 20:05:10.710531 14483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:05:10.711771 14485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:05:11.808902 14484 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250625 20:05:11.808952 14476 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 20:05:11.812515 14476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:05:11.814739 14476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:05:11.816098 14476 hybrid_clock.cc:648] HybridClock initialized: now 1750881911816052 us; error 65 us; skew 500 ppm
I20250625 20:05:11.816916 14476 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:05:11.823104 14476 webserver.cc:469] Webserver started at http://127.9.30.69:37189/ using document root <none> and password file <none>
I20250625 20:05:11.824095 14476 fs_manager.cc:362] Metadata directory not provided
I20250625 20:05:11.824314 14476 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:05:11.824776 14476 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 20:05:11.829525 14476 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "96f0c9a340d443c6a95e1d5e0c442d9c"
format_stamp: "Formatted at 2025-06-25 20:05:11 on dist-test-slave-jhxp"
I20250625 20:05:11.830653 14476 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "96f0c9a340d443c6a95e1d5e0c442d9c"
format_stamp: "Formatted at 2025-06-25 20:05:11 on dist-test-slave-jhxp"
I20250625 20:05:11.837970 14476 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250625 20:05:11.843907 14493 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:05:11.845148 14476 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250625 20:05:11.845620 14476 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "96f0c9a340d443c6a95e1d5e0c442d9c"
format_stamp: "Formatted at 2025-06-25 20:05:11 on dist-test-slave-jhxp"
I20250625 20:05:11.846060 14476 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:05:11.892234 14476 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:05:11.893704 14476 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:05:11.894127 14476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:05:11.896917 14476 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 20:05:11.901312 14476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 20:05:11.901543 14476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 20:05:11.901813 14476 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 20:05:11.901964 14476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 20:05:12.048938 14476 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.69:38029
I20250625 20:05:12.049041 14605 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.69:38029 every 8 connection(s)
I20250625 20:05:12.051855 14476 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250625 20:05:12.053191 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 14476
I20250625 20:05:12.053725 9337 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250625 20:05:12.081161 14606 heartbeater.cc:344] Connected to a master server at 127.9.30.126:43057
I20250625 20:05:12.081679 14606 heartbeater.cc:461] Registering TS with master...
I20250625 20:05:12.082723 14606 heartbeater.cc:507] Master 127.9.30.126:43057 requested a full tablet report, sending...
I20250625 20:05:12.085057 13884 ts_manager.cc:194] Registered new tserver with Master: 96f0c9a340d443c6a95e1d5e0c442d9c (127.9.30.69:38029)
I20250625 20:05:12.086469 13884 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.30.69:49641
I20250625 20:05:12.096519 9337 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250625 20:05:12.135804 13884 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43634:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250625 20:05:12.234980 14141 tablet_service.cc:1468] Processing CreateTablet for tablet 84bcadd235fc4e27a1d81811d11687f5 (DEFAULT_TABLE table=TestTable [id=a426c26d09e24cadbc126173ea91bca4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:05:12.238605 14141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84bcadd235fc4e27a1d81811d11687f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:05:12.241936 14541 tablet_service.cc:1468] Processing CreateTablet for tablet 84bcadd235fc4e27a1d81811d11687f5 (DEFAULT_TABLE table=TestTable [id=a426c26d09e24cadbc126173ea91bca4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:05:12.242332 14008 tablet_service.cc:1468] Processing CreateTablet for tablet 84bcadd235fc4e27a1d81811d11687f5 (DEFAULT_TABLE table=TestTable [id=a426c26d09e24cadbc126173ea91bca4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:05:12.244063 14541 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84bcadd235fc4e27a1d81811d11687f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:05:12.244614 14008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84bcadd235fc4e27a1d81811d11687f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:05:12.269954 14625 tablet_bootstrap.cc:492] T 84bcadd235fc4e27a1d81811d11687f5 P 96f0c9a340d443c6a95e1d5e0c442d9c: Bootstrap starting.
I20250625 20:05:12.272888 14626 tablet_bootstrap.cc:492] T 84bcadd235fc4e27a1d81811d11687f5 P 20448e59f60a4266ab32666d1731cfcf: Bootstrap starting.
I20250625 20:05:12.276887 14627 tablet_bootstrap.cc:492] T 84bcadd235fc4e27a1d81811d11687f5 P f6e300f0d75644c291f12e50a8228f55: Bootstrap starting.
I20250625 20:05:12.281072 14625 tablet_bootstrap.cc:654] T 84bcadd235fc4e27a1d81811d11687f5 P 96f0c9a340d443c6a95e1d5e0c442d9c: Neither blocks nor log segments found. Creating new log.
I20250625 20:05:12.282310 14626 tablet_bootstrap.cc:654] T 84bcadd235fc4e27a1d81811d11687f5 P 20448e59f60a4266ab32666d1731cfcf: Neither blocks nor log segments found. Creating new log.
I20250625 20:05:12.282920 14625 log.cc:826] T 84bcadd235fc4e27a1d81811d11687f5 P 96f0c9a340d443c6a95e1d5e0c442d9c: Log is configured to *not* fsync() on all Append() calls
I20250625 20:05:12.285518 14627 tablet_bootstrap.cc:654] T 84bcadd235fc4e27a1d81811d11687f5 P f6e300f0d75644c291f12e50a8228f55: Neither blocks nor log segments found. Creating new log.
I20250625 20:05:12.285887 14626 log.cc:826] T 84bcadd235fc4e27a1d81811d11687f5 P 20448e59f60a4266ab32666d1731cfcf: Log is configured to *not* fsync() on all Append() calls
I20250625 20:05:12.288182 14627 log.cc:826] T 84bcadd235fc4e27a1d81811d11687f5 P f6e300f0d75644c291f12e50a8228f55: Log is configured to *not* fsync() on all Append() calls
I20250625 20:05:12.288973 14625 tablet_bootstrap.cc:492] T 84bcadd235fc4e27a1d81811d11687f5 P 96f0c9a340d443c6a95e1d5e0c442d9c: No bootstrap required, opened a new log
I20250625 20:05:12.289575 14625 ts_tablet_manager.cc:1397] T 84bcadd235fc4e27a1d81811d11687f5 P 96f0c9a340d443c6a95e1d5e0c442d9c: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.005s
I20250625 20:05:12.292518 14626 tablet_bootstrap.cc:492] T 84bcadd235fc4e27a1d81811d11687f5 P 20448e59f60a4266ab32666d1731cfcf: No bootstrap required, opened a new log
I20250625 20:05:12.293187 14626 ts_tablet_manager.cc:1397] T 84bcadd235fc4e27a1d81811d11687f5 P 20448e59f60a4266ab32666d1731cfcf: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.010s
I20250625 20:05:12.295188 14627 tablet_bootstrap.cc:492] T 84bcadd235fc4e27a1d81811d11687f5 P f6e300f0d75644c291f12e50a8228f55: No bootstrap required, opened a new log
I20250625 20:05:12.295684 14627 ts_tablet_manager.cc:1397] T 84bcadd235fc4e27a1d81811d11687f5 P f6e300f0d75644c291f12e50a8228f55: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.003s
I20250625 20:05:12.318866 14625 raft_consensus.cc:357] T 84bcadd235fc4e27a1d81811d11687f5 P 96f0c9a340d443c6a95e1d5e0c442d9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20448e59f60a4266ab32666d1731cfcf" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 38899 } } peers { permanent_uuid: "96f0c9a340d443c6a95e1d5e0c442d9c" member_type: VOTER last_known_addr { host: "127.9.30.69" port: 38029 } } peers { permanent_uuid: "f6e300f0d75644c291f12e50a8228f55" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 41333 } }
I20250625 20:05:12.319977 14625 raft_consensus.cc:383] T 84bcadd235fc4e27a1d81811d11687f5 P 96f0c9a340d443c6a95e1d5e0c442d9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:05:12.320328 14625 raft_consensus.cc:738] T 84bcadd235fc4e27a1d81811d11687f5 P 96f0c9a340d443c6a95e1d5e0c442d9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96f0c9a340d443c6a95e1d5e0c442d9c, State: Initialized, Role: FOLLOWER
I20250625 20:05:12.321254 14627 raft_consensus.cc:357] T 84bcadd235fc4e27a1d81811d11687f5 P f6e300f0d75644c291f12e50a8228f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20448e59f60a4266ab32666d1731cfcf" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 38899 } } peers { permanent_uuid: "96f0c9a340d443c6a95e1d5e0c442d9c" member_type: VOTER last_known_addr { host: "127.9.30.69" port: 38029 } } peers { permanent_uuid: "f6e300f0d75644c291f12e50a8228f55" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 41333 } }
I20250625 20:05:12.321316 14625 consensus_queue.cc:260] T 84bcadd235fc4e27a1d81811d11687f5 P 96f0c9a340d443c6a95e1d5e0c442d9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20448e59f60a4266ab32666d1731cfcf" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 38899 } } peers { permanent_uuid: "96f0c9a340d443c6a95e1d5e0c442d9c" member_type: VOTER last_known_addr { host: "127.9.30.69" port: 38029 } } peers { permanent_uuid: "f6e300f0d75644c291f12e50a8228f55" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 41333 } }
I20250625 20:05:12.322201 14627 raft_consensus.cc:383] T 84bcadd235fc4e27a1d81811d11687f5 P f6e300f0d75644c291f12e50a8228f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:05:12.322583 14627 raft_consensus.cc:738] T 84bcadd235fc4e27a1d81811d11687f5 P f6e300f0d75644c291f12e50a8228f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6e300f0d75644c291f12e50a8228f55, State: Initialized, Role: FOLLOWER
I20250625 20:05:12.323779 14627 consensus_queue.cc:260] T 84bcadd235fc4e27a1d81811d11687f5 P f6e300f0d75644c291f12e50a8228f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20448e59f60a4266ab32666d1731cfcf" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 38899 } } peers { permanent_uuid: "96f0c9a340d443c6a95e1d5e0c442d9c" member_type: VOTER last_known_addr { host: "127.9.30.69" port: 38029 } } peers { permanent_uuid: "f6e300f0d75644c291f12e50a8228f55" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 41333 } }
I20250625 20:05:12.325917 14626 raft_consensus.cc:357] T 84bcadd235fc4e27a1d81811d11687f5 P 20448e59f60a4266ab32666d1731cfcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20448e59f60a4266ab32666d1731cfcf" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 38899 } } peers { permanent_uuid: "96f0c9a340d443c6a95e1d5e0c442d9c" member_type: VOTER last_known_addr { host: "127.9.30.69" port: 38029 } } peers { permanent_uuid: "f6e300f0d75644c291f12e50a8228f55" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 41333 } }
I20250625 20:05:12.327071 14626 raft_consensus.cc:383] T 84bcadd235fc4e27a1d81811d11687f5 P 20448e59f60a4266ab32666d1731cfcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:05:12.327457 14626 raft_consensus.cc:738] T 84bcadd235fc4e27a1d81811d11687f5 P 20448e59f60a4266ab32666d1731cfcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20448e59f60a4266ab32666d1731cfcf, State: Initialized, Role: FOLLOWER
I20250625 20:05:12.329125 14627 ts_tablet_manager.cc:1428] T 84bcadd235fc4e27a1d81811d11687f5 P f6e300f0d75644c291f12e50a8228f55: Time spent starting tablet: real 0.033s user 0.027s sys 0.004s
I20250625 20:05:12.329279 14626 consensus_queue.cc:260] T 84bcadd235fc4e27a1d81811d11687f5 P 20448e59f60a4266ab32666d1731cfcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20448e59f60a4266ab32666d1731cfcf" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 38899 } } peers { permanent_uuid: "96f0c9a340d443c6a95e1d5e0c442d9c" member_type: VOTER last_known_addr { host: "127.9.30.69" port: 38029 } } peers { permanent_uuid: "f6e300f0d75644c291f12e50a8228f55" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 41333 } }
I20250625 20:05:12.331143 14606 heartbeater.cc:499] Master 127.9.30.126:43057 was elected leader, sending a full tablet report...
I20250625 20:05:12.333004 14625 ts_tablet_manager.cc:1428] T 84bcadd235fc4e27a1d81811d11687f5 P 96f0c9a340d443c6a95e1d5e0c442d9c: Time spent starting tablet: real 0.043s user 0.034s sys 0.008s
I20250625 20:05:12.335911 14626 ts_tablet_manager.cc:1428] T 84bcadd235fc4e27a1d81811d11687f5 P 20448e59f60a4266ab32666d1731cfcf: Time spent starting tablet: real 0.042s user 0.031s sys 0.007s
W20250625 20:05:12.359459 14207 tablet.cc:2378] T 84bcadd235fc4e27a1d81811d11687f5 P 20448e59f60a4266ab32666d1731cfcf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 20:05:12.385097 14632 raft_consensus.cc:491] T 84bcadd235fc4e27a1d81811d11687f5 P f6e300f0d75644c291f12e50a8228f55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:05:12.386042 14632 raft_consensus.cc:513] T 84bcadd235fc4e27a1d81811d11687f5 P f6e300f0d75644c291f12e50a8228f55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20448e59f60a4266ab32666d1731cfcf" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 38899 } } peers { permanent_uuid: "96f0c9a340d443c6a95e1d5e0c442d9c" member_type: VOTER last_known_addr { host: "127.9.30.69" port: 38029 } } peers { permanent_uuid: "f6e300f0d75644c291f12e50a8228f55" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 41333 } }
I20250625 20:05:12.388329 14632 leader_election.cc:290] T 84bcadd235fc4e27a1d81811d11687f5 P f6e300f0d75644c291f12e50a8228f55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20448e59f60a4266ab32666d1731cfcf (127.9.30.66:38899), 96f0c9a340d443c6a95e1d5e0c442d9c (127.9.30.69:38029)
I20250625 20:05:12.401284 14161 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84bcadd235fc4e27a1d81811d11687f5" candidate_uuid: "f6e300f0d75644c291f12e50a8228f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20448e59f60a4266ab32666d1731cfcf" is_pre_election: true
I20250625 20:05:12.402125 14161 raft_consensus.cc:2466] T 84bcadd235fc4e27a1d81811d11687f5 P 20448e59f60a4266ab32666d1731cfcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6e300f0d75644c291f12e50a8228f55 in term 0.
I20250625 20:05:12.403292 13963 leader_election.cc:304] T 84bcadd235fc4e27a1d81811d11687f5 P f6e300f0d75644c291f12e50a8228f55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20448e59f60a4266ab32666d1731cfcf, f6e300f0d75644c291f12e50a8228f55; no voters:
I20250625 20:05:12.403291 14561 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84bcadd235fc4e27a1d81811d11687f5" candidate_uuid: "f6e300f0d75644c291f12e50a8228f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96f0c9a340d443c6a95e1d5e0c442d9c" is_pre_election: true
I20250625 20:05:12.403951 14561 raft_consensus.cc:2466] T 84bcadd235fc4e27a1d81811d11687f5 P 96f0c9a340d443c6a95e1d5e0c442d9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6e300f0d75644c291f12e50a8228f55 in term 0.
I20250625 20:05:12.404071 14632 raft_consensus.cc:2802] T 84bcadd235fc4e27a1d81811d11687f5 P f6e300f0d75644c291f12e50a8228f55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:05:12.404417 14632 raft_consensus.cc:491] T 84bcadd235fc4e27a1d81811d11687f5 P f6e300f0d75644c291f12e50a8228f55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:05:12.404739 14632 raft_consensus.cc:3058] T 84bcadd235fc4e27a1d81811d11687f5 P f6e300f0d75644c291f12e50a8228f55 [term 0 FOLLOWER]: Advancing to term 1
W20250625 20:05:12.407212 14074 tablet.cc:2378] T 84bcadd235fc4e27a1d81811d11687f5 P f6e300f0d75644c291f12e50a8228f55: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 20:05:12.409749 14632 raft_consensus.cc:513] T 84bcadd235fc4e27a1d81811d11687f5 P f6e300f0d75644c291f12e50a8228f55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20448e59f60a4266ab32666d1731cfcf" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 38899 } } peers { permanent_uuid: "96f0c9a340d443c6a95e1d5e0c442d9c" member_type: VOTER last_known_addr { host: "127.9.30.69" port: 38029 } } peers { permanent_uuid: "f6e300f0d75644c291f12e50a8228f55" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 41333 } }
I20250625 20:05:12.411721 14632 leader_election.cc:290] T 84bcadd235fc4e27a1d81811d11687f5 P f6e300f0d75644c291f12e50a8228f55 [CANDIDATE]: Term 1 election: Requested vote from peers 20448e59f60a4266ab32666d1731cfcf (127.9.30.66:38899), 96f0c9a340d443c6a95e1d5e0c442d9c (127.9.30.69:38029)
I20250625 20:05:12.412531 14161 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84bcadd235fc4e27a1d81811d11687f5" candidate_uuid: "f6e300f0d75644c291f12e50a8228f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20448e59f60a4266ab32666d1731cfcf"
I20250625 20:05:12.412701 14561 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84bcadd235fc4e27a1d81811d11687f5" candidate_uuid: "f6e300f0d75644c291f12e50a8228f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96f0c9a340d443c6a95e1d5e0c442d9c"
I20250625 20:05:12.412950 14161 raft_consensus.cc:3058] T 84bcadd235fc4e27a1d81811d11687f5 P 20448e59f60a4266ab32666d1731cfcf [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:05:12.413126 14561 raft_consensus.cc:3058] T 84bcadd235fc4e27a1d81811d11687f5 P 96f0c9a340d443c6a95e1d5e0c442d9c [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:05:12.417881 14561 raft_consensus.cc:2466] T 84bcadd235fc4e27a1d81811d11687f5 P 96f0c9a340d443c6a95e1d5e0c442d9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6e300f0d75644c291f12e50a8228f55 in term 1.
I20250625 20:05:12.417965 14161 raft_consensus.cc:2466] T 84bcadd235fc4e27a1d81811d11687f5 P 20448e59f60a4266ab32666d1731cfcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6e300f0d75644c291f12e50a8228f55 in term 1.
I20250625 20:05:12.418814 13963 leader_election.cc:304] T 84bcadd235fc4e27a1d81811d11687f5 P f6e300f0d75644c291f12e50a8228f55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96f0c9a340d443c6a95e1d5e0c442d9c, f6e300f0d75644c291f12e50a8228f55; no voters:
I20250625 20:05:12.419445 14632 raft_consensus.cc:2802] T 84bcadd235fc4e27a1d81811d11687f5 P f6e300f0d75644c291f12e50a8228f55 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:05:12.421412 14632 raft_consensus.cc:695] T 84bcadd235fc4e27a1d81811d11687f5 P f6e300f0d75644c291f12e50a8228f55 [term 1 LEADER]: Becoming Leader. State: Replica: f6e300f0d75644c291f12e50a8228f55, State: Running, Role: LEADER
I20250625 20:05:12.422164 14632 consensus_queue.cc:237] T 84bcadd235fc4e27a1d81811d11687f5 P f6e300f0d75644c291f12e50a8228f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20448e59f60a4266ab32666d1731cfcf" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 38899 } } peers { permanent_uuid: "96f0c9a340d443c6a95e1d5e0c442d9c" member_type: VOTER last_known_addr { host: "127.9.30.69" port: 38029 } } peers { permanent_uuid: "f6e300f0d75644c291f12e50a8228f55" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 41333 } }
I20250625 20:05:12.434507 13883 catalog_manager.cc:5582] T 84bcadd235fc4e27a1d81811d11687f5 P f6e300f0d75644c291f12e50a8228f55 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6e300f0d75644c291f12e50a8228f55 (127.9.30.65). New cstate: current_term: 1 leader_uuid: "f6e300f0d75644c291f12e50a8228f55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20448e59f60a4266ab32666d1731cfcf" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 38899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96f0c9a340d443c6a95e1d5e0c442d9c" member_type: VOTER last_known_addr { host: "127.9.30.69" port: 38029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6e300f0d75644c291f12e50a8228f55" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 41333 } health_report { overall_health: HEALTHY } } }
I20250625 20:05:12.466043 9337 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250625 20:05:12.469506 9337 ts_itest-base.cc:246] Waiting for 1 tablets on tserver f6e300f0d75644c291f12e50a8228f55 to finish bootstrapping
I20250625 20:05:12.484556 9337 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 20448e59f60a4266ab32666d1731cfcf to finish bootstrapping
I20250625 20:05:12.496496 9337 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 96f0c9a340d443c6a95e1d5e0c442d9c to finish bootstrapping
I20250625 20:05:12.507972 9337 test_util.cc:276] Using random seed: -160412117
I20250625 20:05:12.535187 9337 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20250625 20:05:12.536126 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 13943
W20250625 20:05:12.558183 14607 tablet.cc:2378] T 84bcadd235fc4e27a1d81811d11687f5 P 96f0c9a340d443c6a95e1d5e0c442d9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250625 20:05:12.560957 14643 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 20:05:12.548020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.30.65:41333 (local address 127.0.0.1:42090)
0625 20:05:12.548766 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0625 20:05:12.548814 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0625 20:05:12.549114 (+ 300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0625 20:05:12.559871 (+ 10757us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.30.65:41333: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":91}
W20250625 20:05:12.561581 14641 meta_cache.cc:302] tablet 84bcadd235fc4e27a1d81811d11687f5: replica f6e300f0d75644c291f12e50a8228f55 (127.9.30.65:41333) has failed: Network error: Client connection negotiation failed: client connection to 127.9.30.65:41333: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 20:05:12.591378 14121 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:52922: Illegal state: replica 20448e59f60a4266ab32666d1731cfcf is not leader of this config: current role FOLLOWER
W20250625 20:05:12.609642 14521 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:32956: Illegal state: replica 96f0c9a340d443c6a95e1d5e0c442d9c is not leader of this config: current role FOLLOWER
W20250625 20:05:12.634128 14121 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:52922: Illegal state: replica 20448e59f60a4266ab32666d1731cfcf is not leader of this config: current role FOLLOWER
W20250625 20:05:12.645905 14521 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:32956: Illegal state: replica 96f0c9a340d443c6a95e1d5e0c442d9c is not leader of this config: current role FOLLOWER
W20250625 20:05:12.676227 14121 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:52922: Illegal state: replica 20448e59f60a4266ab32666d1731cfcf is not leader of this config: current role FOLLOWER
W20250625 20:05:12.688356 14521 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:32956: Illegal state: replica 96f0c9a340d443c6a95e1d5e0c442d9c is not leader of this config: current role FOLLOWER
W20250625 20:05:12.718307 14121 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:52922: Illegal state: replica 20448e59f60a4266ab32666d1731cfcf is not leader of this config: current role FOLLOWER
W20250625 20:05:12.732964 14521 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:32956: Illegal state: replica 96f0c9a340d443c6a95e1d5e0c442d9c is not leader of this config: current role FOLLOWER
W20250625 20:05:12.771999 14121 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:52922: Illegal state: replica 20448e59f60a4266ab32666d1731cfcf is not leader of this config: current role FOLLOWER
W20250625 20:05:12.793685 14521 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:32956: Illegal state: replica 96f0c9a340d443c6a95e1d5e0c442d9c is not leader of this config: current role FOLLOWER
W20250625 20:05:12.839215 14121 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:52922: Illegal state: replica 20448e59f60a4266ab32666d1731cfcf is not leader of this config: current role FOLLOWER
W20250625 20:05:12.859709 14521 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:32956: Illegal state: replica 96f0c9a340d443c6a95e1d5e0c442d9c is not leader of this config: current role FOLLOWER
W20250625 20:05:12.908967 14121 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:52922: Illegal state: replica 20448e59f60a4266ab32666d1731cfcf is not leader of this config: current role FOLLOWER
W20250625 20:05:12.933488 14521 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:32956: Illegal state: replica 96f0c9a340d443c6a95e1d5e0c442d9c is not leader of this config: current role FOLLOWER
W20250625 20:05:12.988523 14121 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:52922: Illegal state: replica 20448e59f60a4266ab32666d1731cfcf is not leader of this config: current role FOLLOWER
W20250625 20:05:13.017055 14521 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:32956: Illegal state: replica 96f0c9a340d443c6a95e1d5e0c442d9c is not leader of this config: current role FOLLOWER
W20250625 20:05:13.087076 14121 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:52922: Illegal state: replica 20448e59f60a4266ab32666d1731cfcf is not leader of this config: current role FOLLOWER
W20250625 20:05:13.120889 14521 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:32956: Illegal state: replica 96f0c9a340d443c6a95e1d5e0c442d9c is not leader of this config: current role FOLLOWER
W20250625 20:05:13.196391 14121 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:52922: Illegal state: replica 20448e59f60a4266ab32666d1731cfcf is not leader of this config: current role FOLLOWER
W20250625 20:05:13.232270 14521 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:32956: Illegal state: replica 96f0c9a340d443c6a95e1d5e0c442d9c is not leader of this config: current role FOLLOWER
W20250625 20:05:13.311932 14121 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:52922: Illegal state: replica 20448e59f60a4266ab32666d1731cfcf is not leader of this config: current role FOLLOWER
W20250625 20:05:13.357497 14521 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:32956: Illegal state: replica 96f0c9a340d443c6a95e1d5e0c442d9c is not leader of this config: current role FOLLOWER
W20250625 20:05:13.440119 14121 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:52922: Illegal state: replica 20448e59f60a4266ab32666d1731cfcf is not leader of this config: current role FOLLOWER
W20250625 20:05:13.484640 14521 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:32956: Illegal state: replica 96f0c9a340d443c6a95e1d5e0c442d9c is not leader of this config: current role FOLLOWER
W20250625 20:05:13.575408 14121 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:52922: Illegal state: replica 20448e59f60a4266ab32666d1731cfcf is not leader of this config: current role FOLLOWER
W20250625 20:05:13.621856 14521 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:32956: Illegal state: replica 96f0c9a340d443c6a95e1d5e0c442d9c is not leader of this config: current role FOLLOWER
W20250625 20:05:13.671284 14641 meta_cache.cc:302] tablet 84bcadd235fc4e27a1d81811d11687f5: replica f6e300f0d75644c291f12e50a8228f55 (127.9.30.65:41333) has failed: Network error: Client connection negotiation failed: client connection to 127.9.30.65:41333: connect: Connection refused (error 111)
W20250625 20:05:13.718250 14121 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:52922: Illegal state: replica 20448e59f60a4266ab32666d1731cfcf is not leader of this config: current role FOLLOWER
W20250625 20:05:13.767534 14521 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:32956: Illegal state: replica 96f0c9a340d443c6a95e1d5e0c442d9c is not leader of this config: current role FOLLOWER
W20250625 20:05:13.871999 14121 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:52922: Illegal state: replica 20448e59f60a4266ab32666d1731cfcf is not leader of this config: current role FOLLOWER
W20250625 20:05:13.925038 14521 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:32956: Illegal state: replica 96f0c9a340d443c6a95e1d5e0c442d9c is not leader of this config: current role FOLLOWER
W20250625 20:05:14.149612 14121 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:52922: Illegal state: replica 20448e59f60a4266ab32666d1731cfcf is not leader of this config: current role FOLLOWER
W20250625 20:05:14.205335 14521 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:32956: Illegal state: replica 96f0c9a340d443c6a95e1d5e0c442d9c is not leader of this config: current role FOLLOWER
W20250625 20:05:14.321753 14121 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:52922: Illegal state: replica 20448e59f60a4266ab32666d1731cfcf is not leader of this config: current role FOLLOWER
W20250625 20:05:14.375756 14521 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:32956: Illegal state: replica 96f0c9a340d443c6a95e1d5e0c442d9c is not leader of this config: current role FOLLOWER
I20250625 20:05:14.426738 14660 raft_consensus.cc:491] T 84bcadd235fc4e27a1d81811d11687f5 P 20448e59f60a4266ab32666d1731cfcf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:05:14.427174 14660 raft_consensus.cc:513] T 84bcadd235fc4e27a1d81811d11687f5 P 20448e59f60a4266ab32666d1731cfcf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20448e59f60a4266ab32666d1731cfcf" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 38899 } } peers { permanent_uuid: "96f0c9a340d443c6a95e1d5e0c442d9c" member_type: VOTER last_known_addr { host: "127.9.30.69" port: 38029 } } peers { permanent_uuid: "f6e300f0d75644c291f12e50a8228f55" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 41333 } }
I20250625 20:05:14.429642 14660 leader_election.cc:290] T 84bcadd235fc4e27a1d81811d11687f5 P 20448e59f60a4266ab32666d1731cfcf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 96f0c9a340d443c6a95e1d5e0c442d9c (127.9.30.69:38029), f6e300f0d75644c291f12e50a8228f55 (127.9.30.65:41333)
W20250625 20:05:14.442121 14096 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.30.65:41333: connect: Connection refused (error 111)
W20250625 20:05:14.446586 14096 leader_election.cc:336] T 84bcadd235fc4e27a1d81811d11687f5 P 20448e59f60a4266ab32666d1731cfcf [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f6e300f0d75644c291f12e50a8228f55 (127.9.30.65:41333): Network error: Client connection negotiation failed: client connection to 127.9.30.65:41333: connect: Connection refused (error 111)
I20250625 20:05:14.449179 14561 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84bcadd235fc4e27a1d81811d11687f5" candidate_uuid: "20448e59f60a4266ab32666d1731cfcf" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96f0c9a340d443c6a95e1d5e0c442d9c" is_pre_election: true
I20250625 20:05:14.449708 14561 raft_consensus.cc:2466] T 84bcadd235fc4e27a1d81811d11687f5 P 96f0c9a340d443c6a95e1d5e0c442d9c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20448e59f60a4266ab32666d1731cfcf in term 1.
I20250625 20:05:14.450713 14096 leader_election.cc:304] T 84bcadd235fc4e27a1d81811d11687f5 P 20448e59f60a4266ab32666d1731cfcf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20448e59f60a4266ab32666d1731cfcf, 96f0c9a340d443c6a95e1d5e0c442d9c; no voters: f6e300f0d75644c291f12e50a8228f55
I20250625 20:05:14.451655 14660 raft_consensus.cc:2802] T 84bcadd235fc4e27a1d81811d11687f5 P 20448e59f60a4266ab32666d1731cfcf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250625 20:05:14.452041 14660 raft_consensus.cc:491] T 84bcadd235fc4e27a1d81811d11687f5 P 20448e59f60a4266ab32666d1731cfcf [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:05:14.452399 14660 raft_consensus.cc:3058] T 84bcadd235fc4e27a1d81811d11687f5 P 20448e59f60a4266ab32666d1731cfcf [term 1 FOLLOWER]: Advancing to term 2
I20250625 20:05:14.459251 14660 raft_consensus.cc:513] T 84bcadd235fc4e27a1d81811d11687f5 P 20448e59f60a4266ab32666d1731cfcf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20448e59f60a4266ab32666d1731cfcf" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 38899 } } peers { permanent_uuid: "96f0c9a340d443c6a95e1d5e0c442d9c" member_type: VOTER last_known_addr { host: "127.9.30.69" port: 38029 } } peers { permanent_uuid: "f6e300f0d75644c291f12e50a8228f55" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 41333 } }
I20250625 20:05:14.461004 14660 leader_election.cc:290] T 84bcadd235fc4e27a1d81811d11687f5 P 20448e59f60a4266ab32666d1731cfcf [CANDIDATE]: Term 2 election: Requested vote from peers 96f0c9a340d443c6a95e1d5e0c442d9c (127.9.30.69:38029), f6e300f0d75644c291f12e50a8228f55 (127.9.30.65:41333)
I20250625 20:05:14.462332 14561 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84bcadd235fc4e27a1d81811d11687f5" candidate_uuid: "20448e59f60a4266ab32666d1731cfcf" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96f0c9a340d443c6a95e1d5e0c442d9c"
I20250625 20:05:14.462898 14561 raft_consensus.cc:3058] T 84bcadd235fc4e27a1d81811d11687f5 P 96f0c9a340d443c6a95e1d5e0c442d9c [term 1 FOLLOWER]: Advancing to term 2
W20250625 20:05:14.464984 14096 leader_election.cc:336] T 84bcadd235fc4e27a1d81811d11687f5 P 20448e59f60a4266ab32666d1731cfcf [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f6e300f0d75644c291f12e50a8228f55 (127.9.30.65:41333): Network error: Client connection negotiation failed: client connection to 127.9.30.65:41333: connect: Connection refused (error 111)
I20250625 20:05:14.468868 14561 raft_consensus.cc:2466] T 84bcadd235fc4e27a1d81811d11687f5 P 96f0c9a340d443c6a95e1d5e0c442d9c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20448e59f60a4266ab32666d1731cfcf in term 2.
I20250625 20:05:14.469875 14096 leader_election.cc:304] T 84bcadd235fc4e27a1d81811d11687f5 P 20448e59f60a4266ab32666d1731cfcf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20448e59f60a4266ab32666d1731cfcf, 96f0c9a340d443c6a95e1d5e0c442d9c; no voters: f6e300f0d75644c291f12e50a8228f55
I20250625 20:05:14.470499 14660 raft_consensus.cc:2802] T 84bcadd235fc4e27a1d81811d11687f5 P 20448e59f60a4266ab32666d1731cfcf [term 2 FOLLOWER]: Leader election won for term 2
I20250625 20:05:14.472321 14660 raft_consensus.cc:695] T 84bcadd235fc4e27a1d81811d11687f5 P 20448e59f60a4266ab32666d1731cfcf [term 2 LEADER]: Becoming Leader. State: Replica: 20448e59f60a4266ab32666d1731cfcf, State: Running, Role: LEADER
I20250625 20:05:14.473183 14660 consensus_queue.cc:237] T 84bcadd235fc4e27a1d81811d11687f5 P 20448e59f60a4266ab32666d1731cfcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20448e59f60a4266ab32666d1731cfcf" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 38899 } } peers { permanent_uuid: "96f0c9a340d443c6a95e1d5e0c442d9c" member_type: VOTER last_known_addr { host: "127.9.30.69" port: 38029 } } peers { permanent_uuid: "f6e300f0d75644c291f12e50a8228f55" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 41333 } }
I20250625 20:05:14.484135 13881 catalog_manager.cc:5582] T 84bcadd235fc4e27a1d81811d11687f5 P 20448e59f60a4266ab32666d1731cfcf reported cstate change: term changed from 1 to 2, leader changed from f6e300f0d75644c291f12e50a8228f55 (127.9.30.65) to 20448e59f60a4266ab32666d1731cfcf (127.9.30.66). New cstate: current_term: 2 leader_uuid: "20448e59f60a4266ab32666d1731cfcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20448e59f60a4266ab32666d1731cfcf" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96f0c9a340d443c6a95e1d5e0c442d9c" member_type: VOTER last_known_addr { host: "127.9.30.69" port: 38029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6e300f0d75644c291f12e50a8228f55" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 41333 } health_report { overall_health: UNKNOWN } } }
I20250625 20:05:14.504738 14561 raft_consensus.cc:1273] T 84bcadd235fc4e27a1d81811d11687f5 P 96f0c9a340d443c6a95e1d5e0c442d9c [term 2 FOLLOWER]: Refusing update from remote peer 20448e59f60a4266ab32666d1731cfcf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250625 20:05:14.506922 14670 consensus_queue.cc:1035] T 84bcadd235fc4e27a1d81811d11687f5 P 20448e59f60a4266ab32666d1731cfcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "96f0c9a340d443c6a95e1d5e0c442d9c" member_type: VOTER last_known_addr { host: "127.9.30.69" port: 38029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250625 20:05:14.508742 14096 consensus_peers.cc:487] T 84bcadd235fc4e27a1d81811d11687f5 P 20448e59f60a4266ab32666d1731cfcf -> Peer f6e300f0d75644c291f12e50a8228f55 (127.9.30.65:41333): Couldn't send request to peer f6e300f0d75644c291f12e50a8228f55. Status: Network error: Client connection negotiation failed: client connection to 127.9.30.65:41333: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250625 20:05:14.524317 14681 mvcc.cc:204] Tried to move back new op lower bound from 7171612321791209472 to 7171612321689784320. Current Snapshot: MvccSnapshot[applied={T|T < 7171612321791209472}]
I20250625 20:05:14.607901 14141 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250625 20:05:14.624620 14274 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 20:05:14.643179 14541 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 20:05:14.661880 14407 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250625 20:05:15.145542 14202 debug-util.cc:398] Leaking SignalData structure 0x7b08000b2fe0 after lost signal to thread 14078
I20250625 20:05:16.619556 14141 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250625 20:05:16.655113 14541 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 20:05:16.661110 14274 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 20:05:16.680279 14407 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250625 20:05:17.147912 14096 consensus_peers.cc:487] T 84bcadd235fc4e27a1d81811d11687f5 P 20448e59f60a4266ab32666d1731cfcf -> Peer f6e300f0d75644c291f12e50a8228f55 (127.9.30.65:41333): Couldn't send request to peer f6e300f0d75644c291f12e50a8228f55. Status: Network error: Client connection negotiation failed: client connection to 127.9.30.65:41333: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250625 20:05:19.138236 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 14077
I20250625 20:05:19.170483 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 14210
I20250625 20:05:19.190182 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 14343
I20250625 20:05:19.208010 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 14476
I20250625 20:05:19.237331 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 13851
2025-06-25T20:05:19Z chronyd exiting
[ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4 (17849 ms)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest (17849 ms total)
[----------] 1 test from ListTableCliSimpleParamTest
[ RUN ] ListTableCliSimpleParamTest.TestListTables/2
I20250625 20:05:19.297062 9337 test_util.cc:276] Using random seed: -153623031
I20250625 20:05:19.301513 9337 ts_itest-base.cc:115] Starting cluster with:
I20250625 20:05:19.301693 9337 ts_itest-base.cc:116] --------------
I20250625 20:05:19.301812 9337 ts_itest-base.cc:117] 1 tablet servers
I20250625 20:05:19.301918 9337 ts_itest-base.cc:118] 1 replicas per TS
I20250625 20:05:19.302030 9337 ts_itest-base.cc:119] --------------
2025-06-25T20:05:19Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-25T20:05:19Z Disabled control of system clock
I20250625 20:05:19.345187 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.30.126:34081
--webserver_interface=127.9.30.126
--webserver_port=0
--builtin_ntp_servers=127.9.30.84:33725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.30.126:34081 with env {}
W20250625 20:05:19.640517 14750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:05:19.641081 14750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:05:19.641499 14750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:05:19.672976 14750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 20:05:19.673278 14750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:05:19.673483 14750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 20:05:19.673678 14750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 20:05:19.711517 14750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:33725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.30.126:34081
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.30.126:34081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.9.30.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:05:19.712885 14750 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:05:19.714536 14750 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:05:19.729879 14759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:05:19.730376 14757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:05:19.729867 14756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:05:19.731712 14750 server_base.cc:1048] running on GCE node
I20250625 20:05:20.828755 14750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:05:20.831331 14750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:05:20.832713 14750 hybrid_clock.cc:648] HybridClock initialized: now 1750881920832673 us; error 57 us; skew 500 ppm
I20250625 20:05:20.833539 14750 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:05:20.849803 14750 webserver.cc:469] Webserver started at http://127.9.30.126:42335/ using document root <none> and password file <none>
I20250625 20:05:20.850800 14750 fs_manager.cc:362] Metadata directory not provided
I20250625 20:05:20.851023 14750 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:05:20.851521 14750 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 20:05:20.857627 14750 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "1d38cc35784f477eb7e6149a5b8b3fcb"
format_stamp: "Formatted at 2025-06-25 20:05:20 on dist-test-slave-jhxp"
I20250625 20:05:20.858794 14750 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "1d38cc35784f477eb7e6149a5b8b3fcb"
format_stamp: "Formatted at 2025-06-25 20:05:20 on dist-test-slave-jhxp"
I20250625 20:05:20.866087 14750 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20250625 20:05:20.871888 14766 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:05:20.873008 14750 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250625 20:05:20.873364 14750 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
uuid: "1d38cc35784f477eb7e6149a5b8b3fcb"
format_stamp: "Formatted at 2025-06-25 20:05:20 on dist-test-slave-jhxp"
I20250625 20:05:20.873723 14750 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:05:20.934520 14750 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:05:20.936045 14750 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:05:20.936590 14750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:05:21.005115 14750 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.126:34081
I20250625 20:05:21.005211 14817 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.126:34081 every 8 connection(s)
I20250625 20:05:21.007961 14750 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250625 20:05:21.013070 14818 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:05:21.017716 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 14750
I20250625 20:05:21.018273 9337 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750881784340240-9337-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250625 20:05:21.034618 14818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d38cc35784f477eb7e6149a5b8b3fcb: Bootstrap starting.
I20250625 20:05:21.040271 14818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d38cc35784f477eb7e6149a5b8b3fcb: Neither blocks nor log segments found. Creating new log.
I20250625 20:05:21.042537 14818 log.cc:826] T 00000000000000000000000000000000 P 1d38cc35784f477eb7e6149a5b8b3fcb: Log is configured to *not* fsync() on all Append() calls
I20250625 20:05:21.047305 14818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d38cc35784f477eb7e6149a5b8b3fcb: No bootstrap required, opened a new log
I20250625 20:05:21.065119 14818 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1d38cc35784f477eb7e6149a5b8b3fcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d38cc35784f477eb7e6149a5b8b3fcb" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 34081 } }
I20250625 20:05:21.065822 14818 raft_consensus.cc:383] T 00000000000000000000000000000000 P 1d38cc35784f477eb7e6149a5b8b3fcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:05:21.066057 14818 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1d38cc35784f477eb7e6149a5b8b3fcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d38cc35784f477eb7e6149a5b8b3fcb, State: Initialized, Role: FOLLOWER
I20250625 20:05:21.066761 14818 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d38cc35784f477eb7e6149a5b8b3fcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d38cc35784f477eb7e6149a5b8b3fcb" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 34081 } }
I20250625 20:05:21.067299 14818 raft_consensus.cc:397] T 00000000000000000000000000000000 P 1d38cc35784f477eb7e6149a5b8b3fcb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 20:05:21.067565 14818 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1d38cc35784f477eb7e6149a5b8b3fcb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 20:05:21.067942 14818 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1d38cc35784f477eb7e6149a5b8b3fcb [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:05:21.072237 14818 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1d38cc35784f477eb7e6149a5b8b3fcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d38cc35784f477eb7e6149a5b8b3fcb" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 34081 } }
I20250625 20:05:21.072950 14818 leader_election.cc:304] T 00000000000000000000000000000000 P 1d38cc35784f477eb7e6149a5b8b3fcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d38cc35784f477eb7e6149a5b8b3fcb; no voters:
I20250625 20:05:21.074594 14818 leader_election.cc:290] T 00000000000000000000000000000000 P 1d38cc35784f477eb7e6149a5b8b3fcb [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 20:05:21.075311 14823 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1d38cc35784f477eb7e6149a5b8b3fcb [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:05:21.077538 14823 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1d38cc35784f477eb7e6149a5b8b3fcb [term 1 LEADER]: Becoming Leader. State: Replica: 1d38cc35784f477eb7e6149a5b8b3fcb, State: Running, Role: LEADER
I20250625 20:05:21.078403 14823 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d38cc35784f477eb7e6149a5b8b3fcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d38cc35784f477eb7e6149a5b8b3fcb" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 34081 } }
I20250625 20:05:21.078673 14818 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d38cc35784f477eb7e6149a5b8b3fcb [sys.catalog]: configured and running, proceeding with master startup.
I20250625 20:05:21.088973 14824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d38cc35784f477eb7e6149a5b8b3fcb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d38cc35784f477eb7e6149a5b8b3fcb. Latest consensus state: current_term: 1 leader_uuid: "1d38cc35784f477eb7e6149a5b8b3fcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d38cc35784f477eb7e6149a5b8b3fcb" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 34081 } } }
I20250625 20:05:21.089757 14824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d38cc35784f477eb7e6149a5b8b3fcb [sys.catalog]: This master's current role is: LEADER
I20250625 20:05:21.090600 14825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d38cc35784f477eb7e6149a5b8b3fcb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d38cc35784f477eb7e6149a5b8b3fcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d38cc35784f477eb7e6149a5b8b3fcb" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 34081 } } }
I20250625 20:05:21.091243 14825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d38cc35784f477eb7e6149a5b8b3fcb [sys.catalog]: This master's current role is: LEADER
I20250625 20:05:21.093576 14833 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 20:05:21.103727 14833 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 20:05:21.119824 14833 catalog_manager.cc:1349] Generated new cluster ID: bd336d7b9e484114b437271d863f4aca
I20250625 20:05:21.120199 14833 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 20:05:21.145730 14833 catalog_manager.cc:1372] Generated new certificate authority record
I20250625 20:05:21.147225 14833 catalog_manager.cc:1506] Loading token signing keys...
I20250625 20:05:21.166021 14833 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 1d38cc35784f477eb7e6149a5b8b3fcb: Generated new TSK 0
I20250625 20:05:21.167151 14833 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 20:05:21.188158 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.30.65:0
--local_ip_for_outbound_sockets=127.9.30.65
--webserver_interface=127.9.30.65
--webserver_port=0
--tserver_master_addrs=127.9.30.126:34081
--builtin_ntp_servers=127.9.30.84:33725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250625 20:05:21.497416 14842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:05:21.497933 14842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:05:21.498440 14842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:05:21.531179 14842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:05:21.532112 14842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.30.65
I20250625 20:05:21.568323 14842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:33725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.30.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.9.30.65
--webserver_port=0
--tserver_master_addrs=127.9.30.126:34081
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.30.65
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:05:21.569679 14842 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:05:21.571412 14842 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:05:21.589689 14851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:05:21.589733 14849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:05:21.591300 14848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:05:22.690478 14850 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1102 milliseconds
I20250625 20:05:22.690794 14842 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 20:05:22.694862 14842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:05:22.697489 14842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:05:22.698983 14842 hybrid_clock.cc:648] HybridClock initialized: now 1750881922698906 us; error 85 us; skew 500 ppm
I20250625 20:05:22.700163 14842 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:05:22.707172 14842 webserver.cc:469] Webserver started at http://127.9.30.65:39923/ using document root <none> and password file <none>
I20250625 20:05:22.708155 14842 fs_manager.cc:362] Metadata directory not provided
I20250625 20:05:22.708361 14842 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:05:22.708820 14842 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 20:05:22.713274 14842 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "fe0dfbc9a1cb462489af16cc4266eca5"
format_stamp: "Formatted at 2025-06-25 20:05:22 on dist-test-slave-jhxp"
I20250625 20:05:22.714395 14842 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "fe0dfbc9a1cb462489af16cc4266eca5"
format_stamp: "Formatted at 2025-06-25 20:05:22 on dist-test-slave-jhxp"
I20250625 20:05:22.721841 14842 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20250625 20:05:22.728065 14858 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:05:22.729198 14842 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250625 20:05:22.729512 14842 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "fe0dfbc9a1cb462489af16cc4266eca5"
format_stamp: "Formatted at 2025-06-25 20:05:22 on dist-test-slave-jhxp"
I20250625 20:05:22.729851 14842 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:05:22.790506 14842 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:05:22.792129 14842 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:05:22.792636 14842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:05:22.795660 14842 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 20:05:22.800022 14842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 20:05:22.800249 14842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 20:05:22.800490 14842 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 20:05:22.800647 14842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 20:05:22.935756 14842 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.65:37383
I20250625 20:05:22.935904 14970 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.65:37383 every 8 connection(s)
I20250625 20:05:22.938321 14842 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250625 20:05:22.946206 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 14842
I20250625 20:05:22.946615 9337 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750881784340240-9337-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250625 20:05:22.960155 14971 heartbeater.cc:344] Connected to a master server at 127.9.30.126:34081
I20250625 20:05:22.960637 14971 heartbeater.cc:461] Registering TS with master...
I20250625 20:05:22.961767 14971 heartbeater.cc:507] Master 127.9.30.126:34081 requested a full tablet report, sending...
I20250625 20:05:22.964632 14783 ts_manager.cc:194] Registered new tserver with Master: fe0dfbc9a1cb462489af16cc4266eca5 (127.9.30.65:37383)
I20250625 20:05:22.965828 9337 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250625 20:05:22.966912 14783 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.30.65:55623
I20250625 20:05:22.996372 14783 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39530:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250625 20:05:23.058257 14906 tablet_service.cc:1468] Processing CreateTablet for tablet ebe57e2d137649d4998dca0adf1c6b73 (DEFAULT_TABLE table=TestTable [id=18f5725ba2884b40aa3cd3db06bae4cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:05:23.059857 14906 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebe57e2d137649d4998dca0adf1c6b73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:05:23.079290 14986 tablet_bootstrap.cc:492] T ebe57e2d137649d4998dca0adf1c6b73 P fe0dfbc9a1cb462489af16cc4266eca5: Bootstrap starting.
I20250625 20:05:23.085533 14986 tablet_bootstrap.cc:654] T ebe57e2d137649d4998dca0adf1c6b73 P fe0dfbc9a1cb462489af16cc4266eca5: Neither blocks nor log segments found. Creating new log.
I20250625 20:05:23.087397 14986 log.cc:826] T ebe57e2d137649d4998dca0adf1c6b73 P fe0dfbc9a1cb462489af16cc4266eca5: Log is configured to *not* fsync() on all Append() calls
I20250625 20:05:23.092429 14986 tablet_bootstrap.cc:492] T ebe57e2d137649d4998dca0adf1c6b73 P fe0dfbc9a1cb462489af16cc4266eca5: No bootstrap required, opened a new log
I20250625 20:05:23.092898 14986 ts_tablet_manager.cc:1397] T ebe57e2d137649d4998dca0adf1c6b73 P fe0dfbc9a1cb462489af16cc4266eca5: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.006s
I20250625 20:05:23.111024 14986 raft_consensus.cc:357] T ebe57e2d137649d4998dca0adf1c6b73 P fe0dfbc9a1cb462489af16cc4266eca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe0dfbc9a1cb462489af16cc4266eca5" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 37383 } }
I20250625 20:05:23.111680 14986 raft_consensus.cc:383] T ebe57e2d137649d4998dca0adf1c6b73 P fe0dfbc9a1cb462489af16cc4266eca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:05:23.111902 14986 raft_consensus.cc:738] T ebe57e2d137649d4998dca0adf1c6b73 P fe0dfbc9a1cb462489af16cc4266eca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe0dfbc9a1cb462489af16cc4266eca5, State: Initialized, Role: FOLLOWER
I20250625 20:05:23.112571 14986 consensus_queue.cc:260] T ebe57e2d137649d4998dca0adf1c6b73 P fe0dfbc9a1cb462489af16cc4266eca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe0dfbc9a1cb462489af16cc4266eca5" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 37383 } }
I20250625 20:05:23.113102 14986 raft_consensus.cc:397] T ebe57e2d137649d4998dca0adf1c6b73 P fe0dfbc9a1cb462489af16cc4266eca5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 20:05:23.113343 14986 raft_consensus.cc:491] T ebe57e2d137649d4998dca0adf1c6b73 P fe0dfbc9a1cb462489af16cc4266eca5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 20:05:23.113618 14986 raft_consensus.cc:3058] T ebe57e2d137649d4998dca0adf1c6b73 P fe0dfbc9a1cb462489af16cc4266eca5 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:05:23.118199 14986 raft_consensus.cc:513] T ebe57e2d137649d4998dca0adf1c6b73 P fe0dfbc9a1cb462489af16cc4266eca5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe0dfbc9a1cb462489af16cc4266eca5" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 37383 } }
I20250625 20:05:23.118965 14986 leader_election.cc:304] T ebe57e2d137649d4998dca0adf1c6b73 P fe0dfbc9a1cb462489af16cc4266eca5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe0dfbc9a1cb462489af16cc4266eca5; no voters:
I20250625 20:05:23.120712 14986 leader_election.cc:290] T ebe57e2d137649d4998dca0adf1c6b73 P fe0dfbc9a1cb462489af16cc4266eca5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 20:05:23.121076 14988 raft_consensus.cc:2802] T ebe57e2d137649d4998dca0adf1c6b73 P fe0dfbc9a1cb462489af16cc4266eca5 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:05:23.123193 14988 raft_consensus.cc:695] T ebe57e2d137649d4998dca0adf1c6b73 P fe0dfbc9a1cb462489af16cc4266eca5 [term 1 LEADER]: Becoming Leader. State: Replica: fe0dfbc9a1cb462489af16cc4266eca5, State: Running, Role: LEADER
I20250625 20:05:23.124154 14971 heartbeater.cc:499] Master 127.9.30.126:34081 was elected leader, sending a full tablet report...
I20250625 20:05:23.124284 14988 consensus_queue.cc:237] T ebe57e2d137649d4998dca0adf1c6b73 P fe0dfbc9a1cb462489af16cc4266eca5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe0dfbc9a1cb462489af16cc4266eca5" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 37383 } }
I20250625 20:05:23.125052 14986 ts_tablet_manager.cc:1428] T ebe57e2d137649d4998dca0adf1c6b73 P fe0dfbc9a1cb462489af16cc4266eca5: Time spent starting tablet: real 0.032s user 0.017s sys 0.014s
I20250625 20:05:23.137285 14782 catalog_manager.cc:5582] T ebe57e2d137649d4998dca0adf1c6b73 P fe0dfbc9a1cb462489af16cc4266eca5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe0dfbc9a1cb462489af16cc4266eca5 (127.9.30.65). New cstate: current_term: 1 leader_uuid: "fe0dfbc9a1cb462489af16cc4266eca5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe0dfbc9a1cb462489af16cc4266eca5" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 37383 } health_report { overall_health: HEALTHY } } }
I20250625 20:05:23.165058 9337 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250625 20:05:23.168322 9337 ts_itest-base.cc:246] Waiting for 1 tablets on tserver fe0dfbc9a1cb462489af16cc4266eca5 to finish bootstrapping
I20250625 20:05:25.804713 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 14842
I20250625 20:05:25.826256 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 14750
2025-06-25T20:05:25Z chronyd exiting
[ OK ] ListTableCliSimpleParamTest.TestListTables/2 (6582 ms)
[----------] 1 test from ListTableCliSimpleParamTest (6583 ms total)
[----------] 1 test from ListTableCliParamTest
[ RUN ] ListTableCliParamTest.ListTabletWithPartitionInfo/4
I20250625 20:05:25.880532 9337 test_util.cc:276] Using random seed: -147039560
[ OK ] ListTableCliParamTest.ListTabletWithPartitionInfo/4 (12 ms)
[----------] 1 test from ListTableCliParamTest (12 ms total)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest
[ RUN ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0
2025-06-25T20:05:25Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-25T20:05:25Z Disabled control of system clock
I20250625 20:05:25.931627 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.30.126:33949
--webserver_interface=127.9.30.126
--webserver_port=0
--builtin_ntp_servers=127.9.30.84:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.30.126:33949 with env {}
W20250625 20:05:26.228767 15015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:05:26.229322 15015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:05:26.229753 15015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:05:26.261020 15015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 20:05:26.261348 15015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:05:26.261564 15015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 20:05:26.261767 15015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 20:05:26.297394 15015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.30.126:33949
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.30.126:33949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.30.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:05:26.298702 15015 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:05:26.300367 15015 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:05:26.314388 15021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:05:26.315057 15022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:05:26.316810 15024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:05:27.404865 15023 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250625 20:05:27.404937 15015 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 20:05:27.408583 15015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:05:27.411114 15015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:05:27.412544 15015 hybrid_clock.cc:648] HybridClock initialized: now 1750881927412502 us; error 58 us; skew 500 ppm
I20250625 20:05:27.413383 15015 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:05:27.424793 15015 webserver.cc:469] Webserver started at http://127.9.30.126:38361/ using document root <none> and password file <none>
I20250625 20:05:27.425801 15015 fs_manager.cc:362] Metadata directory not provided
I20250625 20:05:27.426033 15015 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:05:27.426497 15015 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 20:05:27.431026 15015 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/master-0/data/instance:
uuid: "a931edeedec34ef6b8846b7a84d71d7e"
format_stamp: "Formatted at 2025-06-25 20:05:27 on dist-test-slave-jhxp"
I20250625 20:05:27.432194 15015 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/master-0/wal/instance:
uuid: "a931edeedec34ef6b8846b7a84d71d7e"
format_stamp: "Formatted at 2025-06-25 20:05:27 on dist-test-slave-jhxp"
I20250625 20:05:27.439522 15015 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20250625 20:05:27.445154 15031 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:05:27.446250 15015 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250625 20:05:27.446594 15015 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/master-0/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/master-0/wal
uuid: "a931edeedec34ef6b8846b7a84d71d7e"
format_stamp: "Formatted at 2025-06-25 20:05:27 on dist-test-slave-jhxp"
I20250625 20:05:27.446920 15015 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:05:27.508793 15015 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:05:27.510241 15015 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:05:27.510727 15015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:05:27.580019 15015 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.126:33949
I20250625 20:05:27.580093 15082 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.126:33949 every 8 connection(s)
I20250625 20:05:27.582751 15015 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/master-0/data/info.pb
I20250625 20:05:27.587795 15083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:05:27.592953 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 15015
I20250625 20:05:27.593436 9337 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/master-0/wal/instance
I20250625 20:05:27.611979 15083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a931edeedec34ef6b8846b7a84d71d7e: Bootstrap starting.
I20250625 20:05:27.618119 15083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a931edeedec34ef6b8846b7a84d71d7e: Neither blocks nor log segments found. Creating new log.
I20250625 20:05:27.619990 15083 log.cc:826] T 00000000000000000000000000000000 P a931edeedec34ef6b8846b7a84d71d7e: Log is configured to *not* fsync() on all Append() calls
I20250625 20:05:27.625103 15083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a931edeedec34ef6b8846b7a84d71d7e: No bootstrap required, opened a new log
I20250625 20:05:27.642694 15083 raft_consensus.cc:357] T 00000000000000000000000000000000 P a931edeedec34ef6b8846b7a84d71d7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a931edeedec34ef6b8846b7a84d71d7e" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 33949 } }
I20250625 20:05:27.643400 15083 raft_consensus.cc:383] T 00000000000000000000000000000000 P a931edeedec34ef6b8846b7a84d71d7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:05:27.643662 15083 raft_consensus.cc:738] T 00000000000000000000000000000000 P a931edeedec34ef6b8846b7a84d71d7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a931edeedec34ef6b8846b7a84d71d7e, State: Initialized, Role: FOLLOWER
I20250625 20:05:27.644338 15083 consensus_queue.cc:260] T 00000000000000000000000000000000 P a931edeedec34ef6b8846b7a84d71d7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a931edeedec34ef6b8846b7a84d71d7e" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 33949 } }
I20250625 20:05:27.644855 15083 raft_consensus.cc:397] T 00000000000000000000000000000000 P a931edeedec34ef6b8846b7a84d71d7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 20:05:27.645113 15083 raft_consensus.cc:491] T 00000000000000000000000000000000 P a931edeedec34ef6b8846b7a84d71d7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 20:05:27.645394 15083 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a931edeedec34ef6b8846b7a84d71d7e [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:05:27.649646 15083 raft_consensus.cc:513] T 00000000000000000000000000000000 P a931edeedec34ef6b8846b7a84d71d7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a931edeedec34ef6b8846b7a84d71d7e" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 33949 } }
I20250625 20:05:27.650380 15083 leader_election.cc:304] T 00000000000000000000000000000000 P a931edeedec34ef6b8846b7a84d71d7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a931edeedec34ef6b8846b7a84d71d7e; no voters:
I20250625 20:05:27.652024 15083 leader_election.cc:290] T 00000000000000000000000000000000 P a931edeedec34ef6b8846b7a84d71d7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 20:05:27.652667 15088 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a931edeedec34ef6b8846b7a84d71d7e [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:05:27.654649 15088 raft_consensus.cc:695] T 00000000000000000000000000000000 P a931edeedec34ef6b8846b7a84d71d7e [term 1 LEADER]: Becoming Leader. State: Replica: a931edeedec34ef6b8846b7a84d71d7e, State: Running, Role: LEADER
I20250625 20:05:27.655462 15088 consensus_queue.cc:237] T 00000000000000000000000000000000 P a931edeedec34ef6b8846b7a84d71d7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a931edeedec34ef6b8846b7a84d71d7e" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 33949 } }
I20250625 20:05:27.656425 15083 sys_catalog.cc:564] T 00000000000000000000000000000000 P a931edeedec34ef6b8846b7a84d71d7e [sys.catalog]: configured and running, proceeding with master startup.
I20250625 20:05:27.665946 15090 sys_catalog.cc:455] T 00000000000000000000000000000000 P a931edeedec34ef6b8846b7a84d71d7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a931edeedec34ef6b8846b7a84d71d7e. Latest consensus state: current_term: 1 leader_uuid: "a931edeedec34ef6b8846b7a84d71d7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a931edeedec34ef6b8846b7a84d71d7e" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 33949 } } }
I20250625 20:05:27.666949 15090 sys_catalog.cc:458] T 00000000000000000000000000000000 P a931edeedec34ef6b8846b7a84d71d7e [sys.catalog]: This master's current role is: LEADER
I20250625 20:05:27.667182 15089 sys_catalog.cc:455] T 00000000000000000000000000000000 P a931edeedec34ef6b8846b7a84d71d7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a931edeedec34ef6b8846b7a84d71d7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a931edeedec34ef6b8846b7a84d71d7e" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 33949 } } }
I20250625 20:05:27.667876 15089 sys_catalog.cc:458] T 00000000000000000000000000000000 P a931edeedec34ef6b8846b7a84d71d7e [sys.catalog]: This master's current role is: LEADER
I20250625 20:05:27.669643 15097 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 20:05:27.680075 15097 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 20:05:27.696905 15097 catalog_manager.cc:1349] Generated new cluster ID: 628fdd07becb44ba8d6405befb9fdb3a
I20250625 20:05:27.697163 15097 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 20:05:27.724542 15097 catalog_manager.cc:1372] Generated new certificate authority record
I20250625 20:05:27.726006 15097 catalog_manager.cc:1506] Loading token signing keys...
I20250625 20:05:27.737957 15097 catalog_manager.cc:5955] T 00000000000000000000000000000000 P a931edeedec34ef6b8846b7a84d71d7e: Generated new TSK 0
I20250625 20:05:27.738833 15097 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 20:05:27.758227 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.30.65:0
--local_ip_for_outbound_sockets=127.9.30.65
--webserver_interface=127.9.30.65
--webserver_port=0
--tserver_master_addrs=127.9.30.126:33949
--builtin_ntp_servers=127.9.30.84:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20250625 20:05:28.061255 15107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:05:28.061775 15107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:05:28.062265 15107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:05:28.093858 15107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:05:28.094722 15107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.30.65
I20250625 20:05:28.130602 15107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.30.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.30.65
--webserver_port=0
--tserver_master_addrs=127.9.30.126:33949
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.30.65
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:05:28.131965 15107 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:05:28.133618 15107 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:05:28.150847 15114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:05:28.154111 15116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:05:28.151160 15113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:05:29.240304 15115 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250625 20:05:29.240525 15107 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 20:05:29.244323 15107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:05:29.247043 15107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:05:29.248508 15107 hybrid_clock.cc:648] HybridClock initialized: now 1750881929248444 us; error 78 us; skew 500 ppm
I20250625 20:05:29.249356 15107 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:05:29.256039 15107 webserver.cc:469] Webserver started at http://127.9.30.65:39933/ using document root <none> and password file <none>
I20250625 20:05:29.257026 15107 fs_manager.cc:362] Metadata directory not provided
I20250625 20:05:29.257231 15107 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:05:29.257689 15107 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 20:05:29.262321 15107 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-0/data/instance:
uuid: "3206ae854c274fc79f8827d53c8cac18"
format_stamp: "Formatted at 2025-06-25 20:05:29 on dist-test-slave-jhxp"
I20250625 20:05:29.263517 15107 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-0/wal/instance:
uuid: "3206ae854c274fc79f8827d53c8cac18"
format_stamp: "Formatted at 2025-06-25 20:05:29 on dist-test-slave-jhxp"
I20250625 20:05:29.271294 15107 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s
I20250625 20:05:29.277467 15123 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:05:29.278740 15107 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250625 20:05:29.279111 15107 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-0/wal
uuid: "3206ae854c274fc79f8827d53c8cac18"
format_stamp: "Formatted at 2025-06-25 20:05:29 on dist-test-slave-jhxp"
I20250625 20:05:29.279461 15107 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:05:29.337095 15107 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:05:29.338588 15107 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:05:29.339025 15107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:05:29.341677 15107 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 20:05:29.345989 15107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 20:05:29.346225 15107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 20:05:29.346484 15107 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 20:05:29.346644 15107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 20:05:29.500775 15107 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.65:38305
I20250625 20:05:29.500887 15235 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.65:38305 every 8 connection(s)
I20250625 20:05:29.503383 15107 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-0/data/info.pb
I20250625 20:05:29.506748 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 15107
I20250625 20:05:29.507244 9337 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-0/wal/instance
I20250625 20:05:29.513696 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.30.66:0
--local_ip_for_outbound_sockets=127.9.30.66
--webserver_interface=127.9.30.66
--webserver_port=0
--tserver_master_addrs=127.9.30.126:33949
--builtin_ntp_servers=127.9.30.84:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250625 20:05:29.524897 15236 heartbeater.cc:344] Connected to a master server at 127.9.30.126:33949
I20250625 20:05:29.525339 15236 heartbeater.cc:461] Registering TS with master...
I20250625 20:05:29.526324 15236 heartbeater.cc:507] Master 127.9.30.126:33949 requested a full tablet report, sending...
I20250625 20:05:29.528805 15048 ts_manager.cc:194] Registered new tserver with Master: 3206ae854c274fc79f8827d53c8cac18 (127.9.30.65:38305)
I20250625 20:05:29.530807 15048 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.30.65:55979
W20250625 20:05:29.811337 15240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:05:29.811919 15240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:05:29.812433 15240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:05:29.843483 15240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:05:29.844383 15240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.30.66
I20250625 20:05:29.879024 15240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.30.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.30.66
--webserver_port=0
--tserver_master_addrs=127.9.30.126:33949
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.30.66
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:05:29.880415 15240 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:05:29.882102 15240 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:05:29.897632 15246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:05:30.534330 15236 heartbeater.cc:499] Master 127.9.30.126:33949 was elected leader, sending a full tablet report...
W20250625 20:05:29.897702 15247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:05:29.898303 15249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:05:30.970482 15248 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250625 20:05:30.970579 15240 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 20:05:30.974524 15240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:05:30.977185 15240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:05:30.978617 15240 hybrid_clock.cc:648] HybridClock initialized: now 1750881930978578 us; error 56 us; skew 500 ppm
I20250625 20:05:30.979544 15240 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:05:30.986052 15240 webserver.cc:469] Webserver started at http://127.9.30.66:45603/ using document root <none> and password file <none>
I20250625 20:05:30.987001 15240 fs_manager.cc:362] Metadata directory not provided
I20250625 20:05:30.987226 15240 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:05:30.987722 15240 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 20:05:30.993542 15240 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-1/data/instance:
uuid: "cd293a4f251941f5b4b07118f2a30bb5"
format_stamp: "Formatted at 2025-06-25 20:05:30 on dist-test-slave-jhxp"
I20250625 20:05:30.994706 15240 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-1/wal/instance:
uuid: "cd293a4f251941f5b4b07118f2a30bb5"
format_stamp: "Formatted at 2025-06-25 20:05:30 on dist-test-slave-jhxp"
I20250625 20:05:31.002244 15240 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250625 20:05:31.008059 15256 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:05:31.009202 15240 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250625 20:05:31.009593 15240 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-1/wal
uuid: "cd293a4f251941f5b4b07118f2a30bb5"
format_stamp: "Formatted at 2025-06-25 20:05:30 on dist-test-slave-jhxp"
I20250625 20:05:31.009943 15240 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:05:31.056517 15240 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:05:31.057957 15240 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:05:31.058463 15240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:05:31.060931 15240 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 20:05:31.064990 15240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 20:05:31.065203 15240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 20:05:31.065410 15240 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 20:05:31.065543 15240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 20:05:31.197204 15240 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.66:45129
I20250625 20:05:31.197314 15368 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.66:45129 every 8 connection(s)
I20250625 20:05:31.199796 15240 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-1/data/info.pb
I20250625 20:05:31.205386 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 15240
I20250625 20:05:31.205919 9337 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-1/wal/instance
I20250625 20:05:31.213289 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.30.67:0
--local_ip_for_outbound_sockets=127.9.30.67
--webserver_interface=127.9.30.67
--webserver_port=0
--tserver_master_addrs=127.9.30.126:33949
--builtin_ntp_servers=127.9.30.84:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250625 20:05:31.222035 15369 heartbeater.cc:344] Connected to a master server at 127.9.30.126:33949
I20250625 20:05:31.222506 15369 heartbeater.cc:461] Registering TS with master...
I20250625 20:05:31.223587 15369 heartbeater.cc:507] Master 127.9.30.126:33949 requested a full tablet report, sending...
I20250625 20:05:31.225880 15048 ts_manager.cc:194] Registered new tserver with Master: cd293a4f251941f5b4b07118f2a30bb5 (127.9.30.66:45129)
I20250625 20:05:31.227141 15048 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.30.66:51851
W20250625 20:05:31.513458 15373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:05:31.513952 15373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:05:31.514451 15373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:05:31.546157 15373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:05:31.547020 15373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.30.67
I20250625 20:05:31.586270 15373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.30.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.30.67
--webserver_port=0
--tserver_master_addrs=127.9.30.126:33949
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.30.67
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:05:31.587956 15373 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:05:31.590176 15373 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:05:31.608613 15380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:05:32.230420 15369 heartbeater.cc:499] Master 127.9.30.126:33949 was elected leader, sending a full tablet report...
W20250625 20:05:31.610410 15379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:05:31.610617 15382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:05:31.612583 15373 server_base.cc:1048] running on GCE node
I20250625 20:05:32.702626 15373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:05:32.704906 15373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:05:32.706292 15373 hybrid_clock.cc:648] HybridClock initialized: now 1750881932706246 us; error 57 us; skew 500 ppm
I20250625 20:05:32.707145 15373 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:05:32.716998 15373 webserver.cc:469] Webserver started at http://127.9.30.67:38965/ using document root <none> and password file <none>
I20250625 20:05:32.717996 15373 fs_manager.cc:362] Metadata directory not provided
I20250625 20:05:32.718227 15373 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:05:32.718688 15373 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 20:05:32.723357 15373 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-2/data/instance:
uuid: "fb865ee6342943c09c3b0d631257eb89"
format_stamp: "Formatted at 2025-06-25 20:05:32 on dist-test-slave-jhxp"
I20250625 20:05:32.724578 15373 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-2/wal/instance:
uuid: "fb865ee6342943c09c3b0d631257eb89"
format_stamp: "Formatted at 2025-06-25 20:05:32 on dist-test-slave-jhxp"
I20250625 20:05:32.731851 15373 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250625 20:05:32.738710 15389 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:05:32.739858 15373 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250625 20:05:32.740195 15373 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-2/wal
uuid: "fb865ee6342943c09c3b0d631257eb89"
format_stamp: "Formatted at 2025-06-25 20:05:32 on dist-test-slave-jhxp"
I20250625 20:05:32.740548 15373 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:05:32.793581 15373 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:05:32.795073 15373 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:05:32.795534 15373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:05:32.798031 15373 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 20:05:32.802150 15373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 20:05:32.802364 15373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 20:05:32.802634 15373 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 20:05:32.802794 15373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 20:05:32.932808 15373 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.67:33829
I20250625 20:05:32.932929 15501 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.67:33829 every 8 connection(s)
I20250625 20:05:32.935375 15373 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-2/data/info.pb
I20250625 20:05:32.937896 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 15373
I20250625 20:05:32.938439 9337 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-2/wal/instance
I20250625 20:05:32.957124 15502 heartbeater.cc:344] Connected to a master server at 127.9.30.126:33949
I20250625 20:05:32.957649 15502 heartbeater.cc:461] Registering TS with master...
I20250625 20:05:32.958959 15502 heartbeater.cc:507] Master 127.9.30.126:33949 requested a full tablet report, sending...
I20250625 20:05:32.961306 15048 ts_manager.cc:194] Registered new tserver with Master: fb865ee6342943c09c3b0d631257eb89 (127.9.30.67:33829)
I20250625 20:05:32.962548 15048 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.30.67:53673
I20250625 20:05:32.974054 9337 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 20:05:32.999500 9337 test_util.cc:276] Using random seed: -139920589
I20250625 20:05:33.041244 15048 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56686:
name: "pre_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 20:05:33.043766 15048 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table pre_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 20:05:33.097273 15437 tablet_service.cc:1468] Processing CreateTablet for tablet 56087ecb54f9431f9e13ceb2fb3ba969 (DEFAULT_TABLE table=pre_rebuild [id=d94171de772347649e1deab253d8d705]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:05:33.098076 15304 tablet_service.cc:1468] Processing CreateTablet for tablet 56087ecb54f9431f9e13ceb2fb3ba969 (DEFAULT_TABLE table=pre_rebuild [id=d94171de772347649e1deab253d8d705]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:05:33.099104 15171 tablet_service.cc:1468] Processing CreateTablet for tablet 56087ecb54f9431f9e13ceb2fb3ba969 (DEFAULT_TABLE table=pre_rebuild [id=d94171de772347649e1deab253d8d705]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:05:33.099897 15304 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56087ecb54f9431f9e13ceb2fb3ba969. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:05:33.100124 15437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56087ecb54f9431f9e13ceb2fb3ba969. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:05:33.100888 15171 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56087ecb54f9431f9e13ceb2fb3ba969. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:05:33.121896 15526 tablet_bootstrap.cc:492] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5: Bootstrap starting.
I20250625 20:05:33.129345 15527 tablet_bootstrap.cc:492] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89: Bootstrap starting.
I20250625 20:05:33.129841 15526 tablet_bootstrap.cc:654] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5: Neither blocks nor log segments found. Creating new log.
I20250625 20:05:33.131798 15528 tablet_bootstrap.cc:492] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18: Bootstrap starting.
I20250625 20:05:33.132383 15526 log.cc:826] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5: Log is configured to *not* fsync() on all Append() calls
I20250625 20:05:33.136816 15527 tablet_bootstrap.cc:654] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89: Neither blocks nor log segments found. Creating new log.
I20250625 20:05:33.138463 15528 tablet_bootstrap.cc:654] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18: Neither blocks nor log segments found. Creating new log.
I20250625 20:05:33.138995 15526 tablet_bootstrap.cc:492] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5: No bootstrap required, opened a new log
I20250625 20:05:33.139170 15527 log.cc:826] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89: Log is configured to *not* fsync() on all Append() calls
I20250625 20:05:33.140743 15528 log.cc:826] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18: Log is configured to *not* fsync() on all Append() calls
I20250625 20:05:33.141327 15526 ts_tablet_manager.cc:1397] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5: Time spent bootstrapping tablet: real 0.020s user 0.015s sys 0.000s
I20250625 20:05:33.144834 15527 tablet_bootstrap.cc:492] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89: No bootstrap required, opened a new log
I20250625 20:05:33.145278 15527 ts_tablet_manager.cc:1397] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.002s
I20250625 20:05:33.146718 15528 tablet_bootstrap.cc:492] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18: No bootstrap required, opened a new log
I20250625 20:05:33.147274 15528 ts_tablet_manager.cc:1397] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.009s
I20250625 20:05:33.164175 15527 raft_consensus.cc:357] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd293a4f251941f5b4b07118f2a30bb5" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45129 } } peers { permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 } } peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } }
I20250625 20:05:33.164911 15527 raft_consensus.cc:383] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:05:33.165174 15527 raft_consensus.cc:738] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb865ee6342943c09c3b0d631257eb89, State: Initialized, Role: FOLLOWER
I20250625 20:05:33.165921 15527 consensus_queue.cc:260] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd293a4f251941f5b4b07118f2a30bb5" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45129 } } peers { permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 } } peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } }
I20250625 20:05:33.168874 15502 heartbeater.cc:499] Master 127.9.30.126:33949 was elected leader, sending a full tablet report...
I20250625 20:05:33.170281 15527 ts_tablet_manager.cc:1428] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89: Time spent starting tablet: real 0.025s user 0.013s sys 0.011s
I20250625 20:05:33.169824 15526 raft_consensus.cc:357] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd293a4f251941f5b4b07118f2a30bb5" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45129 } } peers { permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 } } peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } }
I20250625 20:05:33.170779 15526 raft_consensus.cc:383] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:05:33.171177 15526 raft_consensus.cc:738] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd293a4f251941f5b4b07118f2a30bb5, State: Initialized, Role: FOLLOWER
I20250625 20:05:33.172268 15526 consensus_queue.cc:260] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd293a4f251941f5b4b07118f2a30bb5" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45129 } } peers { permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 } } peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } }
I20250625 20:05:33.176784 15526 ts_tablet_manager.cc:1428] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5: Time spent starting tablet: real 0.035s user 0.023s sys 0.011s
I20250625 20:05:33.178366 15528 raft_consensus.cc:357] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd293a4f251941f5b4b07118f2a30bb5" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45129 } } peers { permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 } } peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } }
I20250625 20:05:33.179083 15528 raft_consensus.cc:383] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:05:33.179394 15528 raft_consensus.cc:738] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3206ae854c274fc79f8827d53c8cac18, State: Initialized, Role: FOLLOWER
I20250625 20:05:33.180464 15528 consensus_queue.cc:260] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd293a4f251941f5b4b07118f2a30bb5" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45129 } } peers { permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 } } peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } }
I20250625 20:05:33.184643 15528 ts_tablet_manager.cc:1428] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18: Time spent starting tablet: real 0.037s user 0.036s sys 0.001s
W20250625 20:05:33.191908 15503 tablet.cc:2378] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250625 20:05:33.206344 15370 tablet.cc:2378] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250625 20:05:33.262262 15237 tablet.cc:2378] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 20:05:33.388556 15533 raft_consensus.cc:491] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:05:33.389087 15533 raft_consensus.cc:513] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd293a4f251941f5b4b07118f2a30bb5" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45129 } } peers { permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 } } peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } }
I20250625 20:05:33.389603 15532 raft_consensus.cc:491] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:05:33.390120 15532 raft_consensus.cc:513] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd293a4f251941f5b4b07118f2a30bb5" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45129 } } peers { permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 } } peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } }
I20250625 20:05:33.392279 15533 leader_election.cc:290] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb865ee6342943c09c3b0d631257eb89 (127.9.30.67:33829), 3206ae854c274fc79f8827d53c8cac18 (127.9.30.65:38305)
I20250625 20:05:33.393010 15532 leader_election.cc:290] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd293a4f251941f5b4b07118f2a30bb5 (127.9.30.66:45129), 3206ae854c274fc79f8827d53c8cac18 (127.9.30.65:38305)
I20250625 20:05:33.411208 15324 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56087ecb54f9431f9e13ceb2fb3ba969" candidate_uuid: "fb865ee6342943c09c3b0d631257eb89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd293a4f251941f5b4b07118f2a30bb5" is_pre_election: true
I20250625 20:05:33.412132 15324 raft_consensus.cc:2466] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb865ee6342943c09c3b0d631257eb89 in term 0.
I20250625 20:05:33.412297 15191 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56087ecb54f9431f9e13ceb2fb3ba969" candidate_uuid: "fb865ee6342943c09c3b0d631257eb89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3206ae854c274fc79f8827d53c8cac18" is_pre_election: true
I20250625 20:05:33.412915 15457 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56087ecb54f9431f9e13ceb2fb3ba969" candidate_uuid: "cd293a4f251941f5b4b07118f2a30bb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb865ee6342943c09c3b0d631257eb89" is_pre_election: true
I20250625 20:05:33.413232 15191 raft_consensus.cc:2466] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb865ee6342943c09c3b0d631257eb89 in term 0.
I20250625 20:05:33.413654 15457 raft_consensus.cc:2466] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd293a4f251941f5b4b07118f2a30bb5 in term 0.
I20250625 20:05:33.415005 15391 leader_election.cc:304] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd293a4f251941f5b4b07118f2a30bb5, fb865ee6342943c09c3b0d631257eb89; no voters:
I20250625 20:05:33.415006 15259 leader_election.cc:304] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd293a4f251941f5b4b07118f2a30bb5, fb865ee6342943c09c3b0d631257eb89; no voters:
I20250625 20:05:33.415931 15532 raft_consensus.cc:2802] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:05:33.415899 15191 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56087ecb54f9431f9e13ceb2fb3ba969" candidate_uuid: "cd293a4f251941f5b4b07118f2a30bb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3206ae854c274fc79f8827d53c8cac18" is_pre_election: true
I20250625 20:05:33.416024 15533 raft_consensus.cc:2802] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:05:33.416290 15532 raft_consensus.cc:491] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:05:33.416383 15533 raft_consensus.cc:491] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:05:33.416400 15191 raft_consensus.cc:2466] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd293a4f251941f5b4b07118f2a30bb5 in term 0.
I20250625 20:05:33.416613 15532 raft_consensus.cc:3058] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:05:33.416707 15533 raft_consensus.cc:3058] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:05:33.423271 15533 raft_consensus.cc:513] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd293a4f251941f5b4b07118f2a30bb5" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45129 } } peers { permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 } } peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } }
I20250625 20:05:33.423271 15532 raft_consensus.cc:513] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd293a4f251941f5b4b07118f2a30bb5" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45129 } } peers { permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 } } peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } }
I20250625 20:05:33.425168 15532 leader_election.cc:290] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89 [CANDIDATE]: Term 1 election: Requested vote from peers cd293a4f251941f5b4b07118f2a30bb5 (127.9.30.66:45129), 3206ae854c274fc79f8827d53c8cac18 (127.9.30.65:38305)
I20250625 20:05:33.425168 15533 leader_election.cc:290] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5 [CANDIDATE]: Term 1 election: Requested vote from peers fb865ee6342943c09c3b0d631257eb89 (127.9.30.67:33829), 3206ae854c274fc79f8827d53c8cac18 (127.9.30.65:38305)
I20250625 20:05:33.426363 15190 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56087ecb54f9431f9e13ceb2fb3ba969" candidate_uuid: "cd293a4f251941f5b4b07118f2a30bb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3206ae854c274fc79f8827d53c8cac18"
I20250625 20:05:33.426725 15457 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56087ecb54f9431f9e13ceb2fb3ba969" candidate_uuid: "cd293a4f251941f5b4b07118f2a30bb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb865ee6342943c09c3b0d631257eb89"
I20250625 20:05:33.426771 15191 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56087ecb54f9431f9e13ceb2fb3ba969" candidate_uuid: "fb865ee6342943c09c3b0d631257eb89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3206ae854c274fc79f8827d53c8cac18"
I20250625 20:05:33.426756 15324 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56087ecb54f9431f9e13ceb2fb3ba969" candidate_uuid: "fb865ee6342943c09c3b0d631257eb89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd293a4f251941f5b4b07118f2a30bb5"
I20250625 20:05:33.427048 15190 raft_consensus.cc:3058] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:05:33.427454 15457 raft_consensus.cc:2391] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cd293a4f251941f5b4b07118f2a30bb5 in current term 1: Already voted for candidate fb865ee6342943c09c3b0d631257eb89 in this term.
I20250625 20:05:33.427524 15324 raft_consensus.cc:2391] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fb865ee6342943c09c3b0d631257eb89 in current term 1: Already voted for candidate cd293a4f251941f5b4b07118f2a30bb5 in this term.
I20250625 20:05:33.428901 15393 leader_election.cc:304] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fb865ee6342943c09c3b0d631257eb89; no voters: 3206ae854c274fc79f8827d53c8cac18, cd293a4f251941f5b4b07118f2a30bb5
I20250625 20:05:33.429523 15532 raft_consensus.cc:2747] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250625 20:05:33.432202 15190 raft_consensus.cc:2466] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd293a4f251941f5b4b07118f2a30bb5 in term 1.
I20250625 20:05:33.433152 15260 leader_election.cc:304] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3206ae854c274fc79f8827d53c8cac18, cd293a4f251941f5b4b07118f2a30bb5; no voters: fb865ee6342943c09c3b0d631257eb89
I20250625 20:05:33.433807 15533 raft_consensus.cc:2802] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:05:33.435289 15533 raft_consensus.cc:695] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5 [term 1 LEADER]: Becoming Leader. State: Replica: cd293a4f251941f5b4b07118f2a30bb5, State: Running, Role: LEADER
I20250625 20:05:33.436056 15533 consensus_queue.cc:237] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd293a4f251941f5b4b07118f2a30bb5" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45129 } } peers { permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 } } peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } }
I20250625 20:05:33.446645 15047 catalog_manager.cc:5582] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd293a4f251941f5b4b07118f2a30bb5 (127.9.30.66). New cstate: current_term: 1 leader_uuid: "cd293a4f251941f5b4b07118f2a30bb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd293a4f251941f5b4b07118f2a30bb5" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } health_report { overall_health: UNKNOWN } } }
I20250625 20:05:33.614665 15190 raft_consensus.cc:1273] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [term 1 FOLLOWER]: Refusing update from remote peer cd293a4f251941f5b4b07118f2a30bb5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 20:05:33.615149 15457 raft_consensus.cc:1273] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89 [term 1 FOLLOWER]: Refusing update from remote peer cd293a4f251941f5b4b07118f2a30bb5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 20:05:33.616504 15539 consensus_queue.cc:1035] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250625 20:05:33.617448 15533 consensus_queue.cc:1035] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:05:33.654047 15547 mvcc.cc:204] Tried to move back new op lower bound from 7171612400067305472 to 7171612399361011712. Current Snapshot: MvccSnapshot[applied={T|T < 7171612400067305472}]
I20250625 20:05:33.660167 15549 mvcc.cc:204] Tried to move back new op lower bound from 7171612400067305472 to 7171612399361011712. Current Snapshot: MvccSnapshot[applied={T|T < 7171612400067305472}]
I20250625 20:05:33.663000 15548 mvcc.cc:204] Tried to move back new op lower bound from 7171612400067305472 to 7171612399361011712. Current Snapshot: MvccSnapshot[applied={T|T < 7171612400067305472}]
W20250625 20:05:36.156174 15544 meta_cache.cc:1261] Time spent looking up entry by key: real 0.079s user 0.000s sys 0.000s
W20250625 20:05:37.176827 15498 debug-util.cc:398] Leaking SignalData structure 0x7b08000b2140 after lost signal to thread 15374
W20250625 20:05:37.177887 15498 debug-util.cc:398] Leaking SignalData structure 0x7b08000b2800 after lost signal to thread 15501
I20250625 20:05:39.468217 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 15015
W20250625 20:05:39.746181 15236 heartbeater.cc:646] Failed to heartbeat to 127.9.30.126:33949 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.30.126:33949: connect: Connection refused (error 111)
W20250625 20:05:39.747150 15502 heartbeater.cc:646] Failed to heartbeat to 127.9.30.126:33949 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.30.126:33949: connect: Connection refused (error 111)
W20250625 20:05:39.749698 15369 heartbeater.cc:646] Failed to heartbeat to 127.9.30.126:33949 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.30.126:33949: connect: Connection refused (error 111)
W20250625 20:05:39.979396 15585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:05:39.980288 15585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:05:40.038177 15585 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250625 20:05:41.461067 15585 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.366s user 0.568s sys 0.795s
W20250625 20:05:41.461361 15585 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.366s user 0.568s sys 0.795s
I20250625 20:05:41.580544 15585 minidump.cc:252] Setting minidump size limit to 20M
I20250625 20:05:41.582717 15585 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:05:41.583881 15585 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:05:41.594208 15620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:05:41.594230 15621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:05:41.734237 15623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:05:41.734423 15585 server_base.cc:1048] running on GCE node
I20250625 20:05:41.735553 15585 hybrid_clock.cc:584] initializing the hybrid clock with 'system' time source
I20250625 20:05:41.736042 15585 hybrid_clock.cc:648] HybridClock initialized: now 1750881941736020 us; error 26608 us; skew 500 ppm
I20250625 20:05:41.736730 15585 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:05:41.741329 15585 webserver.cc:469] Webserver started at http://0.0.0.0:34641/ using document root <none> and password file <none>
I20250625 20:05:41.742152 15585 fs_manager.cc:362] Metadata directory not provided
I20250625 20:05:41.742365 15585 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:05:41.742753 15585 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 20:05:41.747215 15585 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/master-0/data/instance:
uuid: "abf18a0c4f204d7e8cf0340a9c64e914"
format_stamp: "Formatted at 2025-06-25 20:05:41 on dist-test-slave-jhxp"
I20250625 20:05:41.748397 15585 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/master-0/wal/instance:
uuid: "abf18a0c4f204d7e8cf0340a9c64e914"
format_stamp: "Formatted at 2025-06-25 20:05:41 on dist-test-slave-jhxp"
I20250625 20:05:41.754525 15585 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20250625 20:05:41.759459 15628 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:05:41.760469 15585 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250625 20:05:41.760800 15585 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/master-0/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/master-0/wal
uuid: "abf18a0c4f204d7e8cf0340a9c64e914"
format_stamp: "Formatted at 2025-06-25 20:05:41 on dist-test-slave-jhxp"
I20250625 20:05:41.761132 15585 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:05:41.831775 15585 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:05:41.833204 15585 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:05:41.833669 15585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:05:41.838274 15585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:05:41.853297 15585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914: Bootstrap starting.
I20250625 20:05:41.858628 15585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914: Neither blocks nor log segments found. Creating new log.
I20250625 20:05:41.860463 15585 log.cc:826] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914: Log is configured to *not* fsync() on all Append() calls
I20250625 20:05:41.864796 15585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914: No bootstrap required, opened a new log
I20250625 20:05:41.881268 15585 raft_consensus.cc:357] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf18a0c4f204d7e8cf0340a9c64e914" member_type: VOTER }
I20250625 20:05:41.881834 15585 raft_consensus.cc:383] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:05:41.882066 15585 raft_consensus.cc:738] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abf18a0c4f204d7e8cf0340a9c64e914, State: Initialized, Role: FOLLOWER
I20250625 20:05:41.882766 15585 consensus_queue.cc:260] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf18a0c4f204d7e8cf0340a9c64e914" member_type: VOTER }
I20250625 20:05:41.883224 15585 raft_consensus.cc:397] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 20:05:41.883497 15585 raft_consensus.cc:491] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 20:05:41.883807 15585 raft_consensus.cc:3058] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:05:41.887836 15585 raft_consensus.cc:513] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf18a0c4f204d7e8cf0340a9c64e914" member_type: VOTER }
I20250625 20:05:41.888557 15585 leader_election.cc:304] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abf18a0c4f204d7e8cf0340a9c64e914; no voters:
I20250625 20:05:41.890183 15585 leader_election.cc:290] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 20:05:41.890443 15638 raft_consensus.cc:2802] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:05:41.892438 15638 raft_consensus.cc:695] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914 [term 1 LEADER]: Becoming Leader. State: Replica: abf18a0c4f204d7e8cf0340a9c64e914, State: Running, Role: LEADER
I20250625 20:05:41.893369 15638 consensus_queue.cc:237] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf18a0c4f204d7e8cf0340a9c64e914" member_type: VOTER }
I20250625 20:05:41.901753 15640 sys_catalog.cc:455] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abf18a0c4f204d7e8cf0340a9c64e914. Latest consensus state: current_term: 1 leader_uuid: "abf18a0c4f204d7e8cf0340a9c64e914" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf18a0c4f204d7e8cf0340a9c64e914" member_type: VOTER } }
I20250625 20:05:41.901944 15639 sys_catalog.cc:455] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abf18a0c4f204d7e8cf0340a9c64e914" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf18a0c4f204d7e8cf0340a9c64e914" member_type: VOTER } }
I20250625 20:05:41.902446 15640 sys_catalog.cc:458] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914 [sys.catalog]: This master's current role is: LEADER
I20250625 20:05:41.902453 15639 sys_catalog.cc:458] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914 [sys.catalog]: This master's current role is: LEADER
I20250625 20:05:41.911967 15585 tablet_replica.cc:331] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914: stopping tablet replica
I20250625 20:05:41.912591 15585 raft_consensus.cc:2241] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914 [term 1 LEADER]: Raft consensus shutting down.
I20250625 20:05:41.913028 15585 raft_consensus.cc:2270] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 20:05:41.915110 15585 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250625 20:05:41.915553 15585 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250625 20:05:41.939826 15585 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
I20250625 20:05:42.968024 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 15107
I20250625 20:05:42.996028 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 15240
I20250625 20:05:43.036098 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 15373
I20250625 20:05:43.068555 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.30.126:33949
--webserver_interface=127.9.30.126
--webserver_port=38361
--builtin_ntp_servers=127.9.30.84:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.30.126:33949 with env {}
W20250625 20:05:43.368434 15647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:05:43.369037 15647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:05:43.369493 15647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:05:43.400884 15647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 20:05:43.401232 15647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:05:43.401481 15647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 20:05:43.401721 15647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 20:05:43.437297 15647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.30.126:33949
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.30.126:33949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.30.126
--webserver_port=38361
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:05:43.438905 15647 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:05:43.440656 15647 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:05:43.456727 15653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:05:43.456720 15654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:05:43.458333 15647 server_base.cc:1048] running on GCE node
W20250625 20:05:43.456707 15656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:05:44.558297 15647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:05:44.561403 15647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:05:44.562940 15647 hybrid_clock.cc:648] HybridClock initialized: now 1750881944562907 us; error 49 us; skew 500 ppm
I20250625 20:05:44.563788 15647 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:05:44.570362 15647 webserver.cc:469] Webserver started at http://127.9.30.126:38361/ using document root <none> and password file <none>
I20250625 20:05:44.571319 15647 fs_manager.cc:362] Metadata directory not provided
I20250625 20:05:44.571559 15647 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:05:44.579530 15647 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20250625 20:05:44.584249 15663 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:05:44.585384 15647 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250625 20:05:44.585741 15647 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/master-0/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/master-0/wal
uuid: "abf18a0c4f204d7e8cf0340a9c64e914"
format_stamp: "Formatted at 2025-06-25 20:05:41 on dist-test-slave-jhxp"
I20250625 20:05:44.587723 15647 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:05:44.670818 15647 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:05:44.672312 15647 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:05:44.672766 15647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:05:44.742553 15647 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.126:33949
I20250625 20:05:44.742631 15714 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.126:33949 every 8 connection(s)
I20250625 20:05:44.745414 15647 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/master-0/data/info.pb
I20250625 20:05:44.746063 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 15647
I20250625 20:05:44.747500 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.30.65:38305
--local_ip_for_outbound_sockets=127.9.30.65
--tserver_master_addrs=127.9.30.126:33949
--webserver_port=39933
--webserver_interface=127.9.30.65
--builtin_ntp_servers=127.9.30.84:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250625 20:05:44.756304 15715 sys_catalog.cc:263] Verifying existing consensus state
I20250625 20:05:44.771154 15715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914: Bootstrap starting.
I20250625 20:05:44.781535 15715 log.cc:826] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914: Log is configured to *not* fsync() on all Append() calls
I20250625 20:05:44.794342 15715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 20:05:44.795158 15715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914: Bootstrap complete.
I20250625 20:05:44.816064 15715 raft_consensus.cc:357] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf18a0c4f204d7e8cf0340a9c64e914" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 33949 } }
I20250625 20:05:44.816764 15715 raft_consensus.cc:738] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: abf18a0c4f204d7e8cf0340a9c64e914, State: Initialized, Role: FOLLOWER
I20250625 20:05:44.817525 15715 consensus_queue.cc:260] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf18a0c4f204d7e8cf0340a9c64e914" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 33949 } }
I20250625 20:05:44.818029 15715 raft_consensus.cc:397] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 20:05:44.818276 15715 raft_consensus.cc:491] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 20:05:44.818629 15715 raft_consensus.cc:3058] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914 [term 1 FOLLOWER]: Advancing to term 2
I20250625 20:05:44.823074 15715 raft_consensus.cc:513] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf18a0c4f204d7e8cf0340a9c64e914" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 33949 } }
I20250625 20:05:44.823935 15715 leader_election.cc:304] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abf18a0c4f204d7e8cf0340a9c64e914; no voters:
I20250625 20:05:44.825942 15715 leader_election.cc:290] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250625 20:05:44.826279 15719 raft_consensus.cc:2802] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914 [term 2 FOLLOWER]: Leader election won for term 2
I20250625 20:05:44.831035 15715 sys_catalog.cc:564] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914 [sys.catalog]: configured and running, proceeding with master startup.
I20250625 20:05:44.831984 15719 raft_consensus.cc:695] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914 [term 2 LEADER]: Becoming Leader. State: Replica: abf18a0c4f204d7e8cf0340a9c64e914, State: Running, Role: LEADER
I20250625 20:05:44.832955 15719 consensus_queue.cc:237] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf18a0c4f204d7e8cf0340a9c64e914" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 33949 } }
I20250625 20:05:44.847059 15720 sys_catalog.cc:455] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "abf18a0c4f204d7e8cf0340a9c64e914" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf18a0c4f204d7e8cf0340a9c64e914" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 33949 } } }
I20250625 20:05:44.848002 15720 sys_catalog.cc:458] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914 [sys.catalog]: This master's current role is: LEADER
I20250625 20:05:44.849931 15721 sys_catalog.cc:455] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abf18a0c4f204d7e8cf0340a9c64e914. Latest consensus state: current_term: 2 leader_uuid: "abf18a0c4f204d7e8cf0340a9c64e914" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf18a0c4f204d7e8cf0340a9c64e914" member_type: VOTER last_known_addr { host: "127.9.30.126" port: 33949 } } }
I20250625 20:05:44.850634 15721 sys_catalog.cc:458] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914 [sys.catalog]: This master's current role is: LEADER
I20250625 20:05:44.871867 15727 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 20:05:44.891786 15727 catalog_manager.cc:671] Loaded metadata for table pre_rebuild [id=30d1c4ad44134f0983a78ed397f7eebd]
I20250625 20:05:44.899554 15727 tablet_loader.cc:96] loaded metadata for tablet 56087ecb54f9431f9e13ceb2fb3ba969 (table pre_rebuild [id=30d1c4ad44134f0983a78ed397f7eebd])
I20250625 20:05:44.901161 15727 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 20:05:44.908887 15737 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250625 20:05:44.917198 15727 catalog_manager.cc:1349] Generated new cluster ID: 1cd97b2344314fcc9922fa8ebdf4f109
I20250625 20:05:44.917475 15727 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 20:05:44.932482 15727 catalog_manager.cc:1372] Generated new certificate authority record
I20250625 20:05:44.934012 15727 catalog_manager.cc:1506] Loading token signing keys...
I20250625 20:05:44.945688 15727 catalog_manager.cc:5955] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914: Generated new TSK 0
I20250625 20:05:44.946565 15727 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250625 20:05:45.107342 15717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:05:45.108037 15717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:05:45.108618 15717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:05:45.140249 15717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:05:45.141124 15717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.30.65
I20250625 20:05:45.176470 15717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.30.65:38305
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.30.65
--webserver_port=39933
--tserver_master_addrs=127.9.30.126:33949
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.30.65
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:05:45.177860 15717 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:05:45.179561 15717 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:05:45.197072 15744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:05:45.198948 15745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:05:45.200205 15717 server_base.cc:1048] running on GCE node
W20250625 20:05:45.200376 15747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:05:46.282505 15717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:05:46.285214 15717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:05:46.286664 15717 hybrid_clock.cc:648] HybridClock initialized: now 1750881946286601 us; error 78 us; skew 500 ppm
I20250625 20:05:46.287495 15717 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:05:46.294160 15717 webserver.cc:469] Webserver started at http://127.9.30.65:39933/ using document root <none> and password file <none>
I20250625 20:05:46.295104 15717 fs_manager.cc:362] Metadata directory not provided
I20250625 20:05:46.295325 15717 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:05:46.303531 15717 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20250625 20:05:46.309747 15754 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:05:46.310926 15717 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250625 20:05:46.311245 15717 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-0/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-0/wal
uuid: "3206ae854c274fc79f8827d53c8cac18"
format_stamp: "Formatted at 2025-06-25 20:05:29 on dist-test-slave-jhxp"
I20250625 20:05:46.313231 15717 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:05:46.369308 15717 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:05:46.370810 15717 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:05:46.371254 15717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:05:46.373932 15717 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 20:05:46.380259 15761 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250625 20:05:46.391747 15717 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250625 20:05:46.392011 15717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.002s sys 0.000s
I20250625 20:05:46.392275 15717 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250625 20:05:46.397673 15717 ts_tablet_manager.cc:610] Registered 1 tablets
I20250625 20:05:46.397886 15717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s user 0.000s sys 0.003s
I20250625 20:05:46.398408 15761 tablet_bootstrap.cc:492] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18: Bootstrap starting.
I20250625 20:05:46.579105 15717 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.65:38305
I20250625 20:05:46.579267 15867 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.65:38305 every 8 connection(s)
I20250625 20:05:46.581964 15717 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-0/data/info.pb
I20250625 20:05:46.591948 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 15717
I20250625 20:05:46.594247 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.30.66:45129
--local_ip_for_outbound_sockets=127.9.30.66
--tserver_master_addrs=127.9.30.126:33949
--webserver_port=45603
--webserver_interface=127.9.30.66
--builtin_ntp_servers=127.9.30.84:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250625 20:05:46.632310 15868 heartbeater.cc:344] Connected to a master server at 127.9.30.126:33949
I20250625 20:05:46.632879 15868 heartbeater.cc:461] Registering TS with master...
I20250625 20:05:46.634187 15868 heartbeater.cc:507] Master 127.9.30.126:33949 requested a full tablet report, sending...
I20250625 20:05:46.639329 15680 ts_manager.cc:194] Registered new tserver with Master: 3206ae854c274fc79f8827d53c8cac18 (127.9.30.65:38305)
I20250625 20:05:46.647823 15680 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.30.65:47509
I20250625 20:05:46.698083 15761 log.cc:826] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18: Log is configured to *not* fsync() on all Append() calls
W20250625 20:05:46.981058 15872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:05:46.981585 15872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:05:46.982141 15872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 20:05:47.013999 15872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:05:47.014912 15872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.30.66
I20250625 20:05:47.053790 15872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.30.66:45129
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.30.66
--webserver_port=45603
--tserver_master_addrs=127.9.30.126:33949
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.30.66
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:05:47.055135 15872 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:05:47.056910 15872 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:05:47.074153 15880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:05:47.652710 15868 heartbeater.cc:499] Master 127.9.30.126:33949 was elected leader, sending a full tablet report...
W20250625 20:05:47.076460 15882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:05:47.074828 15879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:05:48.162951 15881 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250625 20:05:48.163002 15872 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 20:05:48.168206 15872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:05:48.170994 15872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:05:48.172600 15872 hybrid_clock.cc:648] HybridClock initialized: now 1750881948172546 us; error 49 us; skew 500 ppm
I20250625 20:05:48.173696 15872 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:05:48.181460 15872 webserver.cc:469] Webserver started at http://127.9.30.66:45603/ using document root <none> and password file <none>
I20250625 20:05:48.182816 15872 fs_manager.cc:362] Metadata directory not provided
I20250625 20:05:48.183136 15872 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:05:48.193926 15872 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.002s sys 0.003s
I20250625 20:05:48.199971 15889 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:05:48.201241 15872 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250625 20:05:48.201668 15872 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-1/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-1/wal
uuid: "cd293a4f251941f5b4b07118f2a30bb5"
format_stamp: "Formatted at 2025-06-25 20:05:30 on dist-test-slave-jhxp"
I20250625 20:05:48.204584 15872 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:05:48.268606 15872 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:05:48.270620 15872 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:05:48.271219 15872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:05:48.274580 15872 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 20:05:48.282220 15896 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250625 20:05:48.290351 15872 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250625 20:05:48.290694 15872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.002s sys 0.001s
I20250625 20:05:48.291033 15872 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250625 20:05:48.298527 15872 ts_tablet_manager.cc:610] Registered 1 tablets
I20250625 20:05:48.298820 15872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.008s user 0.005s sys 0.000s
I20250625 20:05:48.299165 15896 tablet_bootstrap.cc:492] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5: Bootstrap starting.
I20250625 20:05:48.478832 15872 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.66:45129
I20250625 20:05:48.479033 16002 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.66:45129 every 8 connection(s)
I20250625 20:05:48.482748 15872 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-1/data/info.pb
I20250625 20:05:48.493002 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 15872
I20250625 20:05:48.495100 9337 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
/tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.30.67:33829
--local_ip_for_outbound_sockets=127.9.30.67
--tserver_master_addrs=127.9.30.126:33949
--webserver_port=38965
--webserver_interface=127.9.30.67
--builtin_ntp_servers=127.9.30.84:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250625 20:05:48.560873 16003 heartbeater.cc:344] Connected to a master server at 127.9.30.126:33949
I20250625 20:05:48.561451 16003 heartbeater.cc:461] Registering TS with master...
I20250625 20:05:48.562847 16003 heartbeater.cc:507] Master 127.9.30.126:33949 requested a full tablet report, sending...
I20250625 20:05:48.568068 15680 ts_manager.cc:194] Registered new tserver with Master: cd293a4f251941f5b4b07118f2a30bb5 (127.9.30.66:45129)
I20250625 20:05:48.571247 15680 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.30.66:37253
I20250625 20:05:48.601451 15896 log.cc:826] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5: Log is configured to *not* fsync() on all Append() calls
I20250625 20:05:48.976485 15761 tablet_bootstrap.cc:492] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 20:05:48.977531 15761 tablet_bootstrap.cc:492] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18: Bootstrap complete.
I20250625 20:05:48.979238 15761 ts_tablet_manager.cc:1397] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18: Time spent bootstrapping tablet: real 2.581s user 2.498s sys 0.060s
W20250625 20:05:48.986642 16007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 20:05:48.987334 16007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 20:05:48.988159 16007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250625 20:05:48.996651 15761 raft_consensus.cc:357] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd293a4f251941f5b4b07118f2a30bb5" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45129 } } peers { permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 } } peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } }
I20250625 20:05:48.999924 15761 raft_consensus.cc:738] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3206ae854c274fc79f8827d53c8cac18, State: Initialized, Role: FOLLOWER
I20250625 20:05:49.000880 15761 consensus_queue.cc:260] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd293a4f251941f5b4b07118f2a30bb5" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45129 } } peers { permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 } } peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } }
I20250625 20:05:49.004448 15761 ts_tablet_manager.cc:1428] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18: Time spent starting tablet: real 0.025s user 0.028s sys 0.000s
W20250625 20:05:49.033861 16007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 20:05:49.034755 16007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.30.67
I20250625 20:05:49.070238 16007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.30.84:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.30.67:33829
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.30.67
--webserver_port=38965
--tserver_master_addrs=127.9.30.126:33949
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.30.67
--log_dir=/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 25 Jun 2025 19:43:24 UTC on 5fd53c4cbb9d
build id 6769
TSAN enabled
I20250625 20:05:49.071553 16007 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 20:05:49.073287 16007 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 20:05:49.091912 16016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 20:05:49.584206 16003 heartbeater.cc:499] Master 127.9.30.126:33949 was elected leader, sending a full tablet report...
I20250625 20:05:50.146545 16021 raft_consensus.cc:491] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:05:50.147250 16021 raft_consensus.cc:513] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd293a4f251941f5b4b07118f2a30bb5" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45129 } } peers { permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 } } peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } }
I20250625 20:05:50.160311 16021 leader_election.cc:290] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd293a4f251941f5b4b07118f2a30bb5 (127.9.30.66:45129), fb865ee6342943c09c3b0d631257eb89 (127.9.30.67:33829)
W20250625 20:05:50.170173 15757 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.30.67:33829: connect: Connection refused (error 111)
W20250625 20:05:50.179987 15757 leader_election.cc:336] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fb865ee6342943c09c3b0d631257eb89 (127.9.30.67:33829): Network error: Client connection negotiation failed: client connection to 127.9.30.67:33829: connect: Connection refused (error 111)
I20250625 20:05:50.190307 15958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56087ecb54f9431f9e13ceb2fb3ba969" candidate_uuid: "3206ae854c274fc79f8827d53c8cac18" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "cd293a4f251941f5b4b07118f2a30bb5" is_pre_election: true
W20250625 20:05:50.204267 15756 leader_election.cc:343] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd293a4f251941f5b4b07118f2a30bb5 (127.9.30.66:45129): Illegal state: must be running to vote when last-logged opid is not known
I20250625 20:05:50.204813 15756 leader_election.cc:304] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3206ae854c274fc79f8827d53c8cac18; no voters: cd293a4f251941f5b4b07118f2a30bb5, fb865ee6342943c09c3b0d631257eb89
I20250625 20:05:50.206166 16021 raft_consensus.cc:2747] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 20:05:49.094237 16018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:05:49.091900 16015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 20:05:50.307658 16017 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1213 milliseconds
I20250625 20:05:50.307797 16007 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 20:05:50.309072 16007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 20:05:50.311733 16007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 20:05:50.313208 16007 hybrid_clock.cc:648] HybridClock initialized: now 1750881950313147 us; error 76 us; skew 500 ppm
I20250625 20:05:50.314039 16007 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 20:05:50.321003 16007 webserver.cc:469] Webserver started at http://127.9.30.67:38965/ using document root <none> and password file <none>
I20250625 20:05:50.321950 16007 fs_manager.cc:362] Metadata directory not provided
I20250625 20:05:50.322176 16007 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 20:05:50.330349 16007 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.002s sys 0.004s
I20250625 20:05:50.335321 16030 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 20:05:50.336467 16007 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20250625 20:05:50.336807 16007 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-2/data,/tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-2/wal
uuid: "fb865ee6342943c09c3b0d631257eb89"
format_stamp: "Formatted at 2025-06-25 20:05:32 on dist-test-slave-jhxp"
I20250625 20:05:50.338721 16007 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 20:05:50.389465 16007 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 20:05:50.390988 16007 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 20:05:50.391430 16007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 20:05:50.394546 16007 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 20:05:50.401167 16037 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250625 20:05:50.408794 16007 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250625 20:05:50.409032 16007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.002s sys 0.000s
I20250625 20:05:50.409307 16007 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250625 20:05:50.414098 16007 ts_tablet_manager.cc:610] Registered 1 tablets
I20250625 20:05:50.414378 16007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.003s sys 0.000s
I20250625 20:05:50.414775 16037 tablet_bootstrap.cc:492] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89: Bootstrap starting.
I20250625 20:05:50.656118 16007 rpc_server.cc:307] RPC server started. Bound to: 127.9.30.67:33829
I20250625 20:05:50.656447 16143 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.30.67:33829 every 8 connection(s)
I20250625 20:05:50.659027 16007 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-2/data/info.pb
I20250625 20:05:50.668089 9337 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu as pid 16007
I20250625 20:05:50.710081 16144 heartbeater.cc:344] Connected to a master server at 127.9.30.126:33949
I20250625 20:05:50.710713 16144 heartbeater.cc:461] Registering TS with master...
I20250625 20:05:50.711957 16144 heartbeater.cc:507] Master 127.9.30.126:33949 requested a full tablet report, sending...
I20250625 20:05:50.715207 15680 ts_manager.cc:194] Registered new tserver with Master: fb865ee6342943c09c3b0d631257eb89 (127.9.30.67:33829)
I20250625 20:05:50.717381 15680 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.30.67:33673
I20250625 20:05:50.722429 9337 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 20:05:50.736594 16037 log.cc:826] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89: Log is configured to *not* fsync() on all Append() calls
I20250625 20:05:51.127554 15896 tablet_bootstrap.cc:492] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 20:05:51.128388 15896 tablet_bootstrap.cc:492] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5: Bootstrap complete.
I20250625 20:05:51.129616 15896 ts_tablet_manager.cc:1397] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5: Time spent bootstrapping tablet: real 2.831s user 2.726s sys 0.088s
I20250625 20:05:51.135288 15896 raft_consensus.cc:357] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd293a4f251941f5b4b07118f2a30bb5" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45129 } } peers { permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 } } peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } }
I20250625 20:05:51.137332 15896 raft_consensus.cc:738] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd293a4f251941f5b4b07118f2a30bb5, State: Initialized, Role: FOLLOWER
I20250625 20:05:51.138049 15896 consensus_queue.cc:260] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd293a4f251941f5b4b07118f2a30bb5" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45129 } } peers { permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 } } peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } }
I20250625 20:05:51.141040 15896 ts_tablet_manager.cc:1428] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20250625 20:05:51.722291 16144 heartbeater.cc:499] Master 127.9.30.126:33949 was elected leader, sending a full tablet report...
I20250625 20:05:52.376048 16157 raft_consensus.cc:491] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:05:52.376434 16157 raft_consensus.cc:513] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd293a4f251941f5b4b07118f2a30bb5" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45129 } } peers { permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 } } peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } }
I20250625 20:05:52.377997 16157 leader_election.cc:290] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd293a4f251941f5b4b07118f2a30bb5 (127.9.30.66:45129), fb865ee6342943c09c3b0d631257eb89 (127.9.30.67:33829)
I20250625 20:05:52.378937 15958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56087ecb54f9431f9e13ceb2fb3ba969" candidate_uuid: "3206ae854c274fc79f8827d53c8cac18" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "cd293a4f251941f5b4b07118f2a30bb5" is_pre_election: true
I20250625 20:05:52.379745 15958 raft_consensus.cc:2466] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3206ae854c274fc79f8827d53c8cac18 in term 1.
I20250625 20:05:52.381362 15756 leader_election.cc:304] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3206ae854c274fc79f8827d53c8cac18, cd293a4f251941f5b4b07118f2a30bb5; no voters:
I20250625 20:05:52.382999 16157 raft_consensus.cc:2802] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250625 20:05:52.383415 16157 raft_consensus.cc:491] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:05:52.383873 16157 raft_consensus.cc:3058] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [term 1 FOLLOWER]: Advancing to term 2
I20250625 20:05:52.391909 16157 raft_consensus.cc:513] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd293a4f251941f5b4b07118f2a30bb5" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45129 } } peers { permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 } } peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } }
I20250625 20:05:52.393781 16157 leader_election.cc:290] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [CANDIDATE]: Term 2 election: Requested vote from peers cd293a4f251941f5b4b07118f2a30bb5 (127.9.30.66:45129), fb865ee6342943c09c3b0d631257eb89 (127.9.30.67:33829)
I20250625 20:05:52.394646 15958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56087ecb54f9431f9e13ceb2fb3ba969" candidate_uuid: "3206ae854c274fc79f8827d53c8cac18" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "cd293a4f251941f5b4b07118f2a30bb5"
I20250625 20:05:52.395237 15958 raft_consensus.cc:3058] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5 [term 1 FOLLOWER]: Advancing to term 2
I20250625 20:05:52.402299 15958 raft_consensus.cc:2466] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3206ae854c274fc79f8827d53c8cac18 in term 2.
I20250625 20:05:52.403312 15756 leader_election.cc:304] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3206ae854c274fc79f8827d53c8cac18, cd293a4f251941f5b4b07118f2a30bb5; no voters:
I20250625 20:05:52.404042 16157 raft_consensus.cc:2802] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [term 2 FOLLOWER]: Leader election won for term 2
I20250625 20:05:52.405726 16157 raft_consensus.cc:695] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [term 2 LEADER]: Becoming Leader. State: Replica: 3206ae854c274fc79f8827d53c8cac18, State: Running, Role: LEADER
I20250625 20:05:52.406589 16157 consensus_queue.cc:237] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd293a4f251941f5b4b07118f2a30bb5" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45129 } } peers { permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 } } peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } }
I20250625 20:05:52.395954 16099 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56087ecb54f9431f9e13ceb2fb3ba969" candidate_uuid: "3206ae854c274fc79f8827d53c8cac18" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "fb865ee6342943c09c3b0d631257eb89" is_pre_election: true
I20250625 20:05:52.396148 16098 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56087ecb54f9431f9e13ceb2fb3ba969" candidate_uuid: "3206ae854c274fc79f8827d53c8cac18" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "fb865ee6342943c09c3b0d631257eb89"
W20250625 20:05:52.410475 15757 leader_election.cc:343] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer fb865ee6342943c09c3b0d631257eb89 (127.9.30.67:33829): Illegal state: must be running to vote when last-logged opid is not known
W20250625 20:05:52.411545 15757 leader_election.cc:343] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fb865ee6342943c09c3b0d631257eb89 (127.9.30.67:33829): Illegal state: must be running to vote when last-logged opid is not known
I20250625 20:05:52.419008 15680 catalog_manager.cc:5582] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 reported cstate change: term changed from 0 to 2, leader changed from <none> to 3206ae854c274fc79f8827d53c8cac18 (127.9.30.65), VOTER 3206ae854c274fc79f8827d53c8cac18 (127.9.30.65) added, VOTER cd293a4f251941f5b4b07118f2a30bb5 (127.9.30.66) added, VOTER fb865ee6342943c09c3b0d631257eb89 (127.9.30.67) added. New cstate: current_term: 2 leader_uuid: "3206ae854c274fc79f8827d53c8cac18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd293a4f251941f5b4b07118f2a30bb5" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } health_report { overall_health: HEALTHY } } }
I20250625 20:05:52.729837 16037 tablet_bootstrap.cc:492] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 20:05:52.730618 16037 tablet_bootstrap.cc:492] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89: Bootstrap complete.
I20250625 20:05:52.731875 16037 ts_tablet_manager.cc:1397] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89: Time spent bootstrapping tablet: real 2.318s user 2.239s sys 0.052s
I20250625 20:05:52.737150 16037 raft_consensus.cc:357] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd293a4f251941f5b4b07118f2a30bb5" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45129 } } peers { permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 } } peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } }
I20250625 20:05:52.739224 16037 raft_consensus.cc:738] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb865ee6342943c09c3b0d631257eb89, State: Initialized, Role: FOLLOWER
I20250625 20:05:52.739965 16037 consensus_queue.cc:260] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd293a4f251941f5b4b07118f2a30bb5" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45129 } } peers { permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 } } peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } }
I20250625 20:05:52.742931 16037 ts_tablet_manager.cc:1428] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20250625 20:05:52.853250 16099 raft_consensus.cc:3058] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89 [term 1 FOLLOWER]: Advancing to term 2
I20250625 20:05:52.860071 16099 raft_consensus.cc:1273] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89 [term 2 FOLLOWER]: Refusing update from remote peer 3206ae854c274fc79f8827d53c8cac18: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250625 20:05:52.861534 16157 consensus_queue.cc:1035] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
I20250625 20:05:52.873862 15958 raft_consensus.cc:1273] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5 [term 2 FOLLOWER]: Refusing update from remote peer 3206ae854c274fc79f8827d53c8cac18: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250625 20:05:52.875537 16157 consensus_queue.cc:1035] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd293a4f251941f5b4b07118f2a30bb5" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
I20250625 20:05:52.907966 15823 consensus_queue.cc:237] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 2.206, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 } } peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } }
I20250625 20:05:52.912323 16099 raft_consensus.cc:1273] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89 [term 2 FOLLOWER]: Refusing update from remote peer 3206ae854c274fc79f8827d53c8cac18: Log matching property violated. Preceding OpId in replica: term: 2 index: 206. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
I20250625 20:05:52.913887 16157 consensus_queue.cc:1035] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.000s
I20250625 20:05:52.919963 16170 raft_consensus.cc:2953] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [term 2 LEADER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER cd293a4f251941f5b4b07118f2a30bb5 (127.9.30.66) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 } } peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } } }
I20250625 20:05:52.921710 16099 raft_consensus.cc:2953] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89 [term 2 FOLLOWER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER cd293a4f251941f5b4b07118f2a30bb5 (127.9.30.66) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 } } peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } } }
I20250625 20:05:52.931159 15667 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 56087ecb54f9431f9e13ceb2fb3ba969 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250625 20:05:52.934444 15680 catalog_manager.cc:5582] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89 reported cstate change: config changed from index -1 to 207, VOTER cd293a4f251941f5b4b07118f2a30bb5 (127.9.30.66) evicted. New cstate: current_term: 2 leader_uuid: "3206ae854c274fc79f8827d53c8cac18" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 } } peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } } }
I20250625 20:05:52.945642 15823 consensus_queue.cc:237] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 2.207, Last appended by leader: 205, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } }
I20250625 20:05:52.948345 16170 raft_consensus.cc:2953] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [term 2 LEADER]: Committing config change with OpId 2.208: config changed from index 207 to 208, VOTER fb865ee6342943c09c3b0d631257eb89 (127.9.30.67) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } } }
I20250625 20:05:52.957644 15667 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 56087ecb54f9431f9e13ceb2fb3ba969 with cas_config_opid_index 207: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250625 20:05:52.961279 15938 tablet_service.cc:1515] Processing DeleteTablet for tablet 56087ecb54f9431f9e13ceb2fb3ba969 with delete_type TABLET_DATA_TOMBSTONED (TS cd293a4f251941f5b4b07118f2a30bb5 not found in new config with opid_index 207) from {username='slave'} at 127.0.0.1:50660
I20250625 20:05:52.962445 15680 catalog_manager.cc:5582] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 reported cstate change: config changed from index 207 to 208, VOTER fb865ee6342943c09c3b0d631257eb89 (127.9.30.67) evicted. New cstate: current_term: 2 leader_uuid: "3206ae854c274fc79f8827d53c8cac18" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } health_report { overall_health: HEALTHY } } }
I20250625 20:05:52.976137 16181 tablet_replica.cc:331] stopping tablet replica
I20250625 20:05:52.977211 16181 raft_consensus.cc:2241] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250625 20:05:52.977981 16181 raft_consensus.cc:2270] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250625 20:05:52.999639 16079 tablet_service.cc:1515] Processing DeleteTablet for tablet 56087ecb54f9431f9e13ceb2fb3ba969 with delete_type TABLET_DATA_TOMBSTONED (TS fb865ee6342943c09c3b0d631257eb89 not found in new config with opid_index 208) from {username='slave'} at 127.0.0.1:36122
I20250625 20:05:53.006439 16181 ts_tablet_manager.cc:1905] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250625 20:05:53.015372 16183 tablet_replica.cc:331] stopping tablet replica
I20250625 20:05:53.016445 16183 raft_consensus.cc:2241] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250625 20:05:53.021910 16183 raft_consensus.cc:2270] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250625 20:05:53.027663 16181 ts_tablet_manager.cc:1918] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.206
I20250625 20:05:53.028200 16181 log.cc:1199] T 56087ecb54f9431f9e13ceb2fb3ba969 P cd293a4f251941f5b4b07118f2a30bb5: Deleting WAL directory at /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-1/wal/wals/56087ecb54f9431f9e13ceb2fb3ba969
I20250625 20:05:53.030244 15665 catalog_manager.cc:4928] TS cd293a4f251941f5b4b07118f2a30bb5 (127.9.30.66:45129): tablet 56087ecb54f9431f9e13ceb2fb3ba969 (table pre_rebuild [id=30d1c4ad44134f0983a78ed397f7eebd]) successfully deleted
I20250625 20:05:53.050734 16183 ts_tablet_manager.cc:1905] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250625 20:05:53.067273 16183 ts_tablet_manager.cc:1918] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.207
I20250625 20:05:53.067764 16183 log.cc:1199] T 56087ecb54f9431f9e13ceb2fb3ba969 P fb865ee6342943c09c3b0d631257eb89: Deleting WAL directory at /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-2/wal/wals/56087ecb54f9431f9e13ceb2fb3ba969
I20250625 20:05:53.069666 15666 catalog_manager.cc:4928] TS fb865ee6342943c09c3b0d631257eb89 (127.9.30.67:33829): tablet 56087ecb54f9431f9e13ceb2fb3ba969 (table pre_rebuild [id=30d1c4ad44134f0983a78ed397f7eebd]) successfully deleted
W20250625 20:05:53.178594 9337 scanner-internal.cc:458] Time spent opening tablet: real 2.421s user 0.011s sys 0.002s
I20250625 20:05:53.797637 15803 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250625 20:05:53.820886 16079 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 20:05:53.821148 15938 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
abf18a0c4f204d7e8cf0340a9c64e914 | 127.9.30.126:33949 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.9.30.84:42931 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
3206ae854c274fc79f8827d53c8cac18 | 127.9.30.65:38305 | HEALTHY | <none> | 1 | 0
cd293a4f251941f5b4b07118f2a30bb5 | 127.9.30.66:45129 | HEALTHY | <none> | 0 | 0
fb865ee6342943c09c3b0d631257eb89 | 127.9.30.67:33829 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.9.30.65 | experimental | 127.9.30.65:38305
local_ip_for_outbound_sockets | 127.9.30.66 | experimental | 127.9.30.66:45129
local_ip_for_outbound_sockets | 127.9.30.67 | experimental | 127.9.30.67:33829
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-0/data/info.pb | hidden | 127.9.30.65:38305
server_dump_info_path | /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-1/data/info.pb | hidden | 127.9.30.66:45129
server_dump_info_path | /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-2/data/info.pb | hidden | 127.9.30.67:33829
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.9.30.84:42931 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------+----+---------+---------------+---------+------------+------------------+-------------
pre_rebuild | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 0
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 1
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250625 20:05:54.121057 9337 log_verifier.cc:126] Checking tablet 56087ecb54f9431f9e13ceb2fb3ba969
I20250625 20:05:54.377471 9337 log_verifier.cc:177] Verified matching terms for 208 ops in tablet 56087ecb54f9431f9e13ceb2fb3ba969
I20250625 20:05:54.380082 15678 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58438:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250625 20:05:54.380656 15678 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58438:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250625 20:05:54.394133 15678 catalog_manager.cc:5869] T 00000000000000000000000000000000 P abf18a0c4f204d7e8cf0340a9c64e914: Sending DeleteTablet for 1 replicas of tablet 56087ecb54f9431f9e13ceb2fb3ba969
I20250625 20:05:54.396170 9337 test_util.cc:276] Using random seed: -118523922
I20250625 20:05:54.396003 15803 tablet_service.cc:1515] Processing DeleteTablet for tablet 56087ecb54f9431f9e13ceb2fb3ba969 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 20:05:54 UTC) from {username='slave'} at 127.0.0.1:48970
I20250625 20:05:54.397979 16215 tablet_replica.cc:331] stopping tablet replica
I20250625 20:05:54.398818 16215 raft_consensus.cc:2241] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [term 2 LEADER]: Raft consensus shutting down.
I20250625 20:05:54.399526 16215 raft_consensus.cc:2270] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250625 20:05:54.429793 16215 ts_tablet_manager.cc:1905] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 20:05:54.431756 15678 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58482:
name: "post_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 20:05:54.435540 15678 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table post_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 20:05:54.441668 16215 ts_tablet_manager.cc:1918] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.208
I20250625 20:05:54.442114 16215 log.cc:1199] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18: Deleting WAL directory at /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-0/wal/wals/56087ecb54f9431f9e13ceb2fb3ba969
I20250625 20:05:54.443020 16215 ts_tablet_manager.cc:1939] T 56087ecb54f9431f9e13ceb2fb3ba969 P 3206ae854c274fc79f8827d53c8cac18: Deleting consensus metadata
I20250625 20:05:54.445909 15667 catalog_manager.cc:4928] TS 3206ae854c274fc79f8827d53c8cac18 (127.9.30.65:38305): tablet 56087ecb54f9431f9e13ceb2fb3ba969 (table pre_rebuild [id=30d1c4ad44134f0983a78ed397f7eebd]) successfully deleted
I20250625 20:05:54.462013 16079 tablet_service.cc:1468] Processing CreateTablet for tablet 9db294b5b2bb48a6a7b056de005f61c7 (DEFAULT_TABLE table=post_rebuild [id=7d6a7b29bebe4eeb89f4173413536f93]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:05:54.462384 15938 tablet_service.cc:1468] Processing CreateTablet for tablet 9db294b5b2bb48a6a7b056de005f61c7 (DEFAULT_TABLE table=post_rebuild [id=7d6a7b29bebe4eeb89f4173413536f93]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:05:54.463511 15938 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9db294b5b2bb48a6a7b056de005f61c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:05:54.463466 16079 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9db294b5b2bb48a6a7b056de005f61c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:05:54.469975 15803 tablet_service.cc:1468] Processing CreateTablet for tablet 9db294b5b2bb48a6a7b056de005f61c7 (DEFAULT_TABLE table=post_rebuild [id=7d6a7b29bebe4eeb89f4173413536f93]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 20:05:54.471426 15803 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9db294b5b2bb48a6a7b056de005f61c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 20:05:54.483278 16223 tablet_bootstrap.cc:492] T 9db294b5b2bb48a6a7b056de005f61c7 P fb865ee6342943c09c3b0d631257eb89: Bootstrap starting.
I20250625 20:05:54.485525 16222 tablet_bootstrap.cc:492] T 9db294b5b2bb48a6a7b056de005f61c7 P cd293a4f251941f5b4b07118f2a30bb5: Bootstrap starting.
I20250625 20:05:54.491931 16223 tablet_bootstrap.cc:654] T 9db294b5b2bb48a6a7b056de005f61c7 P fb865ee6342943c09c3b0d631257eb89: Neither blocks nor log segments found. Creating new log.
I20250625 20:05:54.499440 16222 tablet_bootstrap.cc:654] T 9db294b5b2bb48a6a7b056de005f61c7 P cd293a4f251941f5b4b07118f2a30bb5: Neither blocks nor log segments found. Creating new log.
I20250625 20:05:54.504768 16224 tablet_bootstrap.cc:492] T 9db294b5b2bb48a6a7b056de005f61c7 P 3206ae854c274fc79f8827d53c8cac18: Bootstrap starting.
I20250625 20:05:54.509043 16223 tablet_bootstrap.cc:492] T 9db294b5b2bb48a6a7b056de005f61c7 P fb865ee6342943c09c3b0d631257eb89: No bootstrap required, opened a new log
I20250625 20:05:54.509490 16223 ts_tablet_manager.cc:1397] T 9db294b5b2bb48a6a7b056de005f61c7 P fb865ee6342943c09c3b0d631257eb89: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.006s
I20250625 20:05:54.512190 16223 raft_consensus.cc:357] T 9db294b5b2bb48a6a7b056de005f61c7 P fb865ee6342943c09c3b0d631257eb89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 } } peers { permanent_uuid: "cd293a4f251941f5b4b07118f2a30bb5" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45129 } } peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } }
I20250625 20:05:54.512912 16223 raft_consensus.cc:383] T 9db294b5b2bb48a6a7b056de005f61c7 P fb865ee6342943c09c3b0d631257eb89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:05:54.513214 16223 raft_consensus.cc:738] T 9db294b5b2bb48a6a7b056de005f61c7 P fb865ee6342943c09c3b0d631257eb89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb865ee6342943c09c3b0d631257eb89, State: Initialized, Role: FOLLOWER
I20250625 20:05:54.513882 16223 consensus_queue.cc:260] T 9db294b5b2bb48a6a7b056de005f61c7 P fb865ee6342943c09c3b0d631257eb89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 } } peers { permanent_uuid: "cd293a4f251941f5b4b07118f2a30bb5" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45129 } } peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } }
I20250625 20:05:54.514813 16222 tablet_bootstrap.cc:492] T 9db294b5b2bb48a6a7b056de005f61c7 P cd293a4f251941f5b4b07118f2a30bb5: No bootstrap required, opened a new log
I20250625 20:05:54.515302 16222 ts_tablet_manager.cc:1397] T 9db294b5b2bb48a6a7b056de005f61c7 P cd293a4f251941f5b4b07118f2a30bb5: Time spent bootstrapping tablet: real 0.030s user 0.014s sys 0.004s
I20250625 20:05:54.516409 16224 tablet_bootstrap.cc:654] T 9db294b5b2bb48a6a7b056de005f61c7 P 3206ae854c274fc79f8827d53c8cac18: Neither blocks nor log segments found. Creating new log.
I20250625 20:05:54.518100 16222 raft_consensus.cc:357] T 9db294b5b2bb48a6a7b056de005f61c7 P cd293a4f251941f5b4b07118f2a30bb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 } } peers { permanent_uuid: "cd293a4f251941f5b4b07118f2a30bb5" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45129 } } peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } }
I20250625 20:05:54.518992 16222 raft_consensus.cc:383] T 9db294b5b2bb48a6a7b056de005f61c7 P cd293a4f251941f5b4b07118f2a30bb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:05:54.519320 16222 raft_consensus.cc:738] T 9db294b5b2bb48a6a7b056de005f61c7 P cd293a4f251941f5b4b07118f2a30bb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd293a4f251941f5b4b07118f2a30bb5, State: Initialized, Role: FOLLOWER
I20250625 20:05:54.521349 16222 consensus_queue.cc:260] T 9db294b5b2bb48a6a7b056de005f61c7 P cd293a4f251941f5b4b07118f2a30bb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 } } peers { permanent_uuid: "cd293a4f251941f5b4b07118f2a30bb5" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45129 } } peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } }
I20250625 20:05:54.524716 16223 ts_tablet_manager.cc:1428] T 9db294b5b2bb48a6a7b056de005f61c7 P fb865ee6342943c09c3b0d631257eb89: Time spent starting tablet: real 0.015s user 0.008s sys 0.008s
I20250625 20:05:54.528033 16224 tablet_bootstrap.cc:492] T 9db294b5b2bb48a6a7b056de005f61c7 P 3206ae854c274fc79f8827d53c8cac18: No bootstrap required, opened a new log
I20250625 20:05:54.528541 16224 ts_tablet_manager.cc:1397] T 9db294b5b2bb48a6a7b056de005f61c7 P 3206ae854c274fc79f8827d53c8cac18: Time spent bootstrapping tablet: real 0.024s user 0.012s sys 0.000s
I20250625 20:05:54.531230 16224 raft_consensus.cc:357] T 9db294b5b2bb48a6a7b056de005f61c7 P 3206ae854c274fc79f8827d53c8cac18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 } } peers { permanent_uuid: "cd293a4f251941f5b4b07118f2a30bb5" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45129 } } peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } }
I20250625 20:05:54.532090 16224 raft_consensus.cc:383] T 9db294b5b2bb48a6a7b056de005f61c7 P 3206ae854c274fc79f8827d53c8cac18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 20:05:54.532423 16224 raft_consensus.cc:738] T 9db294b5b2bb48a6a7b056de005f61c7 P 3206ae854c274fc79f8827d53c8cac18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3206ae854c274fc79f8827d53c8cac18, State: Initialized, Role: FOLLOWER
I20250625 20:05:54.533182 16224 consensus_queue.cc:260] T 9db294b5b2bb48a6a7b056de005f61c7 P 3206ae854c274fc79f8827d53c8cac18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 } } peers { permanent_uuid: "cd293a4f251941f5b4b07118f2a30bb5" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45129 } } peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } }
I20250625 20:05:54.541353 16222 ts_tablet_manager.cc:1428] T 9db294b5b2bb48a6a7b056de005f61c7 P cd293a4f251941f5b4b07118f2a30bb5: Time spent starting tablet: real 0.026s user 0.004s sys 0.004s
I20250625 20:05:54.545292 16224 ts_tablet_manager.cc:1428] T 9db294b5b2bb48a6a7b056de005f61c7 P 3206ae854c274fc79f8827d53c8cac18: Time spent starting tablet: real 0.016s user 0.005s sys 0.010s
W20250625 20:05:54.563452 16004 tablet.cc:2378] T 9db294b5b2bb48a6a7b056de005f61c7 P cd293a4f251941f5b4b07118f2a30bb5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 20:05:54.612155 16229 raft_consensus.cc:491] T 9db294b5b2bb48a6a7b056de005f61c7 P cd293a4f251941f5b4b07118f2a30bb5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250625 20:05:54.613061 15869 tablet.cc:2378] T 9db294b5b2bb48a6a7b056de005f61c7 P 3206ae854c274fc79f8827d53c8cac18: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 20:05:54.612735 16229 raft_consensus.cc:513] T 9db294b5b2bb48a6a7b056de005f61c7 P cd293a4f251941f5b4b07118f2a30bb5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 } } peers { permanent_uuid: "cd293a4f251941f5b4b07118f2a30bb5" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45129 } } peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } }
I20250625 20:05:54.614470 16230 raft_consensus.cc:491] T 9db294b5b2bb48a6a7b056de005f61c7 P 3206ae854c274fc79f8827d53c8cac18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 20:05:54.614984 16230 raft_consensus.cc:513] T 9db294b5b2bb48a6a7b056de005f61c7 P 3206ae854c274fc79f8827d53c8cac18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 } } peers { permanent_uuid: "cd293a4f251941f5b4b07118f2a30bb5" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45129 } } peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } }
I20250625 20:05:54.617029 16230 leader_election.cc:290] T 9db294b5b2bb48a6a7b056de005f61c7 P 3206ae854c274fc79f8827d53c8cac18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb865ee6342943c09c3b0d631257eb89 (127.9.30.67:33829), cd293a4f251941f5b4b07118f2a30bb5 (127.9.30.66:45129)
I20250625 20:05:54.619432 16099 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9db294b5b2bb48a6a7b056de005f61c7" candidate_uuid: "3206ae854c274fc79f8827d53c8cac18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb865ee6342943c09c3b0d631257eb89" is_pre_election: true
I20250625 20:05:54.620213 16099 raft_consensus.cc:2466] T 9db294b5b2bb48a6a7b056de005f61c7 P fb865ee6342943c09c3b0d631257eb89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3206ae854c274fc79f8827d53c8cac18 in term 0.
I20250625 20:05:54.620406 15958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9db294b5b2bb48a6a7b056de005f61c7" candidate_uuid: "3206ae854c274fc79f8827d53c8cac18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd293a4f251941f5b4b07118f2a30bb5" is_pre_election: true
I20250625 20:05:54.621084 15958 raft_consensus.cc:2466] T 9db294b5b2bb48a6a7b056de005f61c7 P cd293a4f251941f5b4b07118f2a30bb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3206ae854c274fc79f8827d53c8cac18 in term 0.
I20250625 20:05:54.621621 15757 leader_election.cc:304] T 9db294b5b2bb48a6a7b056de005f61c7 P 3206ae854c274fc79f8827d53c8cac18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3206ae854c274fc79f8827d53c8cac18, fb865ee6342943c09c3b0d631257eb89; no voters:
I20250625 20:05:54.622526 16230 raft_consensus.cc:2802] T 9db294b5b2bb48a6a7b056de005f61c7 P 3206ae854c274fc79f8827d53c8cac18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 20:05:54.622938 16230 raft_consensus.cc:491] T 9db294b5b2bb48a6a7b056de005f61c7 P 3206ae854c274fc79f8827d53c8cac18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 20:05:54.623268 16229 leader_election.cc:290] T 9db294b5b2bb48a6a7b056de005f61c7 P cd293a4f251941f5b4b07118f2a30bb5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb865ee6342943c09c3b0d631257eb89 (127.9.30.67:33829), 3206ae854c274fc79f8827d53c8cac18 (127.9.30.65:38305)
I20250625 20:05:54.624940 16230 raft_consensus.cc:3058] T 9db294b5b2bb48a6a7b056de005f61c7 P 3206ae854c274fc79f8827d53c8cac18 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:05:54.632205 16230 raft_consensus.cc:513] T 9db294b5b2bb48a6a7b056de005f61c7 P 3206ae854c274fc79f8827d53c8cac18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 } } peers { permanent_uuid: "cd293a4f251941f5b4b07118f2a30bb5" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45129 } } peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } }
I20250625 20:05:54.635897 16230 leader_election.cc:290] T 9db294b5b2bb48a6a7b056de005f61c7 P 3206ae854c274fc79f8827d53c8cac18 [CANDIDATE]: Term 1 election: Requested vote from peers fb865ee6342943c09c3b0d631257eb89 (127.9.30.67:33829), cd293a4f251941f5b4b07118f2a30bb5 (127.9.30.66:45129)
I20250625 20:05:54.636734 16099 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9db294b5b2bb48a6a7b056de005f61c7" candidate_uuid: "3206ae854c274fc79f8827d53c8cac18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb865ee6342943c09c3b0d631257eb89"
I20250625 20:05:54.637301 16099 raft_consensus.cc:3058] T 9db294b5b2bb48a6a7b056de005f61c7 P fb865ee6342943c09c3b0d631257eb89 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:05:54.637166 15958 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9db294b5b2bb48a6a7b056de005f61c7" candidate_uuid: "3206ae854c274fc79f8827d53c8cac18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd293a4f251941f5b4b07118f2a30bb5"
I20250625 20:05:54.637718 15958 raft_consensus.cc:3058] T 9db294b5b2bb48a6a7b056de005f61c7 P cd293a4f251941f5b4b07118f2a30bb5 [term 0 FOLLOWER]: Advancing to term 1
I20250625 20:05:54.644132 16099 raft_consensus.cc:2466] T 9db294b5b2bb48a6a7b056de005f61c7 P fb865ee6342943c09c3b0d631257eb89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3206ae854c274fc79f8827d53c8cac18 in term 1.
I20250625 20:05:54.644948 16098 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9db294b5b2bb48a6a7b056de005f61c7" candidate_uuid: "cd293a4f251941f5b4b07118f2a30bb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb865ee6342943c09c3b0d631257eb89" is_pre_election: true
I20250625 20:05:54.645787 16098 raft_consensus.cc:2391] T 9db294b5b2bb48a6a7b056de005f61c7 P fb865ee6342943c09c3b0d631257eb89 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd293a4f251941f5b4b07118f2a30bb5 in current term 1: Already voted for candidate 3206ae854c274fc79f8827d53c8cac18 in this term.
I20250625 20:05:54.645699 15958 raft_consensus.cc:2466] T 9db294b5b2bb48a6a7b056de005f61c7 P cd293a4f251941f5b4b07118f2a30bb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3206ae854c274fc79f8827d53c8cac18 in term 1.
I20250625 20:05:54.647235 15756 leader_election.cc:304] T 9db294b5b2bb48a6a7b056de005f61c7 P 3206ae854c274fc79f8827d53c8cac18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3206ae854c274fc79f8827d53c8cac18, cd293a4f251941f5b4b07118f2a30bb5; no voters:
I20250625 20:05:54.648415 16230 raft_consensus.cc:2802] T 9db294b5b2bb48a6a7b056de005f61c7 P 3206ae854c274fc79f8827d53c8cac18 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 20:05:54.650569 16230 raft_consensus.cc:695] T 9db294b5b2bb48a6a7b056de005f61c7 P 3206ae854c274fc79f8827d53c8cac18 [term 1 LEADER]: Becoming Leader. State: Replica: 3206ae854c274fc79f8827d53c8cac18, State: Running, Role: LEADER
I20250625 20:05:54.651468 16230 consensus_queue.cc:237] T 9db294b5b2bb48a6a7b056de005f61c7 P 3206ae854c274fc79f8827d53c8cac18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 } } peers { permanent_uuid: "cd293a4f251941f5b4b07118f2a30bb5" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45129 } } peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } }
I20250625 20:05:54.657164 15823 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9db294b5b2bb48a6a7b056de005f61c7" candidate_uuid: "cd293a4f251941f5b4b07118f2a30bb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3206ae854c274fc79f8827d53c8cac18" is_pre_election: true
I20250625 20:05:54.659070 15893 leader_election.cc:304] T 9db294b5b2bb48a6a7b056de005f61c7 P cd293a4f251941f5b4b07118f2a30bb5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd293a4f251941f5b4b07118f2a30bb5; no voters: 3206ae854c274fc79f8827d53c8cac18, fb865ee6342943c09c3b0d631257eb89
I20250625 20:05:54.660384 16229 raft_consensus.cc:2747] T 9db294b5b2bb48a6a7b056de005f61c7 P cd293a4f251941f5b4b07118f2a30bb5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 20:05:54.666059 15678 catalog_manager.cc:5582] T 9db294b5b2bb48a6a7b056de005f61c7 P 3206ae854c274fc79f8827d53c8cac18 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3206ae854c274fc79f8827d53c8cac18 (127.9.30.65). New cstate: current_term: 1 leader_uuid: "3206ae854c274fc79f8827d53c8cac18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd293a4f251941f5b4b07118f2a30bb5" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3206ae854c274fc79f8827d53c8cac18" member_type: VOTER last_known_addr { host: "127.9.30.65" port: 38305 } health_report { overall_health: HEALTHY } } }
W20250625 20:05:54.674151 16145 tablet.cc:2378] T 9db294b5b2bb48a6a7b056de005f61c7 P fb865ee6342943c09c3b0d631257eb89: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 20:05:54.871884 15958 raft_consensus.cc:1273] T 9db294b5b2bb48a6a7b056de005f61c7 P cd293a4f251941f5b4b07118f2a30bb5 [term 1 FOLLOWER]: Refusing update from remote peer 3206ae854c274fc79f8827d53c8cac18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 20:05:54.871948 16098 raft_consensus.cc:1273] T 9db294b5b2bb48a6a7b056de005f61c7 P fb865ee6342943c09c3b0d631257eb89 [term 1 FOLLOWER]: Refusing update from remote peer 3206ae854c274fc79f8827d53c8cac18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 20:05:54.873754 16235 consensus_queue.cc:1035] T 9db294b5b2bb48a6a7b056de005f61c7 P 3206ae854c274fc79f8827d53c8cac18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb865ee6342943c09c3b0d631257eb89" member_type: VOTER last_known_addr { host: "127.9.30.67" port: 33829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250625 20:05:54.874485 16230 consensus_queue.cc:1035] T 9db294b5b2bb48a6a7b056de005f61c7 P 3206ae854c274fc79f8827d53c8cac18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd293a4f251941f5b4b07118f2a30bb5" member_type: VOTER last_known_addr { host: "127.9.30.66" port: 45129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 20:05:54.915930 16242 mvcc.cc:204] Tried to move back new op lower bound from 7171612487139250176 to 7171612486262382592. Current Snapshot: MvccSnapshot[applied={T|T < 7171612487139250176}]
I20250625 20:05:54.919035 16245 mvcc.cc:204] Tried to move back new op lower bound from 7171612487139250176 to 7171612486262382592. Current Snapshot: MvccSnapshot[applied={T|T < 7171612487139250176}]
I20250625 20:05:54.928457 16244 mvcc.cc:204] Tried to move back new op lower bound from 7171612487139250176 to 7171612486262382592. Current Snapshot: MvccSnapshot[applied={T|T < 7171612487139250176}]
W20250625 20:05:57.316756 15711 debug-util.cc:398] Leaking SignalData structure 0x7b08000883e0 after lost signal to thread 15648
W20250625 20:05:57.317452 15711 debug-util.cc:398] Leaking SignalData structure 0x7b08000a0f80 after lost signal to thread 15714
W20250625 20:06:00.288277 15864 debug-util.cc:398] Leaking SignalData structure 0x7b08000ae3c0 after lost signal to thread 15739
W20250625 20:06:00.289397 15864 debug-util.cc:398] Leaking SignalData structure 0x7b08000b9b60 after lost signal to thread 15867
I20250625 20:06:00.779080 15803 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250625 20:06:00.792393 16079 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 20:06:00.793051 15938 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
abf18a0c4f204d7e8cf0340a9c64e914 | 127.9.30.126:33949 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.9.30.84:42931 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
3206ae854c274fc79f8827d53c8cac18 | 127.9.30.65:38305 | HEALTHY | <none> | 1 | 0
cd293a4f251941f5b4b07118f2a30bb5 | 127.9.30.66:45129 | HEALTHY | <none> | 0 | 0
fb865ee6342943c09c3b0d631257eb89 | 127.9.30.67:33829 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.9.30.65 | experimental | 127.9.30.65:38305
local_ip_for_outbound_sockets | 127.9.30.66 | experimental | 127.9.30.66:45129
local_ip_for_outbound_sockets | 127.9.30.67 | experimental | 127.9.30.67:33829
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-0/data/info.pb | hidden | 127.9.30.65:38305
server_dump_info_path | /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-1/data/info.pb | hidden | 127.9.30.66:45129
server_dump_info_path | /tmp/dist-test-taskV0EjIY/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750881784340240-9337-0/minicluster-data/ts-2/data/info.pb | hidden | 127.9.30.67:33829
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.9.30.84:42931 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
--------------+----+---------+---------------+---------+------------+------------------+-------------
post_rebuild | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250625 20:06:01.051507 9337 log_verifier.cc:126] Checking tablet 56087ecb54f9431f9e13ceb2fb3ba969
I20250625 20:06:01.051928 9337 log_verifier.cc:177] Verified matching terms for 0 ops in tablet 56087ecb54f9431f9e13ceb2fb3ba969
I20250625 20:06:01.052088 9337 log_verifier.cc:126] Checking tablet 9db294b5b2bb48a6a7b056de005f61c7
I20250625 20:06:01.901263 9337 log_verifier.cc:177] Verified matching terms for 205 ops in tablet 9db294b5b2bb48a6a7b056de005f61c7
I20250625 20:06:01.926923 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 15717
I20250625 20:06:01.968422 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 15872
I20250625 20:06:02.001293 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 16007
I20250625 20:06:02.033288 9337 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskV0EjIY/build/tsan/bin/kudu with pid 15647
2025-06-25T20:06:02Z chronyd exiting
[ OK ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0 (36198 ms)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest (36198 ms total)
[----------] Global test environment tear-down
[==========] 9 tests from 5 test suites ran. (177678 ms total)
[ PASSED ] 8 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AdminCliTest.TestRebuildTables
1 FAILED TEST
I20250625 20:06:02.096472 9337 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/meta_cache.cc:302: suppressed but not reported on 4 messages since previous log ~48 seconds ago