Note: This is test shard 6 of 8.
[==========] Running 9 tests from 5 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from AdminCliTest
[ RUN ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250627 13:57:04.656422 3765 test_util.cc:276] Using random seed: 227880873
W20250627 13:57:05.876995 3765 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.177s user 0.499s sys 0.675s
W20250627 13:57:05.877342 3765 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.177s user 0.499s sys 0.675s
I20250627 13:57:05.885754 3765 ts_itest-base.cc:115] Starting cluster with:
I20250627 13:57:05.885955 3765 ts_itest-base.cc:116] --------------
I20250627 13:57:05.886142 3765 ts_itest-base.cc:117] 4 tablet servers
I20250627 13:57:05.886309 3765 ts_itest-base.cc:118] 3 replicas per TS
I20250627 13:57:05.886473 3765 ts_itest-base.cc:119] --------------
2025-06-27T13:57:05Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-27T13:57:05Z Disabled control of system clock
I20250627 13:57:05.922482 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.173.126:35991
--webserver_interface=127.3.173.126
--webserver_port=0
--builtin_ntp_servers=127.3.173.84:39377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.173.126:35991 with env {}
W20250627 13:57:06.221058 3779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:57:06.221668 3779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:57:06.222152 3779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:57:06.252596 3779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250627 13:57:06.252920 3779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:57:06.253170 3779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250627 13:57:06.253407 3779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250627 13:57:06.287726 3779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:39377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.173.126:35991
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.173.126:35991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.3.173.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:57:06.289032 3779 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:57:06.290668 3779 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:57:06.306443 3785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:57:06.306496 3786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:57:07.534204 3788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:57:07.536829 3787 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1227 milliseconds
W20250627 13:57:07.538357 3779 thread.cc:640] OpenStack (cloud detector) Time spent creating pthread: real 1.236s user 0.425s sys 0.799s
W20250627 13:57:07.538609 3779 thread.cc:606] OpenStack (cloud detector) Time spent starting thread: real 1.236s user 0.425s sys 0.799s
I20250627 13:57:07.538837 3779 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250627 13:57:07.539885 3779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:57:07.545696 3779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:57:07.547056 3779 hybrid_clock.cc:648] HybridClock initialized: now 1751032627547006 us; error 68 us; skew 500 ppm
I20250627 13:57:07.547854 3779 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:57:07.557986 3779 webserver.cc:469] Webserver started at http://127.3.173.126:37311/ using document root <none> and password file <none>
I20250627 13:57:07.559409 3779 fs_manager.cc:362] Metadata directory not provided
I20250627 13:57:07.559751 3779 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:57:07.560369 3779 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:57:07.567679 3779 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "2f1e690f7f8a4384a7cdb41d26f131df"
format_stamp: "Formatted at 2025-06-27 13:57:07 on dist-test-slave-rn80"
I20250627 13:57:07.569360 3779 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "2f1e690f7f8a4384a7cdb41d26f131df"
format_stamp: "Formatted at 2025-06-27 13:57:07 on dist-test-slave-rn80"
I20250627 13:57:07.579545 3779 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.007s sys 0.001s
I20250627 13:57:07.588110 3795 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:07.589440 3779 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250627 13:57:07.589900 3779 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
uuid: "2f1e690f7f8a4384a7cdb41d26f131df"
format_stamp: "Formatted at 2025-06-27 13:57:07 on dist-test-slave-rn80"
I20250627 13:57:07.590395 3779 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:57:07.668550 3779 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:57:07.669966 3779 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:57:07.670416 3779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:57:07.748562 3779 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.126:35991
I20250627 13:57:07.748632 3846 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.126:35991 every 8 connection(s)
I20250627 13:57:07.751250 3779 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250627 13:57:07.757262 3847 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:57:07.760658 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 3779
I20250627 13:57:07.761185 3765 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250627 13:57:07.778692 3847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df: Bootstrap starting.
I20250627 13:57:07.784139 3847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df: Neither blocks nor log segments found. Creating new log.
I20250627 13:57:07.786078 3847 log.cc:826] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df: Log is configured to *not* fsync() on all Append() calls
I20250627 13:57:07.791594 3847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df: No bootstrap required, opened a new log
I20250627 13:57:07.814671 3847 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1e690f7f8a4384a7cdb41d26f131df" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 35991 } }
I20250627 13:57:07.815311 3847 raft_consensus.cc:383] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:57:07.815510 3847 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f1e690f7f8a4384a7cdb41d26f131df, State: Initialized, Role: FOLLOWER
I20250627 13:57:07.816107 3847 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1e690f7f8a4384a7cdb41d26f131df" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 35991 } }
I20250627 13:57:07.816557 3847 raft_consensus.cc:397] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250627 13:57:07.816779 3847 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250627 13:57:07.817032 3847 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:57:07.821305 3847 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1e690f7f8a4384a7cdb41d26f131df" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 35991 } }
I20250627 13:57:07.821959 3847 leader_election.cc:304] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f1e690f7f8a4384a7cdb41d26f131df; no voters:
I20250627 13:57:07.823985 3847 leader_election.cc:290] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df [CANDIDATE]: Term 1 election: Requested vote from peers
I20250627 13:57:07.824661 3852 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:57:07.826833 3852 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df [term 1 LEADER]: Becoming Leader. State: Replica: 2f1e690f7f8a4384a7cdb41d26f131df, State: Running, Role: LEADER
I20250627 13:57:07.827811 3852 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1e690f7f8a4384a7cdb41d26f131df" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 35991 } }
I20250627 13:57:07.828825 3847 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df [sys.catalog]: configured and running, proceeding with master startup.
I20250627 13:57:07.837878 3854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f1e690f7f8a4384a7cdb41d26f131df. Latest consensus state: current_term: 1 leader_uuid: "2f1e690f7f8a4384a7cdb41d26f131df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1e690f7f8a4384a7cdb41d26f131df" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 35991 } } }
I20250627 13:57:07.837996 3853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f1e690f7f8a4384a7cdb41d26f131df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1e690f7f8a4384a7cdb41d26f131df" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 35991 } } }
I20250627 13:57:07.839035 3854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df [sys.catalog]: This master's current role is: LEADER
I20250627 13:57:07.839217 3853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df [sys.catalog]: This master's current role is: LEADER
I20250627 13:57:07.845419 3860 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250627 13:57:07.859797 3860 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250627 13:57:07.862922 3869 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250627 13:57:07.877352 3860 catalog_manager.cc:1349] Generated new cluster ID: 8134187b821e42c38cf8702c4edf3c3c
I20250627 13:57:07.877604 3860 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250627 13:57:07.913537 3860 catalog_manager.cc:1372] Generated new certificate authority record
I20250627 13:57:07.915560 3860 catalog_manager.cc:1506] Loading token signing keys...
I20250627 13:57:07.941407 3860 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df: Generated new TSK 0
I20250627 13:57:07.942541 3860 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250627 13:57:07.953159 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.173.65:0
--local_ip_for_outbound_sockets=127.3.173.65
--webserver_interface=127.3.173.65
--webserver_port=0
--tserver_master_addrs=127.3.173.126:35991
--builtin_ntp_servers=127.3.173.84:39377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250627 13:57:08.259137 3871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:57:08.259631 3871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:57:08.260134 3871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:57:08.291551 3871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:57:08.292426 3871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.173.65
I20250627 13:57:08.325832 3871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:39377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.173.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.3.173.65
--webserver_port=0
--tserver_master_addrs=127.3.173.126:35991
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.173.65
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:57:08.327101 3871 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:57:08.328657 3871 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:57:08.346998 3878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:57:08.349619 3871 server_base.cc:1048] running on GCE node
W20250627 13:57:08.348333 3880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:57:08.347657 3877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:57:09.520926 3871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:57:09.524358 3871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:57:09.525796 3871 hybrid_clock.cc:648] HybridClock initialized: now 1751032629525736 us; error 68 us; skew 500 ppm
I20250627 13:57:09.526880 3871 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:57:09.538607 3871 webserver.cc:469] Webserver started at http://127.3.173.65:34539/ using document root <none> and password file <none>
I20250627 13:57:09.539814 3871 fs_manager.cc:362] Metadata directory not provided
I20250627 13:57:09.540117 3871 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:57:09.540663 3871 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:57:09.546990 3871 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "589fc4b586ea4f158b5009ac9e67348d"
format_stamp: "Formatted at 2025-06-27 13:57:09 on dist-test-slave-rn80"
I20250627 13:57:09.548548 3871 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "589fc4b586ea4f158b5009ac9e67348d"
format_stamp: "Formatted at 2025-06-27 13:57:09 on dist-test-slave-rn80"
I20250627 13:57:09.558874 3871 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.002s sys 0.009s
I20250627 13:57:09.565501 3887 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:09.566524 3871 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250627 13:57:09.566828 3871 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "589fc4b586ea4f158b5009ac9e67348d"
format_stamp: "Formatted at 2025-06-27 13:57:09 on dist-test-slave-rn80"
I20250627 13:57:09.567134 3871 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:57:09.613705 3871 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:57:09.615115 3871 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:57:09.615528 3871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:57:09.617870 3871 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:57:09.622361 3871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:57:09.622550 3871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:09.622819 3871 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:57:09.622970 3871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:09.777904 3871 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.65:37651
I20250627 13:57:09.777997 3999 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.65:37651 every 8 connection(s)
I20250627 13:57:09.780403 3871 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250627 13:57:09.783735 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 3871
I20250627 13:57:09.784220 3765 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250627 13:57:09.793046 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.173.66:0
--local_ip_for_outbound_sockets=127.3.173.66
--webserver_interface=127.3.173.66
--webserver_port=0
--tserver_master_addrs=127.3.173.126:35991
--builtin_ntp_servers=127.3.173.84:39377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250627 13:57:09.805718 4000 heartbeater.cc:344] Connected to a master server at 127.3.173.126:35991
I20250627 13:57:09.806319 4000 heartbeater.cc:461] Registering TS with master...
I20250627 13:57:09.808275 4000 heartbeater.cc:507] Master 127.3.173.126:35991 requested a full tablet report, sending...
I20250627 13:57:09.811313 3812 ts_manager.cc:194] Registered new tserver with Master: 589fc4b586ea4f158b5009ac9e67348d (127.3.173.65:37651)
I20250627 13:57:09.813146 3812 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.173.65:53315
W20250627 13:57:10.101070 4004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:57:10.101568 4004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:57:10.102082 4004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:57:10.132086 4004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:57:10.132874 4004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.173.66
I20250627 13:57:10.166469 4004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:39377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.173.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.3.173.66
--webserver_port=0
--tserver_master_addrs=127.3.173.126:35991
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.173.66
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:57:10.167692 4004 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:57:10.169209 4004 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:57:10.186129 4011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:57:10.816927 4000 heartbeater.cc:499] Master 127.3.173.126:35991 was elected leader, sending a full tablet report...
W20250627 13:57:10.186730 4010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:57:10.193878 4013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:57:11.357956 4012 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250627 13:57:11.358136 4004 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250627 13:57:11.361722 4004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:57:11.365208 4004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:57:11.366636 4004 hybrid_clock.cc:648] HybridClock initialized: now 1751032631366603 us; error 52 us; skew 500 ppm
I20250627 13:57:11.367420 4004 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:57:11.374017 4004 webserver.cc:469] Webserver started at http://127.3.173.66:35123/ using document root <none> and password file <none>
I20250627 13:57:11.374981 4004 fs_manager.cc:362] Metadata directory not provided
I20250627 13:57:11.375181 4004 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:57:11.375577 4004 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:57:11.379848 4004 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "2b16f516baad43368e6368e633d5fb21"
format_stamp: "Formatted at 2025-06-27 13:57:11 on dist-test-slave-rn80"
I20250627 13:57:11.380867 4004 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "2b16f516baad43368e6368e633d5fb21"
format_stamp: "Formatted at 2025-06-27 13:57:11 on dist-test-slave-rn80"
I20250627 13:57:11.388175 4004 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.001s sys 0.008s
I20250627 13:57:11.394377 4020 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:11.395401 4004 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250627 13:57:11.395730 4004 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "2b16f516baad43368e6368e633d5fb21"
format_stamp: "Formatted at 2025-06-27 13:57:11 on dist-test-slave-rn80"
I20250627 13:57:11.396041 4004 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:57:11.444422 4004 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:57:11.445822 4004 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:57:11.446273 4004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:57:11.449199 4004 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:57:11.453137 4004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:57:11.453323 4004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:11.453590 4004 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:57:11.453732 4004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:11.614845 4004 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.66:43109
I20250627 13:57:11.614943 4132 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.66:43109 every 8 connection(s)
I20250627 13:57:11.617295 4004 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250627 13:57:11.626794 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 4004
I20250627 13:57:11.627319 3765 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250627 13:57:11.634318 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.173.67:0
--local_ip_for_outbound_sockets=127.3.173.67
--webserver_interface=127.3.173.67
--webserver_port=0
--tserver_master_addrs=127.3.173.126:35991
--builtin_ntp_servers=127.3.173.84:39377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250627 13:57:11.641901 4133 heartbeater.cc:344] Connected to a master server at 127.3.173.126:35991
I20250627 13:57:11.642445 4133 heartbeater.cc:461] Registering TS with master...
I20250627 13:57:11.643749 4133 heartbeater.cc:507] Master 127.3.173.126:35991 requested a full tablet report, sending...
I20250627 13:57:11.646253 3812 ts_manager.cc:194] Registered new tserver with Master: 2b16f516baad43368e6368e633d5fb21 (127.3.173.66:43109)
I20250627 13:57:11.647625 3812 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.173.66:46403
W20250627 13:57:11.940258 4137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:57:11.940750 4137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:57:11.941262 4137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:57:11.971907 4137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:57:11.972784 4137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.173.67
I20250627 13:57:12.008590 4137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:39377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.173.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.3.173.67
--webserver_port=0
--tserver_master_addrs=127.3.173.126:35991
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.173.67
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:57:12.009876 4137 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:57:12.011497 4137 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:57:12.028810 4144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:57:12.651314 4133 heartbeater.cc:499] Master 127.3.173.126:35991 was elected leader, sending a full tablet report...
W20250627 13:57:12.029765 4146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:57:12.031395 4137 server_base.cc:1048] running on GCE node
W20250627 13:57:12.028810 4143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:57:13.216171 4137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:57:13.218374 4137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:57:13.219735 4137 hybrid_clock.cc:648] HybridClock initialized: now 1751032633219698 us; error 59 us; skew 500 ppm
I20250627 13:57:13.220532 4137 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:57:13.227164 4137 webserver.cc:469] Webserver started at http://127.3.173.67:46081/ using document root <none> and password file <none>
I20250627 13:57:13.228049 4137 fs_manager.cc:362] Metadata directory not provided
I20250627 13:57:13.228238 4137 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:57:13.228731 4137 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:57:13.233254 4137 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "5aeb2c34139d463ca3f9a33b059c5a36"
format_stamp: "Formatted at 2025-06-27 13:57:13 on dist-test-slave-rn80"
I20250627 13:57:13.234439 4137 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "5aeb2c34139d463ca3f9a33b059c5a36"
format_stamp: "Formatted at 2025-06-27 13:57:13 on dist-test-slave-rn80"
I20250627 13:57:13.241397 4137 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20250627 13:57:13.247291 4153 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:13.248353 4137 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250627 13:57:13.248656 4137 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "5aeb2c34139d463ca3f9a33b059c5a36"
format_stamp: "Formatted at 2025-06-27 13:57:13 on dist-test-slave-rn80"
I20250627 13:57:13.248978 4137 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:57:13.296658 4137 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:57:13.298069 4137 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:57:13.298564 4137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:57:13.304585 4137 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:57:13.312462 4137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:57:13.312777 4137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:13.313095 4137 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:57:13.313334 4137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:13.473819 4137 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.67:40213
I20250627 13:57:13.473963 4265 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.67:40213 every 8 connection(s)
I20250627 13:57:13.476284 4137 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250627 13:57:13.479952 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 4137
I20250627 13:57:13.480422 3765 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250627 13:57:13.487103 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.173.68:0
--local_ip_for_outbound_sockets=127.3.173.68
--webserver_interface=127.3.173.68
--webserver_port=0
--tserver_master_addrs=127.3.173.126:35991
--builtin_ntp_servers=127.3.173.84:39377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250627 13:57:13.500599 4266 heartbeater.cc:344] Connected to a master server at 127.3.173.126:35991
I20250627 13:57:13.501005 4266 heartbeater.cc:461] Registering TS with master...
I20250627 13:57:13.502300 4266 heartbeater.cc:507] Master 127.3.173.126:35991 requested a full tablet report, sending...
I20250627 13:57:13.504559 3812 ts_manager.cc:194] Registered new tserver with Master: 5aeb2c34139d463ca3f9a33b059c5a36 (127.3.173.67:40213)
I20250627 13:57:13.505684 3812 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.173.67:37857
W20250627 13:57:13.796906 4270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:57:13.797425 4270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:57:13.797895 4270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:57:13.829129 4270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:57:13.830061 4270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.173.68
I20250627 13:57:13.863718 4270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:39377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.173.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.3.173.68
--webserver_port=0
--tserver_master_addrs=127.3.173.126:35991
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.173.68
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:57:13.864954 4270 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:57:13.866580 4270 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:57:13.883160 4277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:57:14.509104 4266 heartbeater.cc:499] Master 127.3.173.126:35991 was elected leader, sending a full tablet report...
W20250627 13:57:13.883210 4276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:57:13.885820 4270 server_base.cc:1048] running on GCE node
W20250627 13:57:13.885614 4279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:57:15.010731 4270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:57:15.012882 4270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:57:15.014218 4270 hybrid_clock.cc:648] HybridClock initialized: now 1751032635014177 us; error 57 us; skew 500 ppm
I20250627 13:57:15.014937 4270 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:57:15.025736 4270 webserver.cc:469] Webserver started at http://127.3.173.68:40587/ using document root <none> and password file <none>
I20250627 13:57:15.026641 4270 fs_manager.cc:362] Metadata directory not provided
I20250627 13:57:15.026822 4270 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:57:15.027206 4270 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:57:15.031775 4270 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "03e0969ce0b143fe823bd3486ed9a130"
format_stamp: "Formatted at 2025-06-27 13:57:15 on dist-test-slave-rn80"
I20250627 13:57:15.032763 4270 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "03e0969ce0b143fe823bd3486ed9a130"
format_stamp: "Formatted at 2025-06-27 13:57:15 on dist-test-slave-rn80"
I20250627 13:57:15.039978 4270 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.001s sys 0.007s
I20250627 13:57:15.046286 4287 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:15.047355 4270 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250627 13:57:15.047662 4270 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "03e0969ce0b143fe823bd3486ed9a130"
format_stamp: "Formatted at 2025-06-27 13:57:15 on dist-test-slave-rn80"
I20250627 13:57:15.047989 4270 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:57:15.108471 4270 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:57:15.109911 4270 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:57:15.110337 4270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:57:15.112723 4270 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:57:15.117179 4270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:57:15.117365 4270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:15.117646 4270 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:57:15.117806 4270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:15.272898 4270 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.68:45535
I20250627 13:57:15.273003 4399 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.68:45535 every 8 connection(s)
I20250627 13:57:15.275386 4270 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250627 13:57:15.277274 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 4270
I20250627 13:57:15.278064 3765 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250627 13:57:15.299544 4400 heartbeater.cc:344] Connected to a master server at 127.3.173.126:35991
I20250627 13:57:15.299981 4400 heartbeater.cc:461] Registering TS with master...
I20250627 13:57:15.301095 4400 heartbeater.cc:507] Master 127.3.173.126:35991 requested a full tablet report, sending...
I20250627 13:57:15.303134 3812 ts_manager.cc:194] Registered new tserver with Master: 03e0969ce0b143fe823bd3486ed9a130 (127.3.173.68:45535)
I20250627 13:57:15.304311 3812 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.173.68:55961
I20250627 13:57:15.315694 3765 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250627 13:57:15.358970 3812 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56478:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250627 13:57:15.436254 4068 tablet_service.cc:1468] Processing CreateTablet for tablet f4bf819c041f4231b07d49fda34f476a (DEFAULT_TABLE table=TestTable [id=ed800d9a1b7942a48e3f21ad2776f57a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250627 13:57:15.438180 4335 tablet_service.cc:1468] Processing CreateTablet for tablet f4bf819c041f4231b07d49fda34f476a (DEFAULT_TABLE table=TestTable [id=ed800d9a1b7942a48e3f21ad2776f57a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250627 13:57:15.439885 4335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4bf819c041f4231b07d49fda34f476a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:57:15.440311 4068 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4bf819c041f4231b07d49fda34f476a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:57:15.439222 3935 tablet_service.cc:1468] Processing CreateTablet for tablet f4bf819c041f4231b07d49fda34f476a (DEFAULT_TABLE table=TestTable [id=ed800d9a1b7942a48e3f21ad2776f57a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250627 13:57:15.441239 3935 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4bf819c041f4231b07d49fda34f476a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:57:15.465782 4419 tablet_bootstrap.cc:492] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d: Bootstrap starting.
I20250627 13:57:15.478730 4421 tablet_bootstrap.cc:492] T f4bf819c041f4231b07d49fda34f476a P 03e0969ce0b143fe823bd3486ed9a130: Bootstrap starting.
I20250627 13:57:15.480576 4419 tablet_bootstrap.cc:654] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d: Neither blocks nor log segments found. Creating new log.
I20250627 13:57:15.485271 4419 log.cc:826] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d: Log is configured to *not* fsync() on all Append() calls
I20250627 13:57:15.489468 4421 tablet_bootstrap.cc:654] T f4bf819c041f4231b07d49fda34f476a P 03e0969ce0b143fe823bd3486ed9a130: Neither blocks nor log segments found. Creating new log.
I20250627 13:57:15.492784 4421 log.cc:826] T f4bf819c041f4231b07d49fda34f476a P 03e0969ce0b143fe823bd3486ed9a130: Log is configured to *not* fsync() on all Append() calls
I20250627 13:57:15.506893 4421 tablet_bootstrap.cc:492] T f4bf819c041f4231b07d49fda34f476a P 03e0969ce0b143fe823bd3486ed9a130: No bootstrap required, opened a new log
I20250627 13:57:15.507687 4421 ts_tablet_manager.cc:1397] T f4bf819c041f4231b07d49fda34f476a P 03e0969ce0b143fe823bd3486ed9a130: Time spent bootstrapping tablet: real 0.033s user 0.019s sys 0.007s
I20250627 13:57:15.511998 4420 tablet_bootstrap.cc:492] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21: Bootstrap starting.
I20250627 13:57:15.514683 4419 tablet_bootstrap.cc:492] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d: No bootstrap required, opened a new log
I20250627 13:57:15.515261 4419 ts_tablet_manager.cc:1397] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d: Time spent bootstrapping tablet: real 0.050s user 0.023s sys 0.018s
I20250627 13:57:15.519219 4420 tablet_bootstrap.cc:654] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21: Neither blocks nor log segments found. Creating new log.
I20250627 13:57:15.521468 4420 log.cc:826] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21: Log is configured to *not* fsync() on all Append() calls
I20250627 13:57:15.531473 4420 tablet_bootstrap.cc:492] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21: No bootstrap required, opened a new log
I20250627 13:57:15.531895 4420 ts_tablet_manager.cc:1397] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21: Time spent bootstrapping tablet: real 0.022s user 0.014s sys 0.004s
I20250627 13:57:15.536077 4421 raft_consensus.cc:357] T f4bf819c041f4231b07d49fda34f476a P 03e0969ce0b143fe823bd3486ed9a130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e0969ce0b143fe823bd3486ed9a130" member_type: VOTER last_known_addr { host: "127.3.173.68" port: 45535 } } peers { permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 } } peers { permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37651 } }
I20250627 13:57:15.536996 4421 raft_consensus.cc:383] T f4bf819c041f4231b07d49fda34f476a P 03e0969ce0b143fe823bd3486ed9a130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:57:15.537287 4421 raft_consensus.cc:738] T f4bf819c041f4231b07d49fda34f476a P 03e0969ce0b143fe823bd3486ed9a130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03e0969ce0b143fe823bd3486ed9a130, State: Initialized, Role: FOLLOWER
I20250627 13:57:15.538645 4421 consensus_queue.cc:260] T f4bf819c041f4231b07d49fda34f476a P 03e0969ce0b143fe823bd3486ed9a130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e0969ce0b143fe823bd3486ed9a130" member_type: VOTER last_known_addr { host: "127.3.173.68" port: 45535 } } peers { permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 } } peers { permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37651 } }
I20250627 13:57:15.542387 4400 heartbeater.cc:499] Master 127.3.173.126:35991 was elected leader, sending a full tablet report...
I20250627 13:57:15.543226 4419 raft_consensus.cc:357] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e0969ce0b143fe823bd3486ed9a130" member_type: VOTER last_known_addr { host: "127.3.173.68" port: 45535 } } peers { permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 } } peers { permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37651 } }
I20250627 13:57:15.544170 4419 raft_consensus.cc:383] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:57:15.544544 4421 ts_tablet_manager.cc:1428] T f4bf819c041f4231b07d49fda34f476a P 03e0969ce0b143fe823bd3486ed9a130: Time spent starting tablet: real 0.036s user 0.026s sys 0.008s
I20250627 13:57:15.544502 4419 raft_consensus.cc:738] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 589fc4b586ea4f158b5009ac9e67348d, State: Initialized, Role: FOLLOWER
I20250627 13:57:15.546298 4419 consensus_queue.cc:260] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e0969ce0b143fe823bd3486ed9a130" member_type: VOTER last_known_addr { host: "127.3.173.68" port: 45535 } } peers { permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 } } peers { permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37651 } }
I20250627 13:57:15.551117 4419 ts_tablet_manager.cc:1428] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d: Time spent starting tablet: real 0.036s user 0.022s sys 0.013s
I20250627 13:57:15.558542 4420 raft_consensus.cc:357] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e0969ce0b143fe823bd3486ed9a130" member_type: VOTER last_known_addr { host: "127.3.173.68" port: 45535 } } peers { permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 } } peers { permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37651 } }
I20250627 13:57:15.559283 4420 raft_consensus.cc:383] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:57:15.559528 4420 raft_consensus.cc:738] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b16f516baad43368e6368e633d5fb21, State: Initialized, Role: FOLLOWER
I20250627 13:57:15.560351 4420 consensus_queue.cc:260] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e0969ce0b143fe823bd3486ed9a130" member_type: VOTER last_known_addr { host: "127.3.173.68" port: 45535 } } peers { permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 } } peers { permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37651 } }
I20250627 13:57:15.564379 4420 ts_tablet_manager.cc:1428] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21: Time spent starting tablet: real 0.032s user 0.025s sys 0.006s
W20250627 13:57:15.626948 4134 tablet.cc:2378] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250627 13:57:15.648727 4425 raft_consensus.cc:491] T f4bf819c041f4231b07d49fda34f476a P 03e0969ce0b143fe823bd3486ed9a130 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:57:15.649680 4425 raft_consensus.cc:513] T f4bf819c041f4231b07d49fda34f476a P 03e0969ce0b143fe823bd3486ed9a130 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e0969ce0b143fe823bd3486ed9a130" member_type: VOTER last_known_addr { host: "127.3.173.68" port: 45535 } } peers { permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 } } peers { permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37651 } }
I20250627 13:57:15.652086 4425 leader_election.cc:290] T f4bf819c041f4231b07d49fda34f476a P 03e0969ce0b143fe823bd3486ed9a130 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b16f516baad43368e6368e633d5fb21 (127.3.173.66:43109), 589fc4b586ea4f158b5009ac9e67348d (127.3.173.65:37651)
I20250627 13:57:15.662433 4088 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4bf819c041f4231b07d49fda34f476a" candidate_uuid: "03e0969ce0b143fe823bd3486ed9a130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b16f516baad43368e6368e633d5fb21" is_pre_election: true
I20250627 13:57:15.662560 3955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4bf819c041f4231b07d49fda34f476a" candidate_uuid: "03e0969ce0b143fe823bd3486ed9a130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "589fc4b586ea4f158b5009ac9e67348d" is_pre_election: true
I20250627 13:57:15.663139 4088 raft_consensus.cc:2466] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03e0969ce0b143fe823bd3486ed9a130 in term 0.
I20250627 13:57:15.663421 3955 raft_consensus.cc:2466] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03e0969ce0b143fe823bd3486ed9a130 in term 0.
I20250627 13:57:15.664499 4291 leader_election.cc:304] T f4bf819c041f4231b07d49fda34f476a P 03e0969ce0b143fe823bd3486ed9a130 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03e0969ce0b143fe823bd3486ed9a130, 2b16f516baad43368e6368e633d5fb21; no voters:
I20250627 13:57:15.665649 4425 raft_consensus.cc:2802] T f4bf819c041f4231b07d49fda34f476a P 03e0969ce0b143fe823bd3486ed9a130 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:57:15.666121 4425 raft_consensus.cc:491] T f4bf819c041f4231b07d49fda34f476a P 03e0969ce0b143fe823bd3486ed9a130 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:57:15.666539 4425 raft_consensus.cc:3058] T f4bf819c041f4231b07d49fda34f476a P 03e0969ce0b143fe823bd3486ed9a130 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:57:15.706473 4425 raft_consensus.cc:513] T f4bf819c041f4231b07d49fda34f476a P 03e0969ce0b143fe823bd3486ed9a130 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e0969ce0b143fe823bd3486ed9a130" member_type: VOTER last_known_addr { host: "127.3.173.68" port: 45535 } } peers { permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 } } peers { permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37651 } }
I20250627 13:57:15.708891 4425 leader_election.cc:290] T f4bf819c041f4231b07d49fda34f476a P 03e0969ce0b143fe823bd3486ed9a130 [CANDIDATE]: Term 1 election: Requested vote from peers 2b16f516baad43368e6368e633d5fb21 (127.3.173.66:43109), 589fc4b586ea4f158b5009ac9e67348d (127.3.173.65:37651)
I20250627 13:57:15.709635 4088 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4bf819c041f4231b07d49fda34f476a" candidate_uuid: "03e0969ce0b143fe823bd3486ed9a130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b16f516baad43368e6368e633d5fb21"
I20250627 13:57:15.710006 3955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4bf819c041f4231b07d49fda34f476a" candidate_uuid: "03e0969ce0b143fe823bd3486ed9a130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "589fc4b586ea4f158b5009ac9e67348d"
I20250627 13:57:15.710160 4088 raft_consensus.cc:3058] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:57:15.710485 3955 raft_consensus.cc:3058] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [term 0 FOLLOWER]: Advancing to term 1
W20250627 13:57:15.781231 4401 tablet.cc:2378] T f4bf819c041f4231b07d49fda34f476a P 03e0969ce0b143fe823bd3486ed9a130: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250627 13:57:15.798040 4001 tablet.cc:2378] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250627 13:57:15.806974 4088 raft_consensus.cc:2466] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03e0969ce0b143fe823bd3486ed9a130 in term 1.
I20250627 13:57:15.806974 3955 raft_consensus.cc:2466] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03e0969ce0b143fe823bd3486ed9a130 in term 1.
I20250627 13:57:15.808199 4291 leader_election.cc:304] T f4bf819c041f4231b07d49fda34f476a P 03e0969ce0b143fe823bd3486ed9a130 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03e0969ce0b143fe823bd3486ed9a130, 2b16f516baad43368e6368e633d5fb21; no voters:
I20250627 13:57:15.808849 4425 raft_consensus.cc:2802] T f4bf819c041f4231b07d49fda34f476a P 03e0969ce0b143fe823bd3486ed9a130 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:57:15.810447 4425 raft_consensus.cc:695] T f4bf819c041f4231b07d49fda34f476a P 03e0969ce0b143fe823bd3486ed9a130 [term 1 LEADER]: Becoming Leader. State: Replica: 03e0969ce0b143fe823bd3486ed9a130, State: Running, Role: LEADER
I20250627 13:57:15.811300 4425 consensus_queue.cc:237] T f4bf819c041f4231b07d49fda34f476a P 03e0969ce0b143fe823bd3486ed9a130 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e0969ce0b143fe823bd3486ed9a130" member_type: VOTER last_known_addr { host: "127.3.173.68" port: 45535 } } peers { permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 } } peers { permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37651 } }
I20250627 13:57:15.821277 3812 catalog_manager.cc:5582] T f4bf819c041f4231b07d49fda34f476a P 03e0969ce0b143fe823bd3486ed9a130 reported cstate change: term changed from 0 to 1, leader changed from <none> to 03e0969ce0b143fe823bd3486ed9a130 (127.3.173.68). New cstate: current_term: 1 leader_uuid: "03e0969ce0b143fe823bd3486ed9a130" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e0969ce0b143fe823bd3486ed9a130" member_type: VOTER last_known_addr { host: "127.3.173.68" port: 45535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37651 } health_report { overall_health: UNKNOWN } } }
I20250627 13:57:15.920821 3765 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250627 13:57:15.924541 3765 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 589fc4b586ea4f158b5009ac9e67348d to finish bootstrapping
I20250627 13:57:15.938637 3765 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 2b16f516baad43368e6368e633d5fb21 to finish bootstrapping
I20250627 13:57:15.949950 3765 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 03e0969ce0b143fe823bd3486ed9a130 to finish bootstrapping
I20250627 13:57:15.962437 3765 kudu-admin-test.cc:709] Waiting for Master to see the current replicas...
I20250627 13:57:15.965691 3765 kudu-admin-test.cc:716] Tablet locations:
tablet_locations {
tablet_id: "f4bf819c041f4231b07d49fda34f476a"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: LEADER
}
interned_replicas {
ts_info_idx: 1
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "03e0969ce0b143fe823bd3486ed9a130"
rpc_addresses {
host: "127.3.173.68"
port: 45535
}
}
ts_infos {
permanent_uuid: "2b16f516baad43368e6368e633d5fb21"
rpc_addresses {
host: "127.3.173.66"
port: 43109
}
}
ts_infos {
permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d"
rpc_addresses {
host: "127.3.173.65"
port: 37651
}
}
I20250627 13:57:16.325845 4435 consensus_queue.cc:1035] T f4bf819c041f4231b07d49fda34f476a P 03e0969ce0b143fe823bd3486ed9a130 [LEADER]: Connected to new peer: Peer: permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:57:16.342466 4435 consensus_queue.cc:1035] T f4bf819c041f4231b07d49fda34f476a P 03e0969ce0b143fe823bd3486ed9a130 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:57:16.348596 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 4270
W20250627 13:57:16.379631 4023 connection.cc:537] server connection from 127.3.173.68:35073 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250627 13:57:16.380335 3799 connection.cc:537] server connection from 127.3.173.68:55961 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250627 13:57:16.381335 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 3779
I20250627 13:57:16.411541 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.173.126:35991
--webserver_interface=127.3.173.126
--webserver_port=37311
--builtin_ntp_servers=127.3.173.84:39377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.173.126:35991 with env {}
W20250627 13:57:16.520390 4266 heartbeater.cc:646] Failed to heartbeat to 127.3.173.126:35991 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.173.126:35991: connect: Connection refused (error 111)
W20250627 13:57:16.733857 4446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:57:16.734442 4446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:57:16.734853 4446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:57:16.767418 4446 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250627 13:57:16.767711 4446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:57:16.767913 4446 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250627 13:57:16.768103 4446 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250627 13:57:16.805966 4446 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:39377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.173.126:35991
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.173.126:35991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.3.173.126
--webserver_port=37311
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:57:16.807274 4446 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:57:16.808928 4446 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:57:16.824898 4454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:57:17.356426 4000 heartbeater.cc:646] Failed to heartbeat to 127.3.173.126:35991 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.173.126:35991: connect: Connection refused (error 111)
W20250627 13:57:17.376828 4133 heartbeater.cc:646] Failed to heartbeat to 127.3.173.126:35991 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.173.126:35991: connect: Connection refused (error 111)
I20250627 13:57:17.839712 4462 raft_consensus.cc:491] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03e0969ce0b143fe823bd3486ed9a130)
I20250627 13:57:17.840113 4462 raft_consensus.cc:513] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e0969ce0b143fe823bd3486ed9a130" member_type: VOTER last_known_addr { host: "127.3.173.68" port: 45535 } } peers { permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 } } peers { permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37651 } }
I20250627 13:57:17.842339 4462 leader_election.cc:290] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03e0969ce0b143fe823bd3486ed9a130 (127.3.173.68:45535), 2b16f516baad43368e6368e633d5fb21 (127.3.173.66:43109)
W20250627 13:57:17.845549 3888 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.173.68:45535: connect: Connection refused (error 111)
W20250627 13:57:17.857761 3888 leader_election.cc:336] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 03e0969ce0b143fe823bd3486ed9a130 (127.3.173.68:45535): Network error: Client connection negotiation failed: client connection to 127.3.173.68:45535: connect: Connection refused (error 111)
I20250627 13:57:17.859333 4466 raft_consensus.cc:491] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03e0969ce0b143fe823bd3486ed9a130)
I20250627 13:57:17.859732 4466 raft_consensus.cc:513] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e0969ce0b143fe823bd3486ed9a130" member_type: VOTER last_known_addr { host: "127.3.173.68" port: 45535 } } peers { permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 } } peers { permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37651 } }
I20250627 13:57:17.862351 4466 leader_election.cc:290] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03e0969ce0b143fe823bd3486ed9a130 (127.3.173.68:45535), 589fc4b586ea4f158b5009ac9e67348d (127.3.173.65:37651)
I20250627 13:57:17.863839 4088 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4bf819c041f4231b07d49fda34f476a" candidate_uuid: "589fc4b586ea4f158b5009ac9e67348d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2b16f516baad43368e6368e633d5fb21" is_pre_election: true
I20250627 13:57:17.864451 4088 raft_consensus.cc:2466] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 589fc4b586ea4f158b5009ac9e67348d in term 1.
W20250627 13:57:17.865494 4021 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.173.68:45535: connect: Connection refused (error 111)
I20250627 13:57:17.868753 3891 leader_election.cc:304] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2b16f516baad43368e6368e633d5fb21, 589fc4b586ea4f158b5009ac9e67348d; no voters: 03e0969ce0b143fe823bd3486ed9a130
I20250627 13:57:17.870158 4462 raft_consensus.cc:2802] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250627 13:57:17.870591 4462 raft_consensus.cc:491] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 03e0969ce0b143fe823bd3486ed9a130)
I20250627 13:57:17.871017 4462 raft_consensus.cc:3058] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [term 1 FOLLOWER]: Advancing to term 2
W20250627 13:57:17.871536 4021 leader_election.cc:336] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 03e0969ce0b143fe823bd3486ed9a130 (127.3.173.68:45535): Network error: Client connection negotiation failed: client connection to 127.3.173.68:45535: connect: Connection refused (error 111)
I20250627 13:57:17.878476 3955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4bf819c041f4231b07d49fda34f476a" candidate_uuid: "2b16f516baad43368e6368e633d5fb21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "589fc4b586ea4f158b5009ac9e67348d" is_pre_election: true
I20250627 13:57:17.886827 4462 raft_consensus.cc:513] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e0969ce0b143fe823bd3486ed9a130" member_type: VOTER last_known_addr { host: "127.3.173.68" port: 45535 } } peers { permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 } } peers { permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37651 } }
I20250627 13:57:17.888379 3955 raft_consensus.cc:2391] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2b16f516baad43368e6368e633d5fb21 in current term 2: Already voted for candidate 589fc4b586ea4f158b5009ac9e67348d in this term.
I20250627 13:57:17.891291 4462 leader_election.cc:290] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [CANDIDATE]: Term 2 election: Requested vote from peers 03e0969ce0b143fe823bd3486ed9a130 (127.3.173.68:45535), 2b16f516baad43368e6368e633d5fb21 (127.3.173.66:43109)
I20250627 13:57:17.892289 4023 leader_election.cc:304] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b16f516baad43368e6368e633d5fb21; no voters: 03e0969ce0b143fe823bd3486ed9a130, 589fc4b586ea4f158b5009ac9e67348d
I20250627 13:57:17.892910 4088 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4bf819c041f4231b07d49fda34f476a" candidate_uuid: "589fc4b586ea4f158b5009ac9e67348d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2b16f516baad43368e6368e633d5fb21"
I20250627 13:57:17.893446 4088 raft_consensus.cc:3058] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21 [term 1 FOLLOWER]: Advancing to term 2
W20250627 13:57:17.893858 3888 leader_election.cc:336] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 03e0969ce0b143fe823bd3486ed9a130 (127.3.173.68:45535): Network error: Client connection negotiation failed: client connection to 127.3.173.68:45535: connect: Connection refused (error 111)
I20250627 13:57:17.985265 4088 raft_consensus.cc:2466] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 589fc4b586ea4f158b5009ac9e67348d in term 2.
I20250627 13:57:17.985750 4466 raft_consensus.cc:2747] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:57:17.986677 3891 leader_election.cc:304] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2b16f516baad43368e6368e633d5fb21, 589fc4b586ea4f158b5009ac9e67348d; no voters: 03e0969ce0b143fe823bd3486ed9a130
I20250627 13:57:17.987819 4462 raft_consensus.cc:2802] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [term 2 FOLLOWER]: Leader election won for term 2
I20250627 13:57:17.989956 4462 raft_consensus.cc:695] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [term 2 LEADER]: Becoming Leader. State: Replica: 589fc4b586ea4f158b5009ac9e67348d, State: Running, Role: LEADER
I20250627 13:57:17.990972 4462 consensus_queue.cc:237] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e0969ce0b143fe823bd3486ed9a130" member_type: VOTER last_known_addr { host: "127.3.173.68" port: 45535 } } peers { permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 } } peers { permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37651 } }
W20250627 13:57:16.826397 4453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:57:16.827579 4446 server_base.cc:1048] running on GCE node
W20250627 13:57:16.827739 4456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:57:18.065639 4446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:57:18.068372 4446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:57:18.069729 4446 hybrid_clock.cc:648] HybridClock initialized: now 1751032638069671 us; error 70 us; skew 500 ppm
I20250627 13:57:18.070573 4446 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:57:18.077003 4446 webserver.cc:469] Webserver started at http://127.3.173.126:37311/ using document root <none> and password file <none>
I20250627 13:57:18.077966 4446 fs_manager.cc:362] Metadata directory not provided
I20250627 13:57:18.078240 4446 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:57:18.086421 4446 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.000s sys 0.007s
I20250627 13:57:18.091271 4476 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:18.092410 4446 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250627 13:57:18.092737 4446 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
uuid: "2f1e690f7f8a4384a7cdb41d26f131df"
format_stamp: "Formatted at 2025-06-27 13:57:07 on dist-test-slave-rn80"
I20250627 13:57:18.095064 4446 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:57:18.151484 4446 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:57:18.152926 4446 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:57:18.153357 4446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:57:18.227774 4446 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.126:35991
I20250627 13:57:18.227818 4527 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.126:35991 every 8 connection(s)
I20250627 13:57:18.231423 4446 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250627 13:57:18.238739 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 4446
I20250627 13:57:18.239355 3765 kudu-admin-test.cc:735] Forcing unsafe config change on tserver 589fc4b586ea4f158b5009ac9e67348d
I20250627 13:57:18.245357 4528 sys_catalog.cc:263] Verifying existing consensus state
I20250627 13:57:18.251521 4528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df: Bootstrap starting.
I20250627 13:57:18.303771 4528 log.cc:826] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df: Log is configured to *not* fsync() on all Append() calls
I20250627 13:57:18.325469 4528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:57:18.326318 4528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df: Bootstrap complete.
I20250627 13:57:18.347757 4528 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1e690f7f8a4384a7cdb41d26f131df" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 35991 } }
I20250627 13:57:18.349730 4528 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f1e690f7f8a4384a7cdb41d26f131df, State: Initialized, Role: FOLLOWER
I20250627 13:57:18.350603 4528 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1e690f7f8a4384a7cdb41d26f131df" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 35991 } }
I20250627 13:57:18.351068 4528 raft_consensus.cc:397] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250627 13:57:18.351339 4528 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250627 13:57:18.351630 4528 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:57:18.388757 4528 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1e690f7f8a4384a7cdb41d26f131df" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 35991 } }
I20250627 13:57:18.389087 4088 raft_consensus.cc:1273] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21 [term 2 FOLLOWER]: Refusing update from remote peer 589fc4b586ea4f158b5009ac9e67348d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250627 13:57:18.389721 4528 leader_election.cc:304] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f1e690f7f8a4384a7cdb41d26f131df; no voters:
I20250627 13:57:18.390966 4462 consensus_queue.cc:1035] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250627 13:57:18.400691 4133 heartbeater.cc:344] Connected to a master server at 127.3.173.126:35991
I20250627 13:57:18.400779 4528 leader_election.cc:290] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df [CANDIDATE]: Term 2 election: Requested vote from peers
I20250627 13:57:18.401228 4535 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df [term 2 FOLLOWER]: Leader election won for term 2
I20250627 13:57:18.410205 4535 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df [term 2 LEADER]: Becoming Leader. State: Replica: 2f1e690f7f8a4384a7cdb41d26f131df, State: Running, Role: LEADER
I20250627 13:57:18.411069 4535 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1e690f7f8a4384a7cdb41d26f131df" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 35991 } }
I20250627 13:57:18.414695 4528 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df [sys.catalog]: configured and running, proceeding with master startup.
I20250627 13:57:18.456748 4537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2f1e690f7f8a4384a7cdb41d26f131df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1e690f7f8a4384a7cdb41d26f131df" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 35991 } } }
I20250627 13:57:18.458518 4537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df [sys.catalog]: This master's current role is: LEADER
I20250627 13:57:18.457855 4538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f1e690f7f8a4384a7cdb41d26f131df. Latest consensus state: current_term: 2 leader_uuid: "2f1e690f7f8a4384a7cdb41d26f131df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1e690f7f8a4384a7cdb41d26f131df" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 35991 } } }
I20250627 13:57:18.459766 4538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df [sys.catalog]: This master's current role is: LEADER
I20250627 13:57:18.467594 4550 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250627 13:57:18.471805 4000 heartbeater.cc:344] Connected to a master server at 127.3.173.126:35991
I20250627 13:57:18.489969 4550 catalog_manager.cc:671] Loaded metadata for table TestTable [id=ed800d9a1b7942a48e3f21ad2776f57a]
W20250627 13:57:18.490233 3888 consensus_peers.cc:487] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d -> Peer 03e0969ce0b143fe823bd3486ed9a130 (127.3.173.68:45535): Couldn't send request to peer 03e0969ce0b143fe823bd3486ed9a130. Status: Network error: Client connection negotiation failed: client connection to 127.3.173.68:45535: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250627 13:57:18.502292 4550 tablet_loader.cc:96] loaded metadata for tablet f4bf819c041f4231b07d49fda34f476a (table TestTable [id=ed800d9a1b7942a48e3f21ad2776f57a])
I20250627 13:57:18.503963 4550 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250627 13:57:18.508711 4550 catalog_manager.cc:1261] Loaded cluster ID: 8134187b821e42c38cf8702c4edf3c3c
I20250627 13:57:18.509061 4550 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250627 13:57:18.516635 4550 catalog_manager.cc:1506] Loading token signing keys...
I20250627 13:57:18.521548 4550 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 2f1e690f7f8a4384a7cdb41d26f131df: Loaded TSK: 0
I20250627 13:57:18.523263 4550 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250627 13:57:18.553845 4266 heartbeater.cc:344] Connected to a master server at 127.3.173.126:35991
I20250627 13:57:18.559689 4493 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5aeb2c34139d463ca3f9a33b059c5a36" instance_seqno: 1751032633436827) as {username='slave'} at 127.3.173.67:49833; Asking this server to re-register.
I20250627 13:57:18.561085 4266 heartbeater.cc:461] Registering TS with master...
I20250627 13:57:18.561697 4266 heartbeater.cc:507] Master 127.3.173.126:35991 requested a full tablet report, sending...
I20250627 13:57:18.564204 4493 ts_manager.cc:194] Registered new tserver with Master: 5aeb2c34139d463ca3f9a33b059c5a36 (127.3.173.67:40213)
W20250627 13:57:18.653782 4531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:57:18.654389 4531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:57:18.686935 4531 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250627 13:57:19.451802 4493 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "2b16f516baad43368e6368e633d5fb21" instance_seqno: 1751032631580278) as {username='slave'} at 127.3.173.66:47811; Asking this server to re-register.
I20250627 13:57:19.453506 4133 heartbeater.cc:461] Registering TS with master...
I20250627 13:57:19.454141 4133 heartbeater.cc:507] Master 127.3.173.126:35991 requested a full tablet report, sending...
I20250627 13:57:19.456907 4493 ts_manager.cc:194] Registered new tserver with Master: 2b16f516baad43368e6368e633d5fb21 (127.3.173.66:43109)
I20250627 13:57:19.460750 4493 catalog_manager.cc:5582] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21 reported cstate change: term changed from 1 to 2, leader changed from 03e0969ce0b143fe823bd3486ed9a130 (127.3.173.68) to 589fc4b586ea4f158b5009ac9e67348d (127.3.173.65). New cstate: current_term: 2 leader_uuid: "589fc4b586ea4f158b5009ac9e67348d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e0969ce0b143fe823bd3486ed9a130" member_type: VOTER last_known_addr { host: "127.3.173.68" port: 45535 } } peers { permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 } } peers { permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37651 } } }
I20250627 13:57:19.487766 4492 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d" instance_seqno: 1751032629738806) as {username='slave'} at 127.3.173.65:33243; Asking this server to re-register.
I20250627 13:57:19.489864 4000 heartbeater.cc:461] Registering TS with master...
I20250627 13:57:19.490835 4000 heartbeater.cc:507] Master 127.3.173.126:35991 requested a full tablet report, sending...
I20250627 13:57:19.496061 4492 ts_manager.cc:194] Registered new tserver with Master: 589fc4b586ea4f158b5009ac9e67348d (127.3.173.65:37651)
W20250627 13:57:20.134481 4563 debug-util.cc:398] Leaking SignalData structure 0x7b0800036040 after lost signal to thread 4531
W20250627 13:57:20.135030 4563 kernel_stack_watchdog.cc:198] Thread 4531 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:641 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250627 13:57:20.163861 4531 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.433s user 0.559s sys 0.869s
W20250627 13:57:20.280624 4531 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.550s user 0.559s sys 0.882s
I20250627 13:57:20.326403 3955 tablet_service.cc:1905] Received UnsafeChangeConfig RPC: dest_uuid: "589fc4b586ea4f158b5009ac9e67348d"
tablet_id: "f4bf819c041f4231b07d49fda34f476a"
caller_id: "kudu-tools"
new_config {
peers {
permanent_uuid: "2b16f516baad43368e6368e633d5fb21"
}
peers {
permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d"
}
}
from {username='slave'} at 127.0.0.1:52458
W20250627 13:57:20.327849 3955 raft_consensus.cc:2216] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [term 2 LEADER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e0969ce0b143fe823bd3486ed9a130" member_type: VOTER last_known_addr { host: "127.3.173.68" port: 45535 } } peers { permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 } } peers { permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37651 } }NEW CONFIG: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 } } peers { permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37651 } } unsafe_config_change: true
I20250627 13:57:20.329074 3955 raft_consensus.cc:3053] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [term 2 LEADER]: Stepping down as leader of term 2
I20250627 13:57:20.329357 3955 raft_consensus.cc:738] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 589fc4b586ea4f158b5009ac9e67348d, State: Running, Role: LEADER
I20250627 13:57:20.330122 3955 consensus_queue.cc:260] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e0969ce0b143fe823bd3486ed9a130" member_type: VOTER last_known_addr { host: "127.3.173.68" port: 45535 } } peers { permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 } } peers { permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37651 } }
I20250627 13:57:20.331350 3955 raft_consensus.cc:3058] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [term 2 FOLLOWER]: Advancing to term 3
I20250627 13:57:21.756781 4586 raft_consensus.cc:491] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 589fc4b586ea4f158b5009ac9e67348d)
I20250627 13:57:21.757413 4586 raft_consensus.cc:513] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e0969ce0b143fe823bd3486ed9a130" member_type: VOTER last_known_addr { host: "127.3.173.68" port: 45535 } } peers { permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 } } peers { permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37651 } }
I20250627 13:57:21.759145 4586 leader_election.cc:290] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 03e0969ce0b143fe823bd3486ed9a130 (127.3.173.68:45535), 589fc4b586ea4f158b5009ac9e67348d (127.3.173.65:37651)
I20250627 13:57:21.760407 3955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4bf819c041f4231b07d49fda34f476a" candidate_uuid: "2b16f516baad43368e6368e633d5fb21" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "589fc4b586ea4f158b5009ac9e67348d" is_pre_election: true
W20250627 13:57:21.763690 4021 leader_election.cc:336] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 03e0969ce0b143fe823bd3486ed9a130 (127.3.173.68:45535): Network error: Client connection negotiation failed: client connection to 127.3.173.68:45535: connect: Connection refused (error 111)
I20250627 13:57:21.763973 4021 leader_election.cc:304] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b16f516baad43368e6368e633d5fb21; no voters: 03e0969ce0b143fe823bd3486ed9a130, 589fc4b586ea4f158b5009ac9e67348d
I20250627 13:57:21.764552 4586 raft_consensus.cc:2747] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250627 13:57:21.840016 4589 raft_consensus.cc:491] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools)
I20250627 13:57:21.840420 4589 raft_consensus.cc:513] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 } } peers { permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37651 } } unsafe_config_change: true
I20250627 13:57:21.841490 4589 leader_election.cc:290] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2b16f516baad43368e6368e633d5fb21 (127.3.173.66:43109)
I20250627 13:57:21.842590 4088 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4bf819c041f4231b07d49fda34f476a" candidate_uuid: "589fc4b586ea4f158b5009ac9e67348d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2b16f516baad43368e6368e633d5fb21" is_pre_election: true
I20250627 13:57:21.843130 4088 raft_consensus.cc:2466] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 589fc4b586ea4f158b5009ac9e67348d in term 2.
I20250627 13:57:21.844174 3891 leader_election.cc:304] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2b16f516baad43368e6368e633d5fb21, 589fc4b586ea4f158b5009ac9e67348d; no voters:
I20250627 13:57:21.844753 4589 raft_consensus.cc:2802] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250627 13:57:21.845009 4589 raft_consensus.cc:491] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [term 3 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools)
I20250627 13:57:21.845271 4589 raft_consensus.cc:3058] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [term 3 FOLLOWER]: Advancing to term 4
I20250627 13:57:21.850145 4589 raft_consensus.cc:513] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 } } peers { permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37651 } } unsafe_config_change: true
I20250627 13:57:21.851161 4589 leader_election.cc:290] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [CANDIDATE]: Term 4 election: Requested vote from peers 2b16f516baad43368e6368e633d5fb21 (127.3.173.66:43109)
I20250627 13:57:21.852058 4088 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4bf819c041f4231b07d49fda34f476a" candidate_uuid: "589fc4b586ea4f158b5009ac9e67348d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2b16f516baad43368e6368e633d5fb21"
I20250627 13:57:21.852432 4088 raft_consensus.cc:3058] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21 [term 2 FOLLOWER]: Advancing to term 4
I20250627 13:57:21.856877 4088 raft_consensus.cc:2466] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 589fc4b586ea4f158b5009ac9e67348d in term 4.
I20250627 13:57:21.857759 3891 leader_election.cc:304] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2b16f516baad43368e6368e633d5fb21, 589fc4b586ea4f158b5009ac9e67348d; no voters:
I20250627 13:57:21.858402 4589 raft_consensus.cc:2802] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [term 4 FOLLOWER]: Leader election won for term 4
I20250627 13:57:21.859124 4589 raft_consensus.cc:695] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [term 4 LEADER]: Becoming Leader. State: Replica: 589fc4b586ea4f158b5009ac9e67348d, State: Running, Role: LEADER
I20250627 13:57:21.859699 4589 consensus_queue.cc:237] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 } } peers { permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37651 } } unsafe_config_change: true
I20250627 13:57:21.865067 4492 catalog_manager.cc:5582] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d reported cstate change: term changed from 2 to 4, now has a pending config: VOTER 2b16f516baad43368e6368e633d5fb21 (127.3.173.66), VOTER 589fc4b586ea4f158b5009ac9e67348d (127.3.173.65). New cstate: current_term: 4 leader_uuid: "589fc4b586ea4f158b5009ac9e67348d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e0969ce0b143fe823bd3486ed9a130" member_type: VOTER last_known_addr { host: "127.3.173.68" port: 45535 } } peers { permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37651 } health_report { overall_health: HEALTHY } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 } } peers { permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37651 } } unsafe_config_change: true }
I20250627 13:57:22.247918 4088 raft_consensus.cc:1273] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21 [term 4 FOLLOWER]: Refusing update from remote peer 589fc4b586ea4f158b5009ac9e67348d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250627 13:57:22.249228 4589 consensus_queue.cc:1035] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250627 13:57:22.257220 4589 raft_consensus.cc:2953] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [term 4 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER 03e0969ce0b143fe823bd3486ed9a130 (127.3.173.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 } } peers { permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37651 } } unsafe_config_change: true }
I20250627 13:57:22.258687 4088 raft_consensus.cc:2953] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21 [term 4 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER 03e0969ce0b143fe823bd3486ed9a130 (127.3.173.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 } } peers { permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37651 } } unsafe_config_change: true }
I20250627 13:57:22.269762 4492 catalog_manager.cc:5582] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d reported cstate change: config changed from index -1 to 3, VOTER 03e0969ce0b143fe823bd3486ed9a130 (127.3.173.68) evicted, no longer has a pending config: VOTER 2b16f516baad43368e6368e633d5fb21 (127.3.173.66), VOTER 589fc4b586ea4f158b5009ac9e67348d (127.3.173.65). New cstate: current_term: 4 leader_uuid: "589fc4b586ea4f158b5009ac9e67348d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37651 } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
W20250627 13:57:22.281193 4492 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet f4bf819c041f4231b07d49fda34f476a on TS 03e0969ce0b143fe823bd3486ed9a130: Not found: failed to reset TS proxy: Could not find TS for UUID 03e0969ce0b143fe823bd3486ed9a130
I20250627 13:57:22.299365 3955 consensus_queue.cc:237] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 } } peers { permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37651 } } peers { permanent_uuid: "5aeb2c34139d463ca3f9a33b059c5a36" member_type: NON_VOTER last_known_addr { host: "127.3.173.67" port: 40213 } attrs { promote: true } } unsafe_config_change: true
I20250627 13:57:22.303680 4088 raft_consensus.cc:1273] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21 [term 4 FOLLOWER]: Refusing update from remote peer 589fc4b586ea4f158b5009ac9e67348d: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250627 13:57:22.305435 4590 consensus_queue.cc:1035] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250627 13:57:22.311220 4589 raft_consensus.cc:2953] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER 5aeb2c34139d463ca3f9a33b059c5a36 (127.3.173.67) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 } } peers { permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37651 } } peers { permanent_uuid: "5aeb2c34139d463ca3f9a33b059c5a36" member_type: NON_VOTER last_known_addr { host: "127.3.173.67" port: 40213 } attrs { promote: true } } unsafe_config_change: true }
I20250627 13:57:22.312850 4088 raft_consensus.cc:2953] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER 5aeb2c34139d463ca3f9a33b059c5a36 (127.3.173.67) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 } } peers { permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37651 } } peers { permanent_uuid: "5aeb2c34139d463ca3f9a33b059c5a36" member_type: NON_VOTER last_known_addr { host: "127.3.173.67" port: 40213 } attrs { promote: true } } unsafe_config_change: true }
W20250627 13:57:22.317134 3888 consensus_peers.cc:487] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d -> Peer 5aeb2c34139d463ca3f9a33b059c5a36 (127.3.173.67:40213): Couldn't send request to peer 5aeb2c34139d463ca3f9a33b059c5a36. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4bf819c041f4231b07d49fda34f476a. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:57:22.320377 4479 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f4bf819c041f4231b07d49fda34f476a with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250627 13:57:22.322393 4493 catalog_manager.cc:5582] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21 reported cstate change: config changed from index 3 to 5, NON_VOTER 5aeb2c34139d463ca3f9a33b059c5a36 (127.3.173.67) added. New cstate: current_term: 4 leader_uuid: "589fc4b586ea4f158b5009ac9e67348d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 } } peers { permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37651 } } peers { permanent_uuid: "5aeb2c34139d463ca3f9a33b059c5a36" member_type: NON_VOTER last_known_addr { host: "127.3.173.67" port: 40213 } attrs { promote: true } } unsafe_config_change: true }
W20250627 13:57:22.333681 4478 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet f4bf819c041f4231b07d49fda34f476a on TS 03e0969ce0b143fe823bd3486ed9a130 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 03e0969ce0b143fe823bd3486ed9a130
I20250627 13:57:22.763034 4604 ts_tablet_manager.cc:927] T f4bf819c041f4231b07d49fda34f476a P 5aeb2c34139d463ca3f9a33b059c5a36: Initiating tablet copy from peer 589fc4b586ea4f158b5009ac9e67348d (127.3.173.65:37651)
I20250627 13:57:22.765460 4604 tablet_copy_client.cc:323] T f4bf819c041f4231b07d49fda34f476a P 5aeb2c34139d463ca3f9a33b059c5a36: tablet copy: Beginning tablet copy session from remote peer at address 127.3.173.65:37651
I20250627 13:57:22.775142 3975 tablet_copy_service.cc:140] P 589fc4b586ea4f158b5009ac9e67348d: Received BeginTabletCopySession request for tablet f4bf819c041f4231b07d49fda34f476a from peer 5aeb2c34139d463ca3f9a33b059c5a36 ({username='slave'} at 127.3.173.67:33437)
I20250627 13:57:22.775553 3975 tablet_copy_service.cc:161] P 589fc4b586ea4f158b5009ac9e67348d: Beginning new tablet copy session on tablet f4bf819c041f4231b07d49fda34f476a from peer 5aeb2c34139d463ca3f9a33b059c5a36 at {username='slave'} at 127.3.173.67:33437: session id = 5aeb2c34139d463ca3f9a33b059c5a36-f4bf819c041f4231b07d49fda34f476a
I20250627 13:57:22.783210 3975 tablet_copy_source_session.cc:215] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:57:22.788501 4604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4bf819c041f4231b07d49fda34f476a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:57:22.806674 4604 tablet_copy_client.cc:806] T f4bf819c041f4231b07d49fda34f476a P 5aeb2c34139d463ca3f9a33b059c5a36: tablet copy: Starting download of 0 data blocks...
I20250627 13:57:22.807281 4604 tablet_copy_client.cc:670] T f4bf819c041f4231b07d49fda34f476a P 5aeb2c34139d463ca3f9a33b059c5a36: tablet copy: Starting download of 1 WAL segments...
I20250627 13:57:22.811019 4604 tablet_copy_client.cc:538] T f4bf819c041f4231b07d49fda34f476a P 5aeb2c34139d463ca3f9a33b059c5a36: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:57:22.817222 4604 tablet_bootstrap.cc:492] T f4bf819c041f4231b07d49fda34f476a P 5aeb2c34139d463ca3f9a33b059c5a36: Bootstrap starting.
I20250627 13:57:22.829411 4604 log.cc:826] T f4bf819c041f4231b07d49fda34f476a P 5aeb2c34139d463ca3f9a33b059c5a36: Log is configured to *not* fsync() on all Append() calls
I20250627 13:57:22.839902 4604 tablet_bootstrap.cc:492] T f4bf819c041f4231b07d49fda34f476a P 5aeb2c34139d463ca3f9a33b059c5a36: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:57:22.840617 4604 tablet_bootstrap.cc:492] T f4bf819c041f4231b07d49fda34f476a P 5aeb2c34139d463ca3f9a33b059c5a36: Bootstrap complete.
I20250627 13:57:22.841176 4604 ts_tablet_manager.cc:1397] T f4bf819c041f4231b07d49fda34f476a P 5aeb2c34139d463ca3f9a33b059c5a36: Time spent bootstrapping tablet: real 0.024s user 0.015s sys 0.008s
I20250627 13:57:22.857585 4604 raft_consensus.cc:357] T f4bf819c041f4231b07d49fda34f476a P 5aeb2c34139d463ca3f9a33b059c5a36 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 } } peers { permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37651 } } peers { permanent_uuid: "5aeb2c34139d463ca3f9a33b059c5a36" member_type: NON_VOTER last_known_addr { host: "127.3.173.67" port: 40213 } attrs { promote: true } } unsafe_config_change: true
I20250627 13:57:22.858453 4604 raft_consensus.cc:738] T f4bf819c041f4231b07d49fda34f476a P 5aeb2c34139d463ca3f9a33b059c5a36 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 5aeb2c34139d463ca3f9a33b059c5a36, State: Initialized, Role: LEARNER
I20250627 13:57:22.859071 4604 consensus_queue.cc:260] T f4bf819c041f4231b07d49fda34f476a P 5aeb2c34139d463ca3f9a33b059c5a36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 } } peers { permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37651 } } peers { permanent_uuid: "5aeb2c34139d463ca3f9a33b059c5a36" member_type: NON_VOTER last_known_addr { host: "127.3.173.67" port: 40213 } attrs { promote: true } } unsafe_config_change: true
I20250627 13:57:22.862051 4604 ts_tablet_manager.cc:1428] T f4bf819c041f4231b07d49fda34f476a P 5aeb2c34139d463ca3f9a33b059c5a36: Time spent starting tablet: real 0.021s user 0.018s sys 0.005s
I20250627 13:57:22.863615 3975 tablet_copy_service.cc:342] P 589fc4b586ea4f158b5009ac9e67348d: Request end of tablet copy session 5aeb2c34139d463ca3f9a33b059c5a36-f4bf819c041f4231b07d49fda34f476a received from {username='slave'} at 127.3.173.67:33437
I20250627 13:57:22.863940 3975 tablet_copy_service.cc:434] P 589fc4b586ea4f158b5009ac9e67348d: ending tablet copy session 5aeb2c34139d463ca3f9a33b059c5a36-f4bf819c041f4231b07d49fda34f476a on tablet f4bf819c041f4231b07d49fda34f476a with peer 5aeb2c34139d463ca3f9a33b059c5a36
I20250627 13:57:23.252296 4221 raft_consensus.cc:1215] T f4bf819c041f4231b07d49fda34f476a P 5aeb2c34139d463ca3f9a33b059c5a36 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.4->[4.5-4.5] Dedup: 4.5->[]
W20250627 13:57:23.500615 4478 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet f4bf819c041f4231b07d49fda34f476a on TS 03e0969ce0b143fe823bd3486ed9a130 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 03e0969ce0b143fe823bd3486ed9a130
I20250627 13:57:23.637581 4610 raft_consensus.cc:1062] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d: attempting to promote NON_VOTER 5aeb2c34139d463ca3f9a33b059c5a36 to VOTER
I20250627 13:57:23.639125 4610 consensus_queue.cc:237] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 } } peers { permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37651 } } peers { permanent_uuid: "5aeb2c34139d463ca3f9a33b059c5a36" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40213 } attrs { promote: false } } unsafe_config_change: true
I20250627 13:57:23.643554 4088 raft_consensus.cc:1273] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21 [term 4 FOLLOWER]: Refusing update from remote peer 589fc4b586ea4f158b5009ac9e67348d: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250627 13:57:23.643563 4221 raft_consensus.cc:1273] T f4bf819c041f4231b07d49fda34f476a P 5aeb2c34139d463ca3f9a33b059c5a36 [term 4 LEARNER]: Refusing update from remote peer 589fc4b586ea4f158b5009ac9e67348d: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250627 13:57:23.644629 4589 consensus_queue.cc:1035] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:57:23.645334 4609 consensus_queue.cc:1035] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aeb2c34139d463ca3f9a33b059c5a36" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:57:23.651059 4589 raft_consensus.cc:2953] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 5aeb2c34139d463ca3f9a33b059c5a36 (127.3.173.67) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 } } peers { permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37651 } } peers { permanent_uuid: "5aeb2c34139d463ca3f9a33b059c5a36" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40213 } attrs { promote: false } } unsafe_config_change: true }
I20250627 13:57:23.652460 4088 raft_consensus.cc:2953] T f4bf819c041f4231b07d49fda34f476a P 2b16f516baad43368e6368e633d5fb21 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 5aeb2c34139d463ca3f9a33b059c5a36 (127.3.173.67) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 } } peers { permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37651 } } peers { permanent_uuid: "5aeb2c34139d463ca3f9a33b059c5a36" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40213 } attrs { promote: false } } unsafe_config_change: true }
I20250627 13:57:23.654147 4221 raft_consensus.cc:2953] T f4bf819c041f4231b07d49fda34f476a P 5aeb2c34139d463ca3f9a33b059c5a36 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 5aeb2c34139d463ca3f9a33b059c5a36 (127.3.173.67) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 } } peers { permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37651 } } peers { permanent_uuid: "5aeb2c34139d463ca3f9a33b059c5a36" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40213 } attrs { promote: false } } unsafe_config_change: true }
I20250627 13:57:23.661749 4493 catalog_manager.cc:5582] T f4bf819c041f4231b07d49fda34f476a P 589fc4b586ea4f158b5009ac9e67348d reported cstate change: config changed from index 5 to 6, 5aeb2c34139d463ca3f9a33b059c5a36 (127.3.173.67) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "589fc4b586ea4f158b5009ac9e67348d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2b16f516baad43368e6368e633d5fb21" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 43109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5aeb2c34139d463ca3f9a33b059c5a36" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40213 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
I20250627 13:57:23.735858 3765 kudu-admin-test.cc:751] Waiting for Master to see new config...
I20250627 13:57:23.748934 3765 kudu-admin-test.cc:756] Tablet locations:
tablet_locations {
tablet_id: "f4bf819c041f4231b07d49fda34f476a"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: LEADER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "2b16f516baad43368e6368e633d5fb21"
rpc_addresses {
host: "127.3.173.66"
port: 43109
}
}
ts_infos {
permanent_uuid: "589fc4b586ea4f158b5009ac9e67348d"
rpc_addresses {
host: "127.3.173.65"
port: 37651
}
}
ts_infos {
permanent_uuid: "5aeb2c34139d463ca3f9a33b059c5a36"
rpc_addresses {
host: "127.3.173.67"
port: 40213
}
}
I20250627 13:57:23.752367 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 3871
I20250627 13:57:23.777241 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 4004
I20250627 13:57:23.800688 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 4137
I20250627 13:57:23.826308 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 4446
2025-06-27T13:57:23Z chronyd exiting
[ OK ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes (19232 ms)
[ RUN ] AdminCliTest.TestGracefulSpecificLeaderStepDown
I20250627 13:57:23.886467 3765 test_util.cc:276] Using random seed: 247111020
I20250627 13:57:23.892585 3765 ts_itest-base.cc:115] Starting cluster with:
I20250627 13:57:23.892814 3765 ts_itest-base.cc:116] --------------
I20250627 13:57:23.893025 3765 ts_itest-base.cc:117] 3 tablet servers
I20250627 13:57:23.893240 3765 ts_itest-base.cc:118] 3 replicas per TS
I20250627 13:57:23.893445 3765 ts_itest-base.cc:119] --------------
2025-06-27T13:57:23Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-27T13:57:23Z Disabled control of system clock
I20250627 13:57:23.927574 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.173.126:45597
--webserver_interface=127.3.173.126
--webserver_port=0
--builtin_ntp_servers=127.3.173.84:41439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.173.126:45597
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250627 13:57:24.214313 4628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:57:24.214841 4628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:57:24.215348 4628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:57:24.245703 4628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250627 13:57:24.246122 4628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:57:24.246364 4628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250627 13:57:24.246577 4628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250627 13:57:24.281087 4628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:41439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.3.173.126:45597
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.173.126:45597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.3.173.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:57:24.282351 4628 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:57:24.283877 4628 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:57:24.299620 4635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:57:24.299654 4634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:57:24.299664 4637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:57:25.484035 4636 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1182 milliseconds
I20250627 13:57:25.484150 4628 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250627 13:57:25.485361 4628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:57:25.488405 4628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:57:25.489828 4628 hybrid_clock.cc:648] HybridClock initialized: now 1751032645489788 us; error 56 us; skew 500 ppm
I20250627 13:57:25.490633 4628 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:57:25.499354 4628 webserver.cc:469] Webserver started at http://127.3.173.126:38561/ using document root <none> and password file <none>
I20250627 13:57:25.500236 4628 fs_manager.cc:362] Metadata directory not provided
I20250627 13:57:25.500473 4628 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:57:25.500921 4628 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:57:25.505132 4628 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "0bd6b49f50404c91b938fdc55701c49e"
format_stamp: "Formatted at 2025-06-27 13:57:25 on dist-test-slave-rn80"
I20250627 13:57:25.506194 4628 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "0bd6b49f50404c91b938fdc55701c49e"
format_stamp: "Formatted at 2025-06-27 13:57:25 on dist-test-slave-rn80"
I20250627 13:57:25.512735 4628 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250627 13:57:25.517937 4645 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:25.518921 4628 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250627 13:57:25.519258 4628 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
uuid: "0bd6b49f50404c91b938fdc55701c49e"
format_stamp: "Formatted at 2025-06-27 13:57:25 on dist-test-slave-rn80"
I20250627 13:57:25.519582 4628 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:57:25.565666 4628 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:57:25.567013 4628 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:57:25.567435 4628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:57:25.635715 4628 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.126:45597
I20250627 13:57:25.635782 4696 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.126:45597 every 8 connection(s)
I20250627 13:57:25.638178 4628 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250627 13:57:25.642730 4697 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:57:25.643842 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 4628
I20250627 13:57:25.644344 3765 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250627 13:57:25.668255 4697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bd6b49f50404c91b938fdc55701c49e: Bootstrap starting.
I20250627 13:57:25.673287 4697 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bd6b49f50404c91b938fdc55701c49e: Neither blocks nor log segments found. Creating new log.
I20250627 13:57:25.675187 4697 log.cc:826] T 00000000000000000000000000000000 P 0bd6b49f50404c91b938fdc55701c49e: Log is configured to *not* fsync() on all Append() calls
I20250627 13:57:25.679301 4697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bd6b49f50404c91b938fdc55701c49e: No bootstrap required, opened a new log
I20250627 13:57:25.696262 4697 raft_consensus.cc:357] T 00000000000000000000000000000000 P 0bd6b49f50404c91b938fdc55701c49e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bd6b49f50404c91b938fdc55701c49e" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 45597 } }
I20250627 13:57:25.696933 4697 raft_consensus.cc:383] T 00000000000000000000000000000000 P 0bd6b49f50404c91b938fdc55701c49e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:57:25.697190 4697 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0bd6b49f50404c91b938fdc55701c49e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bd6b49f50404c91b938fdc55701c49e, State: Initialized, Role: FOLLOWER
I20250627 13:57:25.698032 4697 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bd6b49f50404c91b938fdc55701c49e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bd6b49f50404c91b938fdc55701c49e" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 45597 } }
I20250627 13:57:25.698622 4697 raft_consensus.cc:397] T 00000000000000000000000000000000 P 0bd6b49f50404c91b938fdc55701c49e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250627 13:57:25.698859 4697 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0bd6b49f50404c91b938fdc55701c49e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250627 13:57:25.699111 4697 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 0bd6b49f50404c91b938fdc55701c49e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:57:25.703102 4697 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0bd6b49f50404c91b938fdc55701c49e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bd6b49f50404c91b938fdc55701c49e" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 45597 } }
I20250627 13:57:25.703729 4697 leader_election.cc:304] T 00000000000000000000000000000000 P 0bd6b49f50404c91b938fdc55701c49e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0bd6b49f50404c91b938fdc55701c49e; no voters:
I20250627 13:57:25.705835 4697 leader_election.cc:290] T 00000000000000000000000000000000 P 0bd6b49f50404c91b938fdc55701c49e [CANDIDATE]: Term 1 election: Requested vote from peers
I20250627 13:57:25.706583 4702 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 0bd6b49f50404c91b938fdc55701c49e [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:57:25.708626 4702 raft_consensus.cc:695] T 00000000000000000000000000000000 P 0bd6b49f50404c91b938fdc55701c49e [term 1 LEADER]: Becoming Leader. State: Replica: 0bd6b49f50404c91b938fdc55701c49e, State: Running, Role: LEADER
I20250627 13:57:25.709425 4702 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bd6b49f50404c91b938fdc55701c49e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bd6b49f50404c91b938fdc55701c49e" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 45597 } }
I20250627 13:57:25.710206 4697 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bd6b49f50404c91b938fdc55701c49e [sys.catalog]: configured and running, proceeding with master startup.
I20250627 13:57:25.719535 4703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bd6b49f50404c91b938fdc55701c49e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0bd6b49f50404c91b938fdc55701c49e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bd6b49f50404c91b938fdc55701c49e" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 45597 } } }
I20250627 13:57:25.719934 4704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bd6b49f50404c91b938fdc55701c49e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bd6b49f50404c91b938fdc55701c49e. Latest consensus state: current_term: 1 leader_uuid: "0bd6b49f50404c91b938fdc55701c49e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bd6b49f50404c91b938fdc55701c49e" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 45597 } } }
I20250627 13:57:25.720232 4703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bd6b49f50404c91b938fdc55701c49e [sys.catalog]: This master's current role is: LEADER
I20250627 13:57:25.720494 4704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bd6b49f50404c91b938fdc55701c49e [sys.catalog]: This master's current role is: LEADER
I20250627 13:57:25.724012 4711 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250627 13:57:25.734567 4711 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250627 13:57:25.748204 4711 catalog_manager.cc:1349] Generated new cluster ID: d613019d65664e09bea068867d7cd307
I20250627 13:57:25.748485 4711 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250627 13:57:25.763052 4711 catalog_manager.cc:1372] Generated new certificate authority record
I20250627 13:57:25.764508 4711 catalog_manager.cc:1506] Loading token signing keys...
I20250627 13:57:25.775355 4711 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 0bd6b49f50404c91b938fdc55701c49e: Generated new TSK 0
I20250627 13:57:25.776233 4711 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250627 13:57:25.789445 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.173.65:0
--local_ip_for_outbound_sockets=127.3.173.65
--webserver_interface=127.3.173.65
--webserver_port=0
--tserver_master_addrs=127.3.173.126:45597
--builtin_ntp_servers=127.3.173.84:41439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250627 13:57:26.076683 4721 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250627 13:57:26.077276 4721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:57:26.077510 4721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:57:26.077968 4721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:57:26.106539 4721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:57:26.107367 4721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.173.65
I20250627 13:57:26.139786 4721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:41439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.173.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.3.173.65
--webserver_port=0
--tserver_master_addrs=127.3.173.126:45597
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.173.65
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:57:26.141031 4721 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:57:26.142558 4721 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:57:26.158460 4728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:57:26.160692 4727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:57:26.161226 4721 server_base.cc:1048] running on GCE node
W20250627 13:57:26.160109 4730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:57:27.293316 4721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:57:27.295986 4721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:57:27.297457 4721 hybrid_clock.cc:648] HybridClock initialized: now 1751032647297406 us; error 48 us; skew 500 ppm
I20250627 13:57:27.298471 4721 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:57:27.305701 4721 webserver.cc:469] Webserver started at http://127.3.173.65:38113/ using document root <none> and password file <none>
I20250627 13:57:27.306854 4721 fs_manager.cc:362] Metadata directory not provided
I20250627 13:57:27.307104 4721 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:57:27.307631 4721 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:57:27.313949 4721 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "6c52e8b08bb74012a451ecd47d29497d"
format_stamp: "Formatted at 2025-06-27 13:57:27 on dist-test-slave-rn80"
I20250627 13:57:27.315410 4721 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "6c52e8b08bb74012a451ecd47d29497d"
format_stamp: "Formatted at 2025-06-27 13:57:27 on dist-test-slave-rn80"
I20250627 13:57:27.324193 4721 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.005s sys 0.004s
I20250627 13:57:27.331342 4737 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:27.332445 4721 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250627 13:57:27.332840 4721 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "6c52e8b08bb74012a451ecd47d29497d"
format_stamp: "Formatted at 2025-06-27 13:57:27 on dist-test-slave-rn80"
I20250627 13:57:27.333261 4721 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:57:27.398845 4721 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:57:27.400671 4721 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:57:27.401170 4721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:57:27.404384 4721 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:57:27.409727 4721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:57:27.410013 4721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:27.410313 4721 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:57:27.410538 4721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:27.550534 4721 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.65:38879
I20250627 13:57:27.550632 4849 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.65:38879 every 8 connection(s)
I20250627 13:57:27.552939 4721 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250627 13:57:27.556629 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 4721
I20250627 13:57:27.557060 3765 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250627 13:57:27.563488 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.173.66:0
--local_ip_for_outbound_sockets=127.3.173.66
--webserver_interface=127.3.173.66
--webserver_port=0
--tserver_master_addrs=127.3.173.126:45597
--builtin_ntp_servers=127.3.173.84:41439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250627 13:57:27.573325 4850 heartbeater.cc:344] Connected to a master server at 127.3.173.126:45597
I20250627 13:57:27.573727 4850 heartbeater.cc:461] Registering TS with master...
I20250627 13:57:27.574771 4850 heartbeater.cc:507] Master 127.3.173.126:45597 requested a full tablet report, sending...
I20250627 13:57:27.577241 4662 ts_manager.cc:194] Registered new tserver with Master: 6c52e8b08bb74012a451ecd47d29497d (127.3.173.65:38879)
I20250627 13:57:27.579036 4662 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.173.65:60131
W20250627 13:57:27.857966 4854 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250627 13:57:27.858685 4854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:57:27.858949 4854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:57:27.859431 4854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:57:27.889076 4854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:57:27.889895 4854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.173.66
I20250627 13:57:27.922922 4854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:41439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.173.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.3.173.66
--webserver_port=0
--tserver_master_addrs=127.3.173.126:45597
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.173.66
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:57:27.924191 4854 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:57:27.925783 4854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:57:27.940209 4860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:57:28.581993 4850 heartbeater.cc:499] Master 127.3.173.126:45597 was elected leader, sending a full tablet report...
W20250627 13:57:27.940228 4861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:57:27.940809 4863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:57:27.943004 4854 server_base.cc:1048] running on GCE node
I20250627 13:57:29.076308 4854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:57:29.078500 4854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:57:29.079864 4854 hybrid_clock.cc:648] HybridClock initialized: now 1751032649079829 us; error 29 us; skew 500 ppm
I20250627 13:57:29.080590 4854 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:57:29.086643 4854 webserver.cc:469] Webserver started at http://127.3.173.66:33933/ using document root <none> and password file <none>
I20250627 13:57:29.087458 4854 fs_manager.cc:362] Metadata directory not provided
I20250627 13:57:29.087641 4854 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:57:29.088125 4854 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:57:29.092200 4854 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "ea3c94b1386c4e1cb720070bb588509d"
format_stamp: "Formatted at 2025-06-27 13:57:29 on dist-test-slave-rn80"
I20250627 13:57:29.093183 4854 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "ea3c94b1386c4e1cb720070bb588509d"
format_stamp: "Formatted at 2025-06-27 13:57:29 on dist-test-slave-rn80"
I20250627 13:57:29.099622 4854 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.007s
I20250627 13:57:29.104903 4870 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:29.105868 4854 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250627 13:57:29.106184 4854 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ea3c94b1386c4e1cb720070bb588509d"
format_stamp: "Formatted at 2025-06-27 13:57:29 on dist-test-slave-rn80"
I20250627 13:57:29.106505 4854 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:57:29.159569 4854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:57:29.160984 4854 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:57:29.161389 4854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:57:29.163868 4854 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:57:29.168045 4854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:57:29.168252 4854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:29.168483 4854 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:57:29.168663 4854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:29.296195 4854 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.66:39631
I20250627 13:57:29.296303 4982 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.66:39631 every 8 connection(s)
I20250627 13:57:29.298676 4854 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250627 13:57:29.302685 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 4854
I20250627 13:57:29.303166 3765 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250627 13:57:29.309859 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.173.67:0
--local_ip_for_outbound_sockets=127.3.173.67
--webserver_interface=127.3.173.67
--webserver_port=0
--tserver_master_addrs=127.3.173.126:45597
--builtin_ntp_servers=127.3.173.84:41439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250627 13:57:29.319859 4983 heartbeater.cc:344] Connected to a master server at 127.3.173.126:45597
I20250627 13:57:29.320286 4983 heartbeater.cc:461] Registering TS with master...
I20250627 13:57:29.321229 4983 heartbeater.cc:507] Master 127.3.173.126:45597 requested a full tablet report, sending...
I20250627 13:57:29.323787 4662 ts_manager.cc:194] Registered new tserver with Master: ea3c94b1386c4e1cb720070bb588509d (127.3.173.66:39631)
I20250627 13:57:29.325621 4662 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.173.66:38341
W20250627 13:57:29.601558 4987 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250627 13:57:29.602191 4987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:57:29.602416 4987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:57:29.602826 4987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:57:29.635860 4987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:57:29.636656 4987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.173.67
I20250627 13:57:29.672281 4987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:41439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.173.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.3.173.67
--webserver_port=0
--tserver_master_addrs=127.3.173.126:45597
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.173.67
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:57:29.673635 4987 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:57:29.675228 4987 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:57:29.690523 4993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:57:30.328928 4983 heartbeater.cc:499] Master 127.3.173.126:45597 was elected leader, sending a full tablet report...
W20250627 13:57:29.690546 4994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:57:29.691502 4996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:57:29.692617 4987 server_base.cc:1048] running on GCE node
I20250627 13:57:30.836517 4987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:57:30.838714 4987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:57:30.840044 4987 hybrid_clock.cc:648] HybridClock initialized: now 1751032650840028 us; error 44 us; skew 500 ppm
I20250627 13:57:30.840811 4987 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:57:30.851876 4987 webserver.cc:469] Webserver started at http://127.3.173.67:33721/ using document root <none> and password file <none>
I20250627 13:57:30.852798 4987 fs_manager.cc:362] Metadata directory not provided
I20250627 13:57:30.852998 4987 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:57:30.853435 4987 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:57:30.858161 4987 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "fee8d741619e4ea58fe8fb339c86446e"
format_stamp: "Formatted at 2025-06-27 13:57:30 on dist-test-slave-rn80"
I20250627 13:57:30.859185 4987 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "fee8d741619e4ea58fe8fb339c86446e"
format_stamp: "Formatted at 2025-06-27 13:57:30 on dist-test-slave-rn80"
I20250627 13:57:30.865871 4987 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250627 13:57:30.871153 5003 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:30.872131 4987 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250627 13:57:30.872413 4987 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "fee8d741619e4ea58fe8fb339c86446e"
format_stamp: "Formatted at 2025-06-27 13:57:30 on dist-test-slave-rn80"
I20250627 13:57:30.872733 4987 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:57:30.935828 4987 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:57:30.937261 4987 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:57:30.937669 4987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:57:30.940183 4987 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:57:30.944590 4987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:57:30.944826 4987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:30.945068 4987 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:57:30.945235 4987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:31.072465 4987 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.67:32915
I20250627 13:57:31.072558 5115 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.67:32915 every 8 connection(s)
I20250627 13:57:31.074988 4987 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250627 13:57:31.079721 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 4987
I20250627 13:57:31.080184 3765 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250627 13:57:31.094579 5116 heartbeater.cc:344] Connected to a master server at 127.3.173.126:45597
I20250627 13:57:31.094982 5116 heartbeater.cc:461] Registering TS with master...
I20250627 13:57:31.095925 5116 heartbeater.cc:507] Master 127.3.173.126:45597 requested a full tablet report, sending...
I20250627 13:57:31.097994 4662 ts_manager.cc:194] Registered new tserver with Master: fee8d741619e4ea58fe8fb339c86446e (127.3.173.67:32915)
I20250627 13:57:31.099363 3765 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250627 13:57:31.099537 4662 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.173.67:34039
I20250627 13:57:31.134567 4662 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45116:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250627 13:57:31.153095 4662 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250627 13:57:31.202607 4785 tablet_service.cc:1468] Processing CreateTablet for tablet 84007ebcd3594809984bf02c335d5032 (DEFAULT_TABLE table=TestTable [id=f62d71a489b94d88be133de53d9af701]), partition=RANGE (key) PARTITION UNBOUNDED
I20250627 13:57:31.204782 4785 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84007ebcd3594809984bf02c335d5032. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:57:31.211419 4918 tablet_service.cc:1468] Processing CreateTablet for tablet 84007ebcd3594809984bf02c335d5032 (DEFAULT_TABLE table=TestTable [id=f62d71a489b94d88be133de53d9af701]), partition=RANGE (key) PARTITION UNBOUNDED
I20250627 13:57:31.211355 5051 tablet_service.cc:1468] Processing CreateTablet for tablet 84007ebcd3594809984bf02c335d5032 (DEFAULT_TABLE table=TestTable [id=f62d71a489b94d88be133de53d9af701]), partition=RANGE (key) PARTITION UNBOUNDED
I20250627 13:57:31.213068 4918 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84007ebcd3594809984bf02c335d5032. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:57:31.213203 5051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84007ebcd3594809984bf02c335d5032. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:57:31.233172 5135 tablet_bootstrap.cc:492] T 84007ebcd3594809984bf02c335d5032 P 6c52e8b08bb74012a451ecd47d29497d: Bootstrap starting.
I20250627 13:57:31.234376 5136 tablet_bootstrap.cc:492] T 84007ebcd3594809984bf02c335d5032 P ea3c94b1386c4e1cb720070bb588509d: Bootstrap starting.
I20250627 13:57:31.239960 5135 tablet_bootstrap.cc:654] T 84007ebcd3594809984bf02c335d5032 P 6c52e8b08bb74012a451ecd47d29497d: Neither blocks nor log segments found. Creating new log.
I20250627 13:57:31.241711 5136 tablet_bootstrap.cc:654] T 84007ebcd3594809984bf02c335d5032 P ea3c94b1386c4e1cb720070bb588509d: Neither blocks nor log segments found. Creating new log.
I20250627 13:57:31.241691 5137 tablet_bootstrap.cc:492] T 84007ebcd3594809984bf02c335d5032 P fee8d741619e4ea58fe8fb339c86446e: Bootstrap starting.
I20250627 13:57:31.242612 5135 log.cc:826] T 84007ebcd3594809984bf02c335d5032 P 6c52e8b08bb74012a451ecd47d29497d: Log is configured to *not* fsync() on all Append() calls
I20250627 13:57:31.243561 5136 log.cc:826] T 84007ebcd3594809984bf02c335d5032 P ea3c94b1386c4e1cb720070bb588509d: Log is configured to *not* fsync() on all Append() calls
I20250627 13:57:31.248294 5135 tablet_bootstrap.cc:492] T 84007ebcd3594809984bf02c335d5032 P 6c52e8b08bb74012a451ecd47d29497d: No bootstrap required, opened a new log
I20250627 13:57:31.248776 5135 ts_tablet_manager.cc:1397] T 84007ebcd3594809984bf02c335d5032 P 6c52e8b08bb74012a451ecd47d29497d: Time spent bootstrapping tablet: real 0.016s user 0.015s sys 0.000s
I20250627 13:57:31.249141 5137 tablet_bootstrap.cc:654] T 84007ebcd3594809984bf02c335d5032 P fee8d741619e4ea58fe8fb339c86446e: Neither blocks nor log segments found. Creating new log.
I20250627 13:57:31.251060 5136 tablet_bootstrap.cc:492] T 84007ebcd3594809984bf02c335d5032 P ea3c94b1386c4e1cb720070bb588509d: No bootstrap required, opened a new log
I20250627 13:57:31.251493 5136 ts_tablet_manager.cc:1397] T 84007ebcd3594809984bf02c335d5032 P ea3c94b1386c4e1cb720070bb588509d: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.008s
I20250627 13:57:31.251480 5137 log.cc:826] T 84007ebcd3594809984bf02c335d5032 P fee8d741619e4ea58fe8fb339c86446e: Log is configured to *not* fsync() on all Append() calls
I20250627 13:57:31.256883 5137 tablet_bootstrap.cc:492] T 84007ebcd3594809984bf02c335d5032 P fee8d741619e4ea58fe8fb339c86446e: No bootstrap required, opened a new log
I20250627 13:57:31.257349 5137 ts_tablet_manager.cc:1397] T 84007ebcd3594809984bf02c335d5032 P fee8d741619e4ea58fe8fb339c86446e: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20250627 13:57:31.269737 5136 raft_consensus.cc:357] T 84007ebcd3594809984bf02c335d5032 P ea3c94b1386c4e1cb720070bb588509d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c52e8b08bb74012a451ecd47d29497d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 38879 } } peers { permanent_uuid: "ea3c94b1386c4e1cb720070bb588509d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 39631 } } peers { permanent_uuid: "fee8d741619e4ea58fe8fb339c86446e" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 32915 } }
I20250627 13:57:31.270476 5136 raft_consensus.cc:738] T 84007ebcd3594809984bf02c335d5032 P ea3c94b1386c4e1cb720070bb588509d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea3c94b1386c4e1cb720070bb588509d, State: Initialized, Role: FOLLOWER
I20250627 13:57:31.271060 5136 consensus_queue.cc:260] T 84007ebcd3594809984bf02c335d5032 P ea3c94b1386c4e1cb720070bb588509d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c52e8b08bb74012a451ecd47d29497d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 38879 } } peers { permanent_uuid: "ea3c94b1386c4e1cb720070bb588509d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 39631 } } peers { permanent_uuid: "fee8d741619e4ea58fe8fb339c86446e" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 32915 } }
I20250627 13:57:31.275116 5136 ts_tablet_manager.cc:1428] T 84007ebcd3594809984bf02c335d5032 P ea3c94b1386c4e1cb720070bb588509d: Time spent starting tablet: real 0.023s user 0.023s sys 0.001s
I20250627 13:57:31.275581 5135 raft_consensus.cc:357] T 84007ebcd3594809984bf02c335d5032 P 6c52e8b08bb74012a451ecd47d29497d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c52e8b08bb74012a451ecd47d29497d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 38879 } } peers { permanent_uuid: "ea3c94b1386c4e1cb720070bb588509d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 39631 } } peers { permanent_uuid: "fee8d741619e4ea58fe8fb339c86446e" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 32915 } }
I20250627 13:57:31.276458 5135 raft_consensus.cc:738] T 84007ebcd3594809984bf02c335d5032 P 6c52e8b08bb74012a451ecd47d29497d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c52e8b08bb74012a451ecd47d29497d, State: Initialized, Role: FOLLOWER
I20250627 13:57:31.277123 5135 consensus_queue.cc:260] T 84007ebcd3594809984bf02c335d5032 P 6c52e8b08bb74012a451ecd47d29497d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c52e8b08bb74012a451ecd47d29497d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 38879 } } peers { permanent_uuid: "ea3c94b1386c4e1cb720070bb588509d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 39631 } } peers { permanent_uuid: "fee8d741619e4ea58fe8fb339c86446e" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 32915 } }
I20250627 13:57:31.281100 5135 ts_tablet_manager.cc:1428] T 84007ebcd3594809984bf02c335d5032 P 6c52e8b08bb74012a451ecd47d29497d: Time spent starting tablet: real 0.032s user 0.023s sys 0.008s
I20250627 13:57:31.283366 5137 raft_consensus.cc:357] T 84007ebcd3594809984bf02c335d5032 P fee8d741619e4ea58fe8fb339c86446e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c52e8b08bb74012a451ecd47d29497d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 38879 } } peers { permanent_uuid: "ea3c94b1386c4e1cb720070bb588509d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 39631 } } peers { permanent_uuid: "fee8d741619e4ea58fe8fb339c86446e" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 32915 } }
I20250627 13:57:31.284005 5137 raft_consensus.cc:738] T 84007ebcd3594809984bf02c335d5032 P fee8d741619e4ea58fe8fb339c86446e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fee8d741619e4ea58fe8fb339c86446e, State: Initialized, Role: FOLLOWER
I20250627 13:57:31.284591 5137 consensus_queue.cc:260] T 84007ebcd3594809984bf02c335d5032 P fee8d741619e4ea58fe8fb339c86446e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c52e8b08bb74012a451ecd47d29497d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 38879 } } peers { permanent_uuid: "ea3c94b1386c4e1cb720070bb588509d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 39631 } } peers { permanent_uuid: "fee8d741619e4ea58fe8fb339c86446e" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 32915 } }
I20250627 13:57:31.287915 5116 heartbeater.cc:499] Master 127.3.173.126:45597 was elected leader, sending a full tablet report...
I20250627 13:57:31.289384 5137 ts_tablet_manager.cc:1428] T 84007ebcd3594809984bf02c335d5032 P fee8d741619e4ea58fe8fb339c86446e: Time spent starting tablet: real 0.032s user 0.033s sys 0.000s
W20250627 13:57:31.304965 4984 tablet.cc:2378] T 84007ebcd3594809984bf02c335d5032 P ea3c94b1386c4e1cb720070bb588509d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250627 13:57:31.311230 3765 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
W20250627 13:57:31.311321 4851 tablet.cc:2378] T 84007ebcd3594809984bf02c335d5032 P 6c52e8b08bb74012a451ecd47d29497d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250627 13:57:31.314764 3765 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 6c52e8b08bb74012a451ecd47d29497d to finish bootstrapping
I20250627 13:57:31.328997 3765 ts_itest-base.cc:246] Waiting for 1 tablets on tserver ea3c94b1386c4e1cb720070bb588509d to finish bootstrapping
W20250627 13:57:31.329473 5117 tablet.cc:2378] T 84007ebcd3594809984bf02c335d5032 P fee8d741619e4ea58fe8fb339c86446e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250627 13:57:31.339645 3765 ts_itest-base.cc:246] Waiting for 1 tablets on tserver fee8d741619e4ea58fe8fb339c86446e to finish bootstrapping
I20250627 13:57:31.379024 4805 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "84007ebcd3594809984bf02c335d5032"
dest_uuid: "6c52e8b08bb74012a451ecd47d29497d"
from {username='slave'} at 127.0.0.1:39358
I20250627 13:57:31.379607 4805 raft_consensus.cc:491] T 84007ebcd3594809984bf02c335d5032 P 6c52e8b08bb74012a451ecd47d29497d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250627 13:57:31.379905 4805 raft_consensus.cc:3058] T 84007ebcd3594809984bf02c335d5032 P 6c52e8b08bb74012a451ecd47d29497d [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:57:31.384402 4805 raft_consensus.cc:513] T 84007ebcd3594809984bf02c335d5032 P 6c52e8b08bb74012a451ecd47d29497d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c52e8b08bb74012a451ecd47d29497d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 38879 } } peers { permanent_uuid: "ea3c94b1386c4e1cb720070bb588509d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 39631 } } peers { permanent_uuid: "fee8d741619e4ea58fe8fb339c86446e" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 32915 } }
I20250627 13:57:31.386621 4805 leader_election.cc:290] T 84007ebcd3594809984bf02c335d5032 P 6c52e8b08bb74012a451ecd47d29497d [CANDIDATE]: Term 1 election: Requested vote from peers ea3c94b1386c4e1cb720070bb588509d (127.3.173.66:39631), fee8d741619e4ea58fe8fb339c86446e (127.3.173.67:32915)
I20250627 13:57:31.396459 3765 cluster_itest_util.cc:257] Not converged past 1 yet: 0.0 0.0 0.0
I20250627 13:57:31.398916 5071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84007ebcd3594809984bf02c335d5032" candidate_uuid: "6c52e8b08bb74012a451ecd47d29497d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fee8d741619e4ea58fe8fb339c86446e"
I20250627 13:57:31.399202 4938 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84007ebcd3594809984bf02c335d5032" candidate_uuid: "6c52e8b08bb74012a451ecd47d29497d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ea3c94b1386c4e1cb720070bb588509d"
I20250627 13:57:31.399502 5071 raft_consensus.cc:3058] T 84007ebcd3594809984bf02c335d5032 P fee8d741619e4ea58fe8fb339c86446e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:57:31.399828 4938 raft_consensus.cc:3058] T 84007ebcd3594809984bf02c335d5032 P ea3c94b1386c4e1cb720070bb588509d [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:57:31.403908 5071 raft_consensus.cc:2466] T 84007ebcd3594809984bf02c335d5032 P fee8d741619e4ea58fe8fb339c86446e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c52e8b08bb74012a451ecd47d29497d in term 1.
I20250627 13:57:31.404037 4938 raft_consensus.cc:2466] T 84007ebcd3594809984bf02c335d5032 P ea3c94b1386c4e1cb720070bb588509d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c52e8b08bb74012a451ecd47d29497d in term 1.
I20250627 13:57:31.404979 4740 leader_election.cc:304] T 84007ebcd3594809984bf02c335d5032 P 6c52e8b08bb74012a451ecd47d29497d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c52e8b08bb74012a451ecd47d29497d, fee8d741619e4ea58fe8fb339c86446e; no voters:
I20250627 13:57:31.405709 5142 raft_consensus.cc:2802] T 84007ebcd3594809984bf02c335d5032 P 6c52e8b08bb74012a451ecd47d29497d [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:57:31.407330 5142 raft_consensus.cc:695] T 84007ebcd3594809984bf02c335d5032 P 6c52e8b08bb74012a451ecd47d29497d [term 1 LEADER]: Becoming Leader. State: Replica: 6c52e8b08bb74012a451ecd47d29497d, State: Running, Role: LEADER
I20250627 13:57:31.408188 5142 consensus_queue.cc:237] T 84007ebcd3594809984bf02c335d5032 P 6c52e8b08bb74012a451ecd47d29497d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c52e8b08bb74012a451ecd47d29497d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 38879 } } peers { permanent_uuid: "ea3c94b1386c4e1cb720070bb588509d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 39631 } } peers { permanent_uuid: "fee8d741619e4ea58fe8fb339c86446e" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 32915 } }
I20250627 13:57:31.418711 4660 catalog_manager.cc:5582] T 84007ebcd3594809984bf02c335d5032 P 6c52e8b08bb74012a451ecd47d29497d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c52e8b08bb74012a451ecd47d29497d (127.3.173.65). New cstate: current_term: 1 leader_uuid: "6c52e8b08bb74012a451ecd47d29497d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c52e8b08bb74012a451ecd47d29497d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 38879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea3c94b1386c4e1cb720070bb588509d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 39631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fee8d741619e4ea58fe8fb339c86446e" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 32915 } health_report { overall_health: UNKNOWN } } }
I20250627 13:57:31.501621 3765 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250627 13:57:31.706745 3765 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250627 13:57:31.821697 5142 consensus_queue.cc:1035] T 84007ebcd3594809984bf02c335d5032 P 6c52e8b08bb74012a451ecd47d29497d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fee8d741619e4ea58fe8fb339c86446e" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 32915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:57:31.837460 5148 consensus_queue.cc:1035] T 84007ebcd3594809984bf02c335d5032 P 6c52e8b08bb74012a451ecd47d29497d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea3c94b1386c4e1cb720070bb588509d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 39631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:57:33.701092 4805 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "84007ebcd3594809984bf02c335d5032"
dest_uuid: "6c52e8b08bb74012a451ecd47d29497d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39366
I20250627 13:57:33.701582 4805 raft_consensus.cc:604] T 84007ebcd3594809984bf02c335d5032 P 6c52e8b08bb74012a451ecd47d29497d [term 1 LEADER]: Received request to transfer leadership
I20250627 13:57:33.810493 5179 raft_consensus.cc:991] T 84007ebcd3594809984bf02c335d5032 P 6c52e8b08bb74012a451ecd47d29497d: : Instructing follower fee8d741619e4ea58fe8fb339c86446e to start an election
I20250627 13:57:33.810869 5162 raft_consensus.cc:1079] T 84007ebcd3594809984bf02c335d5032 P 6c52e8b08bb74012a451ecd47d29497d [term 1 LEADER]: Signalling peer fee8d741619e4ea58fe8fb339c86446e to start an election
I20250627 13:57:33.812122 5071 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "84007ebcd3594809984bf02c335d5032"
dest_uuid: "fee8d741619e4ea58fe8fb339c86446e"
from {username='slave'} at 127.3.173.65:53245
I20250627 13:57:33.812618 5071 raft_consensus.cc:491] T 84007ebcd3594809984bf02c335d5032 P fee8d741619e4ea58fe8fb339c86446e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250627 13:57:33.812868 5071 raft_consensus.cc:3058] T 84007ebcd3594809984bf02c335d5032 P fee8d741619e4ea58fe8fb339c86446e [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:57:33.816725 5071 raft_consensus.cc:513] T 84007ebcd3594809984bf02c335d5032 P fee8d741619e4ea58fe8fb339c86446e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c52e8b08bb74012a451ecd47d29497d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 38879 } } peers { permanent_uuid: "ea3c94b1386c4e1cb720070bb588509d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 39631 } } peers { permanent_uuid: "fee8d741619e4ea58fe8fb339c86446e" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 32915 } }
I20250627 13:57:33.818704 5071 leader_election.cc:290] T 84007ebcd3594809984bf02c335d5032 P fee8d741619e4ea58fe8fb339c86446e [CANDIDATE]: Term 2 election: Requested vote from peers 6c52e8b08bb74012a451ecd47d29497d (127.3.173.65:38879), ea3c94b1386c4e1cb720070bb588509d (127.3.173.66:39631)
I20250627 13:57:33.828169 4805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84007ebcd3594809984bf02c335d5032" candidate_uuid: "fee8d741619e4ea58fe8fb339c86446e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6c52e8b08bb74012a451ecd47d29497d"
I20250627 13:57:33.828634 4805 raft_consensus.cc:3053] T 84007ebcd3594809984bf02c335d5032 P 6c52e8b08bb74012a451ecd47d29497d [term 1 LEADER]: Stepping down as leader of term 1
I20250627 13:57:33.828959 4805 raft_consensus.cc:738] T 84007ebcd3594809984bf02c335d5032 P 6c52e8b08bb74012a451ecd47d29497d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6c52e8b08bb74012a451ecd47d29497d, State: Running, Role: LEADER
I20250627 13:57:33.829553 4805 consensus_queue.cc:260] T 84007ebcd3594809984bf02c335d5032 P 6c52e8b08bb74012a451ecd47d29497d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c52e8b08bb74012a451ecd47d29497d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 38879 } } peers { permanent_uuid: "ea3c94b1386c4e1cb720070bb588509d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 39631 } } peers { permanent_uuid: "fee8d741619e4ea58fe8fb339c86446e" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 32915 } }
I20250627 13:57:33.830832 4805 raft_consensus.cc:3058] T 84007ebcd3594809984bf02c335d5032 P 6c52e8b08bb74012a451ecd47d29497d [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:57:33.831270 4938 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84007ebcd3594809984bf02c335d5032" candidate_uuid: "fee8d741619e4ea58fe8fb339c86446e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ea3c94b1386c4e1cb720070bb588509d"
I20250627 13:57:33.831682 4938 raft_consensus.cc:3058] T 84007ebcd3594809984bf02c335d5032 P ea3c94b1386c4e1cb720070bb588509d [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:57:33.834908 4805 raft_consensus.cc:2466] T 84007ebcd3594809984bf02c335d5032 P 6c52e8b08bb74012a451ecd47d29497d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fee8d741619e4ea58fe8fb339c86446e in term 2.
I20250627 13:57:33.835636 4938 raft_consensus.cc:2466] T 84007ebcd3594809984bf02c335d5032 P ea3c94b1386c4e1cb720070bb588509d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fee8d741619e4ea58fe8fb339c86446e in term 2.
I20250627 13:57:33.835845 5007 leader_election.cc:304] T 84007ebcd3594809984bf02c335d5032 P fee8d741619e4ea58fe8fb339c86446e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c52e8b08bb74012a451ecd47d29497d, fee8d741619e4ea58fe8fb339c86446e; no voters:
I20250627 13:57:33.837517 5183 raft_consensus.cc:2802] T 84007ebcd3594809984bf02c335d5032 P fee8d741619e4ea58fe8fb339c86446e [term 2 FOLLOWER]: Leader election won for term 2
I20250627 13:57:33.838775 5183 raft_consensus.cc:695] T 84007ebcd3594809984bf02c335d5032 P fee8d741619e4ea58fe8fb339c86446e [term 2 LEADER]: Becoming Leader. State: Replica: fee8d741619e4ea58fe8fb339c86446e, State: Running, Role: LEADER
I20250627 13:57:33.839462 5183 consensus_queue.cc:237] T 84007ebcd3594809984bf02c335d5032 P fee8d741619e4ea58fe8fb339c86446e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c52e8b08bb74012a451ecd47d29497d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 38879 } } peers { permanent_uuid: "ea3c94b1386c4e1cb720070bb588509d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 39631 } } peers { permanent_uuid: "fee8d741619e4ea58fe8fb339c86446e" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 32915 } }
I20250627 13:57:33.846655 4660 catalog_manager.cc:5582] T 84007ebcd3594809984bf02c335d5032 P fee8d741619e4ea58fe8fb339c86446e reported cstate change: term changed from 1 to 2, leader changed from 6c52e8b08bb74012a451ecd47d29497d (127.3.173.65) to fee8d741619e4ea58fe8fb339c86446e (127.3.173.67). New cstate: current_term: 2 leader_uuid: "fee8d741619e4ea58fe8fb339c86446e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c52e8b08bb74012a451ecd47d29497d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 38879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea3c94b1386c4e1cb720070bb588509d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 39631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fee8d741619e4ea58fe8fb339c86446e" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 32915 } health_report { overall_health: HEALTHY } } }
I20250627 13:57:34.265570 4805 raft_consensus.cc:1273] T 84007ebcd3594809984bf02c335d5032 P 6c52e8b08bb74012a451ecd47d29497d [term 2 FOLLOWER]: Refusing update from remote peer fee8d741619e4ea58fe8fb339c86446e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250627 13:57:34.266778 5183 consensus_queue.cc:1035] T 84007ebcd3594809984bf02c335d5032 P fee8d741619e4ea58fe8fb339c86446e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c52e8b08bb74012a451ecd47d29497d" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 38879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250627 13:57:34.276945 4938 raft_consensus.cc:1273] T 84007ebcd3594809984bf02c335d5032 P ea3c94b1386c4e1cb720070bb588509d [term 2 FOLLOWER]: Refusing update from remote peer fee8d741619e4ea58fe8fb339c86446e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250627 13:57:34.278352 5184 consensus_queue.cc:1035] T 84007ebcd3594809984bf02c335d5032 P fee8d741619e4ea58fe8fb339c86446e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea3c94b1386c4e1cb720070bb588509d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 39631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20250627 13:57:34.309403 4979 debug-util.cc:398] Leaking SignalData structure 0x7b08000ac600 after lost signal to thread 4855
I20250627 13:57:36.498673 4805 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "84007ebcd3594809984bf02c335d5032"
dest_uuid: "6c52e8b08bb74012a451ecd47d29497d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36292
I20250627 13:57:36.499307 4805 raft_consensus.cc:604] T 84007ebcd3594809984bf02c335d5032 P 6c52e8b08bb74012a451ecd47d29497d [term 2 FOLLOWER]: Received request to transfer leadership
I20250627 13:57:36.499645 4805 raft_consensus.cc:612] T 84007ebcd3594809984bf02c335d5032 P 6c52e8b08bb74012a451ecd47d29497d [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250627 13:57:37.537354 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 4721
I20250627 13:57:37.560920 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 4854
I20250627 13:57:37.587292 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 4987
I20250627 13:57:37.612548 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 4628
2025-06-27T13:57:37Z chronyd exiting
[ OK ] AdminCliTest.TestGracefulSpecificLeaderStepDown (13781 ms)
[ RUN ] AdminCliTest.TestDescribeTableColumnFlags
I20250627 13:57:37.669327 3765 test_util.cc:276] Using random seed: 260893879
I20250627 13:57:37.673290 3765 ts_itest-base.cc:115] Starting cluster with:
I20250627 13:57:37.673441 3765 ts_itest-base.cc:116] --------------
I20250627 13:57:37.673553 3765 ts_itest-base.cc:117] 3 tablet servers
I20250627 13:57:37.673657 3765 ts_itest-base.cc:118] 3 replicas per TS
I20250627 13:57:37.673770 3765 ts_itest-base.cc:119] --------------
2025-06-27T13:57:37Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-27T13:57:37Z Disabled control of system clock
I20250627 13:57:37.708108 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.173.126:41315
--webserver_interface=127.3.173.126
--webserver_port=0
--builtin_ntp_servers=127.3.173.84:41745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.173.126:41315 with env {}
W20250627 13:57:38.000684 5227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:57:38.001246 5227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:57:38.001648 5227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:57:38.031986 5227 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250627 13:57:38.032307 5227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:57:38.032502 5227 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250627 13:57:38.032689 5227 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250627 13:57:38.066941 5227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:41745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.173.126:41315
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.173.126:41315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.3.173.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:57:38.068194 5227 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:57:38.069846 5227 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:57:38.084179 5234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:57:38.084498 5233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:57:38.084969 5236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:57:38.085623 5227 server_base.cc:1048] running on GCE node
I20250627 13:57:39.231213 5227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:57:39.233708 5227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:57:39.235054 5227 hybrid_clock.cc:648] HybridClock initialized: now 1751032659235018 us; error 53 us; skew 500 ppm
I20250627 13:57:39.235802 5227 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:57:39.241863 5227 webserver.cc:469] Webserver started at http://127.3.173.126:40663/ using document root <none> and password file <none>
I20250627 13:57:39.242799 5227 fs_manager.cc:362] Metadata directory not provided
I20250627 13:57:39.243011 5227 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:57:39.243466 5227 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:57:39.248894 5227 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "498da380faf948bfb06b04e14209a4dd"
format_stamp: "Formatted at 2025-06-27 13:57:39 on dist-test-slave-rn80"
I20250627 13:57:39.249951 5227 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "498da380faf948bfb06b04e14209a4dd"
format_stamp: "Formatted at 2025-06-27 13:57:39 on dist-test-slave-rn80"
I20250627 13:57:39.256783 5227 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20250627 13:57:39.262168 5243 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:39.263168 5227 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250627 13:57:39.263494 5227 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
uuid: "498da380faf948bfb06b04e14209a4dd"
format_stamp: "Formatted at 2025-06-27 13:57:39 on dist-test-slave-rn80"
I20250627 13:57:39.263818 5227 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:57:39.314360 5227 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:57:39.315793 5227 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:57:39.316219 5227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:57:39.384737 5227 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.126:41315
I20250627 13:57:39.384814 5294 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.126:41315 every 8 connection(s)
I20250627 13:57:39.387440 5227 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250627 13:57:39.392338 5295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:57:39.393098 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 5227
I20250627 13:57:39.393600 3765 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250627 13:57:39.417555 5295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 498da380faf948bfb06b04e14209a4dd: Bootstrap starting.
I20250627 13:57:39.423058 5295 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 498da380faf948bfb06b04e14209a4dd: Neither blocks nor log segments found. Creating new log.
I20250627 13:57:39.424751 5295 log.cc:826] T 00000000000000000000000000000000 P 498da380faf948bfb06b04e14209a4dd: Log is configured to *not* fsync() on all Append() calls
I20250627 13:57:39.429150 5295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 498da380faf948bfb06b04e14209a4dd: No bootstrap required, opened a new log
I20250627 13:57:39.446467 5295 raft_consensus.cc:357] T 00000000000000000000000000000000 P 498da380faf948bfb06b04e14209a4dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "498da380faf948bfb06b04e14209a4dd" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 41315 } }
I20250627 13:57:39.447093 5295 raft_consensus.cc:383] T 00000000000000000000000000000000 P 498da380faf948bfb06b04e14209a4dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:57:39.447340 5295 raft_consensus.cc:738] T 00000000000000000000000000000000 P 498da380faf948bfb06b04e14209a4dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 498da380faf948bfb06b04e14209a4dd, State: Initialized, Role: FOLLOWER
I20250627 13:57:39.448040 5295 consensus_queue.cc:260] T 00000000000000000000000000000000 P 498da380faf948bfb06b04e14209a4dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "498da380faf948bfb06b04e14209a4dd" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 41315 } }
I20250627 13:57:39.448582 5295 raft_consensus.cc:397] T 00000000000000000000000000000000 P 498da380faf948bfb06b04e14209a4dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250627 13:57:39.448837 5295 raft_consensus.cc:491] T 00000000000000000000000000000000 P 498da380faf948bfb06b04e14209a4dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250627 13:57:39.449118 5295 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 498da380faf948bfb06b04e14209a4dd [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:57:39.453140 5295 raft_consensus.cc:513] T 00000000000000000000000000000000 P 498da380faf948bfb06b04e14209a4dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "498da380faf948bfb06b04e14209a4dd" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 41315 } }
I20250627 13:57:39.453882 5295 leader_election.cc:304] T 00000000000000000000000000000000 P 498da380faf948bfb06b04e14209a4dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 498da380faf948bfb06b04e14209a4dd; no voters:
I20250627 13:57:39.455482 5295 leader_election.cc:290] T 00000000000000000000000000000000 P 498da380faf948bfb06b04e14209a4dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20250627 13:57:39.456151 5300 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 498da380faf948bfb06b04e14209a4dd [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:57:39.458276 5300 raft_consensus.cc:695] T 00000000000000000000000000000000 P 498da380faf948bfb06b04e14209a4dd [term 1 LEADER]: Becoming Leader. State: Replica: 498da380faf948bfb06b04e14209a4dd, State: Running, Role: LEADER
I20250627 13:57:39.458920 5300 consensus_queue.cc:237] T 00000000000000000000000000000000 P 498da380faf948bfb06b04e14209a4dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "498da380faf948bfb06b04e14209a4dd" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 41315 } }
I20250627 13:57:39.459317 5295 sys_catalog.cc:564] T 00000000000000000000000000000000 P 498da380faf948bfb06b04e14209a4dd [sys.catalog]: configured and running, proceeding with master startup.
I20250627 13:57:39.473096 5302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 498da380faf948bfb06b04e14209a4dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 498da380faf948bfb06b04e14209a4dd. Latest consensus state: current_term: 1 leader_uuid: "498da380faf948bfb06b04e14209a4dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "498da380faf948bfb06b04e14209a4dd" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 41315 } } }
I20250627 13:57:39.474728 5302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 498da380faf948bfb06b04e14209a4dd [sys.catalog]: This master's current role is: LEADER
I20250627 13:57:39.476298 5301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 498da380faf948bfb06b04e14209a4dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "498da380faf948bfb06b04e14209a4dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "498da380faf948bfb06b04e14209a4dd" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 41315 } } }
I20250627 13:57:39.478120 5301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 498da380faf948bfb06b04e14209a4dd [sys.catalog]: This master's current role is: LEADER
I20250627 13:57:39.491403 5308 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250627 13:57:39.501744 5308 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250627 13:57:39.517731 5308 catalog_manager.cc:1349] Generated new cluster ID: 6a0fb1236a2548879bf6bc891fe98bee
I20250627 13:57:39.518055 5308 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250627 13:57:39.535717 5308 catalog_manager.cc:1372] Generated new certificate authority record
I20250627 13:57:39.537070 5308 catalog_manager.cc:1506] Loading token signing keys...
I20250627 13:57:39.550040 5308 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 498da380faf948bfb06b04e14209a4dd: Generated new TSK 0
I20250627 13:57:39.550877 5308 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250627 13:57:39.565369 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.173.65:0
--local_ip_for_outbound_sockets=127.3.173.65
--webserver_interface=127.3.173.65
--webserver_port=0
--tserver_master_addrs=127.3.173.126:41315
--builtin_ntp_servers=127.3.173.84:41745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250627 13:57:39.879055 5319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:57:39.879547 5319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:57:39.880019 5319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:57:39.910912 5319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:57:39.911769 5319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.173.65
I20250627 13:57:39.946382 5319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:41745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.173.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.3.173.65
--webserver_port=0
--tserver_master_addrs=127.3.173.126:41315
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.173.65
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:57:39.947674 5319 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:57:39.949365 5319 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:57:39.966507 5326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:57:39.968606 5328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:57:39.967482 5325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:57:41.128314 5327 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250627 13:57:41.128496 5319 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250627 13:57:41.132680 5319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:57:41.135380 5319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:57:41.136898 5319 hybrid_clock.cc:648] HybridClock initialized: now 1751032661136863 us; error 55 us; skew 500 ppm
I20250627 13:57:41.137931 5319 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:57:41.145471 5319 webserver.cc:469] Webserver started at http://127.3.173.65:42067/ using document root <none> and password file <none>
I20250627 13:57:41.146631 5319 fs_manager.cc:362] Metadata directory not provided
I20250627 13:57:41.146903 5319 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:57:41.147460 5319 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:57:41.153710 5319 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "aa2c2d51dd634f88bc3a29fd4ffb5925"
format_stamp: "Formatted at 2025-06-27 13:57:41 on dist-test-slave-rn80"
I20250627 13:57:41.155134 5319 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "aa2c2d51dd634f88bc3a29fd4ffb5925"
format_stamp: "Formatted at 2025-06-27 13:57:41 on dist-test-slave-rn80"
I20250627 13:57:41.163743 5319 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s
I20250627 13:57:41.170823 5335 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:41.171881 5319 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250627 13:57:41.172278 5319 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "aa2c2d51dd634f88bc3a29fd4ffb5925"
format_stamp: "Formatted at 2025-06-27 13:57:41 on dist-test-slave-rn80"
I20250627 13:57:41.172690 5319 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:57:41.240118 5319 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:57:41.241550 5319 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:57:41.241990 5319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:57:41.244963 5319 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:57:41.249322 5319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:57:41.249517 5319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:41.249796 5319 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:57:41.249959 5319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:41.381855 5319 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.65:43889
I20250627 13:57:41.381928 5447 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.65:43889 every 8 connection(s)
I20250627 13:57:41.385284 5319 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250627 13:57:41.391683 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 5319
I20250627 13:57:41.392140 3765 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250627 13:57:41.398890 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.173.66:0
--local_ip_for_outbound_sockets=127.3.173.66
--webserver_interface=127.3.173.66
--webserver_port=0
--tserver_master_addrs=127.3.173.126:41315
--builtin_ntp_servers=127.3.173.84:41745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250627 13:57:41.407141 5448 heartbeater.cc:344] Connected to a master server at 127.3.173.126:41315
I20250627 13:57:41.407697 5448 heartbeater.cc:461] Registering TS with master...
I20250627 13:57:41.408990 5448 heartbeater.cc:507] Master 127.3.173.126:41315 requested a full tablet report, sending...
I20250627 13:57:41.411633 5260 ts_manager.cc:194] Registered new tserver with Master: aa2c2d51dd634f88bc3a29fd4ffb5925 (127.3.173.65:43889)
I20250627 13:57:41.413468 5260 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.173.65:37665
W20250627 13:57:41.697939 5452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:57:41.698505 5452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:57:41.699011 5452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:57:41.729691 5452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:57:41.730576 5452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.173.66
I20250627 13:57:41.764640 5452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:41745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.173.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.3.173.66
--webserver_port=0
--tserver_master_addrs=127.3.173.126:41315
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.173.66
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:57:41.765889 5452 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:57:41.767493 5452 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:57:41.783258 5458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:57:42.417527 5448 heartbeater.cc:499] Master 127.3.173.126:41315 was elected leader, sending a full tablet report...
W20250627 13:57:41.783334 5459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:57:41.785413 5452 server_base.cc:1048] running on GCE node
W20250627 13:57:41.784297 5461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:57:42.952057 5452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:57:42.954258 5452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:57:42.955606 5452 hybrid_clock.cc:648] HybridClock initialized: now 1751032662955566 us; error 60 us; skew 500 ppm
I20250627 13:57:42.956395 5452 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:57:42.967126 5452 webserver.cc:469] Webserver started at http://127.3.173.66:42573/ using document root <none> and password file <none>
I20250627 13:57:42.968039 5452 fs_manager.cc:362] Metadata directory not provided
I20250627 13:57:42.968230 5452 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:57:42.968703 5452 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:57:42.973093 5452 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "f399d76f5690464c9d1497cdec0fd39c"
format_stamp: "Formatted at 2025-06-27 13:57:42 on dist-test-slave-rn80"
I20250627 13:57:42.974192 5452 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "f399d76f5690464c9d1497cdec0fd39c"
format_stamp: "Formatted at 2025-06-27 13:57:42 on dist-test-slave-rn80"
I20250627 13:57:42.981194 5452 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20250627 13:57:42.986677 5468 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:42.987679 5452 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20250627 13:57:42.987987 5452 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "f399d76f5690464c9d1497cdec0fd39c"
format_stamp: "Formatted at 2025-06-27 13:57:42 on dist-test-slave-rn80"
I20250627 13:57:42.988319 5452 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:57:43.037585 5452 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:57:43.039009 5452 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:57:43.039469 5452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:57:43.041872 5452 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:57:43.045814 5452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:57:43.046036 5452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:43.046298 5452 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:57:43.046484 5452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:43.178532 5452 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.66:37231
I20250627 13:57:43.178634 5580 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.66:37231 every 8 connection(s)
I20250627 13:57:43.181017 5452 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250627 13:57:43.191103 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 5452
I20250627 13:57:43.191592 3765 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250627 13:57:43.198410 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.173.67:0
--local_ip_for_outbound_sockets=127.3.173.67
--webserver_interface=127.3.173.67
--webserver_port=0
--tserver_master_addrs=127.3.173.126:41315
--builtin_ntp_servers=127.3.173.84:41745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250627 13:57:43.202908 5581 heartbeater.cc:344] Connected to a master server at 127.3.173.126:41315
I20250627 13:57:43.203328 5581 heartbeater.cc:461] Registering TS with master...
I20250627 13:57:43.204316 5581 heartbeater.cc:507] Master 127.3.173.126:41315 requested a full tablet report, sending...
I20250627 13:57:43.206496 5260 ts_manager.cc:194] Registered new tserver with Master: f399d76f5690464c9d1497cdec0fd39c (127.3.173.66:37231)
I20250627 13:57:43.207710 5260 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.173.66:37741
W20250627 13:57:43.508654 5585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:57:43.509171 5585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:57:43.509680 5585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:57:43.540829 5585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:57:43.541652 5585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.173.67
I20250627 13:57:43.576303 5585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:41745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.173.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.3.173.67
--webserver_port=0
--tserver_master_addrs=127.3.173.126:41315
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.173.67
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:57:43.577544 5585 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:57:43.579166 5585 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:57:43.594982 5592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:57:44.211293 5581 heartbeater.cc:499] Master 127.3.173.126:41315 was elected leader, sending a full tablet report...
W20250627 13:57:43.595132 5591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:57:43.596087 5594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:57:44.750324 5593 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250627 13:57:44.750442 5585 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250627 13:57:44.754216 5585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:57:44.756299 5585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:57:44.757642 5585 hybrid_clock.cc:648] HybridClock initialized: now 1751032664757599 us; error 64 us; skew 500 ppm
I20250627 13:57:44.758455 5585 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:57:44.764453 5585 webserver.cc:469] Webserver started at http://127.3.173.67:34481/ using document root <none> and password file <none>
I20250627 13:57:44.765343 5585 fs_manager.cc:362] Metadata directory not provided
I20250627 13:57:44.765554 5585 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:57:44.766013 5585 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:57:44.770421 5585 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "9f0cfc4965f1490bb71ad77dcc4131c5"
format_stamp: "Formatted at 2025-06-27 13:57:44 on dist-test-slave-rn80"
I20250627 13:57:44.771544 5585 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "9f0cfc4965f1490bb71ad77dcc4131c5"
format_stamp: "Formatted at 2025-06-27 13:57:44 on dist-test-slave-rn80"
I20250627 13:57:44.778358 5585 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20250627 13:57:44.783773 5602 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:44.784758 5585 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20250627 13:57:44.785055 5585 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9f0cfc4965f1490bb71ad77dcc4131c5"
format_stamp: "Formatted at 2025-06-27 13:57:44 on dist-test-slave-rn80"
I20250627 13:57:44.785378 5585 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:57:44.849571 5585 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:57:44.851065 5585 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:57:44.851502 5585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:57:44.853915 5585 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:57:44.857903 5585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:57:44.858126 5585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:44.858371 5585 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:57:44.858541 5585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:44.987303 5585 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.67:40589
I20250627 13:57:44.987411 5714 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.67:40589 every 8 connection(s)
I20250627 13:57:44.989745 5585 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250627 13:57:44.992143 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 5585
I20250627 13:57:44.992836 3765 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250627 13:57:45.011075 5715 heartbeater.cc:344] Connected to a master server at 127.3.173.126:41315
I20250627 13:57:45.011560 5715 heartbeater.cc:461] Registering TS with master...
I20250627 13:57:45.012650 5715 heartbeater.cc:507] Master 127.3.173.126:41315 requested a full tablet report, sending...
I20250627 13:57:45.014845 5260 ts_manager.cc:194] Registered new tserver with Master: 9f0cfc4965f1490bb71ad77dcc4131c5 (127.3.173.67:40589)
I20250627 13:57:45.016037 5260 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.173.67:49253
I20250627 13:57:45.027240 3765 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250627 13:57:45.059273 5260 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59346:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250627 13:57:45.077785 5260 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250627 13:57:45.124788 5383 tablet_service.cc:1468] Processing CreateTablet for tablet f39376a63b224e06bac9089e6368a2a9 (DEFAULT_TABLE table=TestTable [id=adcbe7a4a3c24a1192f2f55f51e6b108]), partition=RANGE (key) PARTITION UNBOUNDED
I20250627 13:57:45.126925 5383 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f39376a63b224e06bac9089e6368a2a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:57:45.131304 5516 tablet_service.cc:1468] Processing CreateTablet for tablet f39376a63b224e06bac9089e6368a2a9 (DEFAULT_TABLE table=TestTable [id=adcbe7a4a3c24a1192f2f55f51e6b108]), partition=RANGE (key) PARTITION UNBOUNDED
I20250627 13:57:45.132095 5650 tablet_service.cc:1468] Processing CreateTablet for tablet f39376a63b224e06bac9089e6368a2a9 (DEFAULT_TABLE table=TestTable [id=adcbe7a4a3c24a1192f2f55f51e6b108]), partition=RANGE (key) PARTITION UNBOUNDED
I20250627 13:57:45.133358 5516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f39376a63b224e06bac9089e6368a2a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:57:45.133991 5650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f39376a63b224e06bac9089e6368a2a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:57:45.150974 5734 tablet_bootstrap.cc:492] T f39376a63b224e06bac9089e6368a2a9 P aa2c2d51dd634f88bc3a29fd4ffb5925: Bootstrap starting.
I20250627 13:57:45.157301 5734 tablet_bootstrap.cc:654] T f39376a63b224e06bac9089e6368a2a9 P aa2c2d51dd634f88bc3a29fd4ffb5925: Neither blocks nor log segments found. Creating new log.
I20250627 13:57:45.159472 5734 log.cc:826] T f39376a63b224e06bac9089e6368a2a9 P aa2c2d51dd634f88bc3a29fd4ffb5925: Log is configured to *not* fsync() on all Append() calls
I20250627 13:57:45.161631 5735 tablet_bootstrap.cc:492] T f39376a63b224e06bac9089e6368a2a9 P f399d76f5690464c9d1497cdec0fd39c: Bootstrap starting.
I20250627 13:57:45.163087 5736 tablet_bootstrap.cc:492] T f39376a63b224e06bac9089e6368a2a9 P 9f0cfc4965f1490bb71ad77dcc4131c5: Bootstrap starting.
I20250627 13:57:45.168793 5734 tablet_bootstrap.cc:492] T f39376a63b224e06bac9089e6368a2a9 P aa2c2d51dd634f88bc3a29fd4ffb5925: No bootstrap required, opened a new log
I20250627 13:57:45.169219 5734 ts_tablet_manager.cc:1397] T f39376a63b224e06bac9089e6368a2a9 P aa2c2d51dd634f88bc3a29fd4ffb5925: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.010s
I20250627 13:57:45.169588 5735 tablet_bootstrap.cc:654] T f39376a63b224e06bac9089e6368a2a9 P f399d76f5690464c9d1497cdec0fd39c: Neither blocks nor log segments found. Creating new log.
I20250627 13:57:45.170687 5736 tablet_bootstrap.cc:654] T f39376a63b224e06bac9089e6368a2a9 P 9f0cfc4965f1490bb71ad77dcc4131c5: Neither blocks nor log segments found. Creating new log.
I20250627 13:57:45.172017 5735 log.cc:826] T f39376a63b224e06bac9089e6368a2a9 P f399d76f5690464c9d1497cdec0fd39c: Log is configured to *not* fsync() on all Append() calls
I20250627 13:57:45.173077 5736 log.cc:826] T f39376a63b224e06bac9089e6368a2a9 P 9f0cfc4965f1490bb71ad77dcc4131c5: Log is configured to *not* fsync() on all Append() calls
I20250627 13:57:45.178292 5735 tablet_bootstrap.cc:492] T f39376a63b224e06bac9089e6368a2a9 P f399d76f5690464c9d1497cdec0fd39c: No bootstrap required, opened a new log
I20250627 13:57:45.178874 5735 ts_tablet_manager.cc:1397] T f39376a63b224e06bac9089e6368a2a9 P f399d76f5690464c9d1497cdec0fd39c: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.007s
I20250627 13:57:45.178874 5736 tablet_bootstrap.cc:492] T f39376a63b224e06bac9089e6368a2a9 P 9f0cfc4965f1490bb71ad77dcc4131c5: No bootstrap required, opened a new log
I20250627 13:57:45.179415 5736 ts_tablet_manager.cc:1397] T f39376a63b224e06bac9089e6368a2a9 P 9f0cfc4965f1490bb71ad77dcc4131c5: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.009s
I20250627 13:57:45.200671 5734 raft_consensus.cc:357] T f39376a63b224e06bac9089e6368a2a9 P aa2c2d51dd634f88bc3a29fd4ffb5925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa2c2d51dd634f88bc3a29fd4ffb5925" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43889 } } peers { permanent_uuid: "9f0cfc4965f1490bb71ad77dcc4131c5" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40589 } } peers { permanent_uuid: "f399d76f5690464c9d1497cdec0fd39c" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 37231 } }
I20250627 13:57:45.201646 5734 raft_consensus.cc:383] T f39376a63b224e06bac9089e6368a2a9 P aa2c2d51dd634f88bc3a29fd4ffb5925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:57:45.201992 5734 raft_consensus.cc:738] T f39376a63b224e06bac9089e6368a2a9 P aa2c2d51dd634f88bc3a29fd4ffb5925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa2c2d51dd634f88bc3a29fd4ffb5925, State: Initialized, Role: FOLLOWER
I20250627 13:57:45.203042 5734 consensus_queue.cc:260] T f39376a63b224e06bac9089e6368a2a9 P aa2c2d51dd634f88bc3a29fd4ffb5925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa2c2d51dd634f88bc3a29fd4ffb5925" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43889 } } peers { permanent_uuid: "9f0cfc4965f1490bb71ad77dcc4131c5" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40589 } } peers { permanent_uuid: "f399d76f5690464c9d1497cdec0fd39c" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 37231 } }
I20250627 13:57:45.207932 5736 raft_consensus.cc:357] T f39376a63b224e06bac9089e6368a2a9 P 9f0cfc4965f1490bb71ad77dcc4131c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa2c2d51dd634f88bc3a29fd4ffb5925" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43889 } } peers { permanent_uuid: "9f0cfc4965f1490bb71ad77dcc4131c5" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40589 } } peers { permanent_uuid: "f399d76f5690464c9d1497cdec0fd39c" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 37231 } }
I20250627 13:57:45.208910 5736 raft_consensus.cc:383] T f39376a63b224e06bac9089e6368a2a9 P 9f0cfc4965f1490bb71ad77dcc4131c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:57:45.208562 5735 raft_consensus.cc:357] T f39376a63b224e06bac9089e6368a2a9 P f399d76f5690464c9d1497cdec0fd39c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa2c2d51dd634f88bc3a29fd4ffb5925" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43889 } } peers { permanent_uuid: "9f0cfc4965f1490bb71ad77dcc4131c5" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40589 } } peers { permanent_uuid: "f399d76f5690464c9d1497cdec0fd39c" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 37231 } }
I20250627 13:57:45.209230 5736 raft_consensus.cc:738] T f39376a63b224e06bac9089e6368a2a9 P 9f0cfc4965f1490bb71ad77dcc4131c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f0cfc4965f1490bb71ad77dcc4131c5, State: Initialized, Role: FOLLOWER
I20250627 13:57:45.209476 5735 raft_consensus.cc:383] T f39376a63b224e06bac9089e6368a2a9 P f399d76f5690464c9d1497cdec0fd39c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:57:45.209825 5735 raft_consensus.cc:738] T f39376a63b224e06bac9089e6368a2a9 P f399d76f5690464c9d1497cdec0fd39c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f399d76f5690464c9d1497cdec0fd39c, State: Initialized, Role: FOLLOWER
I20250627 13:57:45.210390 5736 consensus_queue.cc:260] T f39376a63b224e06bac9089e6368a2a9 P 9f0cfc4965f1490bb71ad77dcc4131c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa2c2d51dd634f88bc3a29fd4ffb5925" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43889 } } peers { permanent_uuid: "9f0cfc4965f1490bb71ad77dcc4131c5" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40589 } } peers { permanent_uuid: "f399d76f5690464c9d1497cdec0fd39c" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 37231 } }
I20250627 13:57:45.210870 5735 consensus_queue.cc:260] T f39376a63b224e06bac9089e6368a2a9 P f399d76f5690464c9d1497cdec0fd39c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa2c2d51dd634f88bc3a29fd4ffb5925" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43889 } } peers { permanent_uuid: "9f0cfc4965f1490bb71ad77dcc4131c5" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40589 } } peers { permanent_uuid: "f399d76f5690464c9d1497cdec0fd39c" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 37231 } }
I20250627 13:57:45.216975 5734 ts_tablet_manager.cc:1428] T f39376a63b224e06bac9089e6368a2a9 P aa2c2d51dd634f88bc3a29fd4ffb5925: Time spent starting tablet: real 0.048s user 0.029s sys 0.010s
I20250627 13:57:45.233570 5715 heartbeater.cc:499] Master 127.3.173.126:41315 was elected leader, sending a full tablet report...
I20250627 13:57:45.234381 5736 ts_tablet_manager.cc:1428] T f39376a63b224e06bac9089e6368a2a9 P 9f0cfc4965f1490bb71ad77dcc4131c5: Time spent starting tablet: real 0.055s user 0.036s sys 0.013s
I20250627 13:57:45.236488 5735 ts_tablet_manager.cc:1428] T f39376a63b224e06bac9089e6368a2a9 P f399d76f5690464c9d1497cdec0fd39c: Time spent starting tablet: real 0.057s user 0.034s sys 0.019s
I20250627 13:57:45.237984 5742 raft_consensus.cc:491] T f39376a63b224e06bac9089e6368a2a9 P f399d76f5690464c9d1497cdec0fd39c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:57:45.238610 5742 raft_consensus.cc:513] T f39376a63b224e06bac9089e6368a2a9 P f399d76f5690464c9d1497cdec0fd39c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa2c2d51dd634f88bc3a29fd4ffb5925" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43889 } } peers { permanent_uuid: "9f0cfc4965f1490bb71ad77dcc4131c5" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40589 } } peers { permanent_uuid: "f399d76f5690464c9d1497cdec0fd39c" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 37231 } }
I20250627 13:57:45.241904 5742 leader_election.cc:290] T f39376a63b224e06bac9089e6368a2a9 P f399d76f5690464c9d1497cdec0fd39c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa2c2d51dd634f88bc3a29fd4ffb5925 (127.3.173.65:43889), 9f0cfc4965f1490bb71ad77dcc4131c5 (127.3.173.67:40589)
W20250627 13:57:45.244482 5716 tablet.cc:2378] T f39376a63b224e06bac9089e6368a2a9 P 9f0cfc4965f1490bb71ad77dcc4131c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250627 13:57:45.253873 5403 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f39376a63b224e06bac9089e6368a2a9" candidate_uuid: "f399d76f5690464c9d1497cdec0fd39c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa2c2d51dd634f88bc3a29fd4ffb5925" is_pre_election: true
I20250627 13:57:45.254659 5403 raft_consensus.cc:2466] T f39376a63b224e06bac9089e6368a2a9 P aa2c2d51dd634f88bc3a29fd4ffb5925 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f399d76f5690464c9d1497cdec0fd39c in term 0.
I20250627 13:57:45.256044 5470 leader_election.cc:304] T f39376a63b224e06bac9089e6368a2a9 P f399d76f5690464c9d1497cdec0fd39c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa2c2d51dd634f88bc3a29fd4ffb5925, f399d76f5690464c9d1497cdec0fd39c; no voters:
I20250627 13:57:45.257052 5742 raft_consensus.cc:2802] T f39376a63b224e06bac9089e6368a2a9 P f399d76f5690464c9d1497cdec0fd39c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:57:45.257401 5742 raft_consensus.cc:491] T f39376a63b224e06bac9089e6368a2a9 P f399d76f5690464c9d1497cdec0fd39c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:57:45.257712 5742 raft_consensus.cc:3058] T f39376a63b224e06bac9089e6368a2a9 P f399d76f5690464c9d1497cdec0fd39c [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:57:45.259191 5670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f39376a63b224e06bac9089e6368a2a9" candidate_uuid: "f399d76f5690464c9d1497cdec0fd39c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f0cfc4965f1490bb71ad77dcc4131c5" is_pre_election: true
I20250627 13:57:45.259858 5670 raft_consensus.cc:2466] T f39376a63b224e06bac9089e6368a2a9 P 9f0cfc4965f1490bb71ad77dcc4131c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f399d76f5690464c9d1497cdec0fd39c in term 0.
I20250627 13:57:45.262753 5742 raft_consensus.cc:513] T f39376a63b224e06bac9089e6368a2a9 P f399d76f5690464c9d1497cdec0fd39c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa2c2d51dd634f88bc3a29fd4ffb5925" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43889 } } peers { permanent_uuid: "9f0cfc4965f1490bb71ad77dcc4131c5" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40589 } } peers { permanent_uuid: "f399d76f5690464c9d1497cdec0fd39c" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 37231 } }
I20250627 13:57:45.264099 5742 leader_election.cc:290] T f39376a63b224e06bac9089e6368a2a9 P f399d76f5690464c9d1497cdec0fd39c [CANDIDATE]: Term 1 election: Requested vote from peers aa2c2d51dd634f88bc3a29fd4ffb5925 (127.3.173.65:43889), 9f0cfc4965f1490bb71ad77dcc4131c5 (127.3.173.67:40589)
I20250627 13:57:45.264765 5403 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f39376a63b224e06bac9089e6368a2a9" candidate_uuid: "f399d76f5690464c9d1497cdec0fd39c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa2c2d51dd634f88bc3a29fd4ffb5925"
I20250627 13:57:45.265121 5670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f39376a63b224e06bac9089e6368a2a9" candidate_uuid: "f399d76f5690464c9d1497cdec0fd39c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f0cfc4965f1490bb71ad77dcc4131c5"
I20250627 13:57:45.265275 5403 raft_consensus.cc:3058] T f39376a63b224e06bac9089e6368a2a9 P aa2c2d51dd634f88bc3a29fd4ffb5925 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:57:45.265597 5670 raft_consensus.cc:3058] T f39376a63b224e06bac9089e6368a2a9 P 9f0cfc4965f1490bb71ad77dcc4131c5 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:57:45.269611 5403 raft_consensus.cc:2466] T f39376a63b224e06bac9089e6368a2a9 P aa2c2d51dd634f88bc3a29fd4ffb5925 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f399d76f5690464c9d1497cdec0fd39c in term 1.
I20250627 13:57:45.269780 5670 raft_consensus.cc:2466] T f39376a63b224e06bac9089e6368a2a9 P 9f0cfc4965f1490bb71ad77dcc4131c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f399d76f5690464c9d1497cdec0fd39c in term 1.
I20250627 13:57:45.270689 5470 leader_election.cc:304] T f39376a63b224e06bac9089e6368a2a9 P f399d76f5690464c9d1497cdec0fd39c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa2c2d51dd634f88bc3a29fd4ffb5925, f399d76f5690464c9d1497cdec0fd39c; no voters:
I20250627 13:57:45.271320 5742 raft_consensus.cc:2802] T f39376a63b224e06bac9089e6368a2a9 P f399d76f5690464c9d1497cdec0fd39c [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:57:45.272806 5742 raft_consensus.cc:695] T f39376a63b224e06bac9089e6368a2a9 P f399d76f5690464c9d1497cdec0fd39c [term 1 LEADER]: Becoming Leader. State: Replica: f399d76f5690464c9d1497cdec0fd39c, State: Running, Role: LEADER
I20250627 13:57:45.273634 5742 consensus_queue.cc:237] T f39376a63b224e06bac9089e6368a2a9 P f399d76f5690464c9d1497cdec0fd39c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa2c2d51dd634f88bc3a29fd4ffb5925" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43889 } } peers { permanent_uuid: "9f0cfc4965f1490bb71ad77dcc4131c5" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40589 } } peers { permanent_uuid: "f399d76f5690464c9d1497cdec0fd39c" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 37231 } }
I20250627 13:57:45.283979 5259 catalog_manager.cc:5582] T f39376a63b224e06bac9089e6368a2a9 P f399d76f5690464c9d1497cdec0fd39c reported cstate change: term changed from 0 to 1, leader changed from <none> to f399d76f5690464c9d1497cdec0fd39c (127.3.173.66). New cstate: current_term: 1 leader_uuid: "f399d76f5690464c9d1497cdec0fd39c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa2c2d51dd634f88bc3a29fd4ffb5925" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f0cfc4965f1490bb71ad77dcc4131c5" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f399d76f5690464c9d1497cdec0fd39c" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 37231 } health_report { overall_health: HEALTHY } } }
I20250627 13:57:45.334060 3765 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250627 13:57:45.337208 3765 ts_itest-base.cc:246] Waiting for 1 tablets on tserver aa2c2d51dd634f88bc3a29fd4ffb5925 to finish bootstrapping
I20250627 13:57:45.349469 3765 ts_itest-base.cc:246] Waiting for 1 tablets on tserver f399d76f5690464c9d1497cdec0fd39c to finish bootstrapping
I20250627 13:57:45.360543 3765 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 9f0cfc4965f1490bb71ad77dcc4131c5 to finish bootstrapping
I20250627 13:57:45.374001 5259 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59346:
name: "TestAnotherTable"
schema {
columns {
name: "foo"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "bar"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
comment: "comment for bar"
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "foo"
}
}
}
W20250627 13:57:45.375582 5259 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAnotherTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250627 13:57:45.394032 5449 tablet.cc:2378] T f39376a63b224e06bac9089e6368a2a9 P aa2c2d51dd634f88bc3a29fd4ffb5925: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250627 13:57:45.394834 5383 tablet_service.cc:1468] Processing CreateTablet for tablet b1e3a793a8404d11b9abc3d2bf1fce2c (DEFAULT_TABLE table=TestAnotherTable [id=6764adda9b4a46119f209ed0810b1f9a]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250627 13:57:45.395963 5383 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1e3a793a8404d11b9abc3d2bf1fce2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:57:45.397039 5516 tablet_service.cc:1468] Processing CreateTablet for tablet b1e3a793a8404d11b9abc3d2bf1fce2c (DEFAULT_TABLE table=TestAnotherTable [id=6764adda9b4a46119f209ed0810b1f9a]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250627 13:57:45.398092 5516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1e3a793a8404d11b9abc3d2bf1fce2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:57:45.398219 5650 tablet_service.cc:1468] Processing CreateTablet for tablet b1e3a793a8404d11b9abc3d2bf1fce2c (DEFAULT_TABLE table=TestAnotherTable [id=6764adda9b4a46119f209ed0810b1f9a]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250627 13:57:45.399250 5650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1e3a793a8404d11b9abc3d2bf1fce2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:57:45.412019 5734 tablet_bootstrap.cc:492] T b1e3a793a8404d11b9abc3d2bf1fce2c P aa2c2d51dd634f88bc3a29fd4ffb5925: Bootstrap starting.
I20250627 13:57:45.415814 5736 tablet_bootstrap.cc:492] T b1e3a793a8404d11b9abc3d2bf1fce2c P 9f0cfc4965f1490bb71ad77dcc4131c5: Bootstrap starting.
I20250627 13:57:45.417878 5734 tablet_bootstrap.cc:654] T b1e3a793a8404d11b9abc3d2bf1fce2c P aa2c2d51dd634f88bc3a29fd4ffb5925: Neither blocks nor log segments found. Creating new log.
I20250627 13:57:45.421859 5735 tablet_bootstrap.cc:492] T b1e3a793a8404d11b9abc3d2bf1fce2c P f399d76f5690464c9d1497cdec0fd39c: Bootstrap starting.
I20250627 13:57:45.424638 5736 tablet_bootstrap.cc:654] T b1e3a793a8404d11b9abc3d2bf1fce2c P 9f0cfc4965f1490bb71ad77dcc4131c5: Neither blocks nor log segments found. Creating new log.
I20250627 13:57:45.428911 5735 tablet_bootstrap.cc:654] T b1e3a793a8404d11b9abc3d2bf1fce2c P f399d76f5690464c9d1497cdec0fd39c: Neither blocks nor log segments found. Creating new log.
I20250627 13:57:45.429636 5734 tablet_bootstrap.cc:492] T b1e3a793a8404d11b9abc3d2bf1fce2c P aa2c2d51dd634f88bc3a29fd4ffb5925: No bootstrap required, opened a new log
I20250627 13:57:45.430168 5734 ts_tablet_manager.cc:1397] T b1e3a793a8404d11b9abc3d2bf1fce2c P aa2c2d51dd634f88bc3a29fd4ffb5925: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.012s
I20250627 13:57:45.433282 5734 raft_consensus.cc:357] T b1e3a793a8404d11b9abc3d2bf1fce2c P aa2c2d51dd634f88bc3a29fd4ffb5925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa2c2d51dd634f88bc3a29fd4ffb5925" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43889 } } peers { permanent_uuid: "f399d76f5690464c9d1497cdec0fd39c" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 37231 } } peers { permanent_uuid: "9f0cfc4965f1490bb71ad77dcc4131c5" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40589 } }
I20250627 13:57:45.434010 5734 raft_consensus.cc:383] T b1e3a793a8404d11b9abc3d2bf1fce2c P aa2c2d51dd634f88bc3a29fd4ffb5925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:57:45.434348 5734 raft_consensus.cc:738] T b1e3a793a8404d11b9abc3d2bf1fce2c P aa2c2d51dd634f88bc3a29fd4ffb5925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa2c2d51dd634f88bc3a29fd4ffb5925, State: Initialized, Role: FOLLOWER
I20250627 13:57:45.435250 5734 consensus_queue.cc:260] T b1e3a793a8404d11b9abc3d2bf1fce2c P aa2c2d51dd634f88bc3a29fd4ffb5925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa2c2d51dd634f88bc3a29fd4ffb5925" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43889 } } peers { permanent_uuid: "f399d76f5690464c9d1497cdec0fd39c" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 37231 } } peers { permanent_uuid: "9f0cfc4965f1490bb71ad77dcc4131c5" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40589 } }
I20250627 13:57:45.437625 5734 ts_tablet_manager.cc:1428] T b1e3a793a8404d11b9abc3d2bf1fce2c P aa2c2d51dd634f88bc3a29fd4ffb5925: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
W20250627 13:57:45.437716 5582 tablet.cc:2378] T f39376a63b224e06bac9089e6368a2a9 P f399d76f5690464c9d1497cdec0fd39c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250627 13:57:45.449646 5736 tablet_bootstrap.cc:492] T b1e3a793a8404d11b9abc3d2bf1fce2c P 9f0cfc4965f1490bb71ad77dcc4131c5: No bootstrap required, opened a new log
I20250627 13:57:45.450071 5736 ts_tablet_manager.cc:1397] T b1e3a793a8404d11b9abc3d2bf1fce2c P 9f0cfc4965f1490bb71ad77dcc4131c5: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.004s
I20250627 13:57:45.452585 5736 raft_consensus.cc:357] T b1e3a793a8404d11b9abc3d2bf1fce2c P 9f0cfc4965f1490bb71ad77dcc4131c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa2c2d51dd634f88bc3a29fd4ffb5925" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43889 } } peers { permanent_uuid: "f399d76f5690464c9d1497cdec0fd39c" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 37231 } } peers { permanent_uuid: "9f0cfc4965f1490bb71ad77dcc4131c5" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40589 } }
I20250627 13:57:45.454244 5736 raft_consensus.cc:383] T b1e3a793a8404d11b9abc3d2bf1fce2c P 9f0cfc4965f1490bb71ad77dcc4131c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:57:45.454568 5736 raft_consensus.cc:738] T b1e3a793a8404d11b9abc3d2bf1fce2c P 9f0cfc4965f1490bb71ad77dcc4131c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f0cfc4965f1490bb71ad77dcc4131c5, State: Initialized, Role: FOLLOWER
I20250627 13:57:45.455247 5736 consensus_queue.cc:260] T b1e3a793a8404d11b9abc3d2bf1fce2c P 9f0cfc4965f1490bb71ad77dcc4131c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa2c2d51dd634f88bc3a29fd4ffb5925" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43889 } } peers { permanent_uuid: "f399d76f5690464c9d1497cdec0fd39c" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 37231 } } peers { permanent_uuid: "9f0cfc4965f1490bb71ad77dcc4131c5" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40589 } }
I20250627 13:57:45.455839 5735 tablet_bootstrap.cc:492] T b1e3a793a8404d11b9abc3d2bf1fce2c P f399d76f5690464c9d1497cdec0fd39c: No bootstrap required, opened a new log
I20250627 13:57:45.456287 5735 ts_tablet_manager.cc:1397] T b1e3a793a8404d11b9abc3d2bf1fce2c P f399d76f5690464c9d1497cdec0fd39c: Time spent bootstrapping tablet: real 0.035s user 0.016s sys 0.004s
I20250627 13:57:45.458783 5735 raft_consensus.cc:357] T b1e3a793a8404d11b9abc3d2bf1fce2c P f399d76f5690464c9d1497cdec0fd39c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa2c2d51dd634f88bc3a29fd4ffb5925" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43889 } } peers { permanent_uuid: "f399d76f5690464c9d1497cdec0fd39c" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 37231 } } peers { permanent_uuid: "9f0cfc4965f1490bb71ad77dcc4131c5" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40589 } }
I20250627 13:57:45.459468 5735 raft_consensus.cc:383] T b1e3a793a8404d11b9abc3d2bf1fce2c P f399d76f5690464c9d1497cdec0fd39c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:57:45.459772 5735 raft_consensus.cc:738] T b1e3a793a8404d11b9abc3d2bf1fce2c P f399d76f5690464c9d1497cdec0fd39c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f399d76f5690464c9d1497cdec0fd39c, State: Initialized, Role: FOLLOWER
I20250627 13:57:45.460489 5735 consensus_queue.cc:260] T b1e3a793a8404d11b9abc3d2bf1fce2c P f399d76f5690464c9d1497cdec0fd39c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa2c2d51dd634f88bc3a29fd4ffb5925" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43889 } } peers { permanent_uuid: "f399d76f5690464c9d1497cdec0fd39c" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 37231 } } peers { permanent_uuid: "9f0cfc4965f1490bb71ad77dcc4131c5" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40589 } }
I20250627 13:57:45.462558 5736 ts_tablet_manager.cc:1428] T b1e3a793a8404d11b9abc3d2bf1fce2c P 9f0cfc4965f1490bb71ad77dcc4131c5: Time spent starting tablet: real 0.012s user 0.003s sys 0.003s
I20250627 13:57:45.464833 5735 ts_tablet_manager.cc:1428] T b1e3a793a8404d11b9abc3d2bf1fce2c P f399d76f5690464c9d1497cdec0fd39c: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20250627 13:57:45.481655 5742 raft_consensus.cc:491] T b1e3a793a8404d11b9abc3d2bf1fce2c P f399d76f5690464c9d1497cdec0fd39c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:57:45.482033 5742 raft_consensus.cc:513] T b1e3a793a8404d11b9abc3d2bf1fce2c P f399d76f5690464c9d1497cdec0fd39c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa2c2d51dd634f88bc3a29fd4ffb5925" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43889 } } peers { permanent_uuid: "f399d76f5690464c9d1497cdec0fd39c" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 37231 } } peers { permanent_uuid: "9f0cfc4965f1490bb71ad77dcc4131c5" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40589 } }
I20250627 13:57:45.483534 5742 leader_election.cc:290] T b1e3a793a8404d11b9abc3d2bf1fce2c P f399d76f5690464c9d1497cdec0fd39c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa2c2d51dd634f88bc3a29fd4ffb5925 (127.3.173.65:43889), 9f0cfc4965f1490bb71ad77dcc4131c5 (127.3.173.67:40589)
I20250627 13:57:45.484263 5403 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1e3a793a8404d11b9abc3d2bf1fce2c" candidate_uuid: "f399d76f5690464c9d1497cdec0fd39c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa2c2d51dd634f88bc3a29fd4ffb5925" is_pre_election: true
I20250627 13:57:45.484586 5670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1e3a793a8404d11b9abc3d2bf1fce2c" candidate_uuid: "f399d76f5690464c9d1497cdec0fd39c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f0cfc4965f1490bb71ad77dcc4131c5" is_pre_election: true
I20250627 13:57:45.484776 5403 raft_consensus.cc:2466] T b1e3a793a8404d11b9abc3d2bf1fce2c P aa2c2d51dd634f88bc3a29fd4ffb5925 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f399d76f5690464c9d1497cdec0fd39c in term 0.
I20250627 13:57:45.485056 5670 raft_consensus.cc:2466] T b1e3a793a8404d11b9abc3d2bf1fce2c P 9f0cfc4965f1490bb71ad77dcc4131c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f399d76f5690464c9d1497cdec0fd39c in term 0.
I20250627 13:57:45.485664 5470 leader_election.cc:304] T b1e3a793a8404d11b9abc3d2bf1fce2c P f399d76f5690464c9d1497cdec0fd39c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa2c2d51dd634f88bc3a29fd4ffb5925, f399d76f5690464c9d1497cdec0fd39c; no voters:
I20250627 13:57:45.486294 5742 raft_consensus.cc:2802] T b1e3a793a8404d11b9abc3d2bf1fce2c P f399d76f5690464c9d1497cdec0fd39c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:57:45.486531 5742 raft_consensus.cc:491] T b1e3a793a8404d11b9abc3d2bf1fce2c P f399d76f5690464c9d1497cdec0fd39c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:57:45.486768 5742 raft_consensus.cc:3058] T b1e3a793a8404d11b9abc3d2bf1fce2c P f399d76f5690464c9d1497cdec0fd39c [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:57:45.491046 5742 raft_consensus.cc:513] T b1e3a793a8404d11b9abc3d2bf1fce2c P f399d76f5690464c9d1497cdec0fd39c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa2c2d51dd634f88bc3a29fd4ffb5925" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43889 } } peers { permanent_uuid: "f399d76f5690464c9d1497cdec0fd39c" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 37231 } } peers { permanent_uuid: "9f0cfc4965f1490bb71ad77dcc4131c5" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40589 } }
I20250627 13:57:45.492331 5742 leader_election.cc:290] T b1e3a793a8404d11b9abc3d2bf1fce2c P f399d76f5690464c9d1497cdec0fd39c [CANDIDATE]: Term 1 election: Requested vote from peers aa2c2d51dd634f88bc3a29fd4ffb5925 (127.3.173.65:43889), 9f0cfc4965f1490bb71ad77dcc4131c5 (127.3.173.67:40589)
I20250627 13:57:45.493039 5403 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1e3a793a8404d11b9abc3d2bf1fce2c" candidate_uuid: "f399d76f5690464c9d1497cdec0fd39c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa2c2d51dd634f88bc3a29fd4ffb5925"
I20250627 13:57:45.493240 5670 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1e3a793a8404d11b9abc3d2bf1fce2c" candidate_uuid: "f399d76f5690464c9d1497cdec0fd39c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f0cfc4965f1490bb71ad77dcc4131c5"
I20250627 13:57:45.493424 5403 raft_consensus.cc:3058] T b1e3a793a8404d11b9abc3d2bf1fce2c P aa2c2d51dd634f88bc3a29fd4ffb5925 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:57:45.493662 5670 raft_consensus.cc:3058] T b1e3a793a8404d11b9abc3d2bf1fce2c P 9f0cfc4965f1490bb71ad77dcc4131c5 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:57:45.497679 5403 raft_consensus.cc:2466] T b1e3a793a8404d11b9abc3d2bf1fce2c P aa2c2d51dd634f88bc3a29fd4ffb5925 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f399d76f5690464c9d1497cdec0fd39c in term 1.
I20250627 13:57:45.498014 5670 raft_consensus.cc:2466] T b1e3a793a8404d11b9abc3d2bf1fce2c P 9f0cfc4965f1490bb71ad77dcc4131c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f399d76f5690464c9d1497cdec0fd39c in term 1.
I20250627 13:57:45.498559 5470 leader_election.cc:304] T b1e3a793a8404d11b9abc3d2bf1fce2c P f399d76f5690464c9d1497cdec0fd39c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa2c2d51dd634f88bc3a29fd4ffb5925, f399d76f5690464c9d1497cdec0fd39c; no voters:
I20250627 13:57:45.499100 5742 raft_consensus.cc:2802] T b1e3a793a8404d11b9abc3d2bf1fce2c P f399d76f5690464c9d1497cdec0fd39c [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:57:45.499454 5742 raft_consensus.cc:695] T b1e3a793a8404d11b9abc3d2bf1fce2c P f399d76f5690464c9d1497cdec0fd39c [term 1 LEADER]: Becoming Leader. State: Replica: f399d76f5690464c9d1497cdec0fd39c, State: Running, Role: LEADER
I20250627 13:57:45.499977 5742 consensus_queue.cc:237] T b1e3a793a8404d11b9abc3d2bf1fce2c P f399d76f5690464c9d1497cdec0fd39c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa2c2d51dd634f88bc3a29fd4ffb5925" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43889 } } peers { permanent_uuid: "f399d76f5690464c9d1497cdec0fd39c" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 37231 } } peers { permanent_uuid: "9f0cfc4965f1490bb71ad77dcc4131c5" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40589 } }
I20250627 13:57:45.506783 5258 catalog_manager.cc:5582] T b1e3a793a8404d11b9abc3d2bf1fce2c P f399d76f5690464c9d1497cdec0fd39c reported cstate change: term changed from 0 to 1, leader changed from <none> to f399d76f5690464c9d1497cdec0fd39c (127.3.173.66). New cstate: current_term: 1 leader_uuid: "f399d76f5690464c9d1497cdec0fd39c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa2c2d51dd634f88bc3a29fd4ffb5925" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f399d76f5690464c9d1497cdec0fd39c" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 37231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f0cfc4965f1490bb71ad77dcc4131c5" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40589 } health_report { overall_health: UNKNOWN } } }
I20250627 13:57:45.744477 5742 consensus_queue.cc:1035] T f39376a63b224e06bac9089e6368a2a9 P f399d76f5690464c9d1497cdec0fd39c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f0cfc4965f1490bb71ad77dcc4131c5" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:57:45.765863 5756 consensus_queue.cc:1035] T f39376a63b224e06bac9089e6368a2a9 P f399d76f5690464c9d1497cdec0fd39c [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa2c2d51dd634f88bc3a29fd4ffb5925" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250627 13:57:45.830586 5752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:57:45.831207 5752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:57:45.862802 5752 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250627 13:57:46.005064 5756 consensus_queue.cc:1035] T b1e3a793a8404d11b9abc3d2bf1fce2c P f399d76f5690464c9d1497cdec0fd39c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f0cfc4965f1490bb71ad77dcc4131c5" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250627 13:57:46.048763 5745 consensus_queue.cc:1035] T b1e3a793a8404d11b9abc3d2bf1fce2c P f399d76f5690464c9d1497cdec0fd39c [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa2c2d51dd634f88bc3a29fd4ffb5925" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250627 13:57:47.188318 5752 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.283s user 0.514s sys 0.767s
W20250627 13:57:47.188695 5752 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.284s user 0.514s sys 0.767s
W20250627 13:57:48.549541 5774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:57:48.550150 5774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:57:48.579896 5774 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250627 13:57:49.850049 5774 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.229s user 0.428s sys 0.799s
W20250627 13:57:49.850440 5774 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.229s user 0.428s sys 0.802s
W20250627 13:57:51.236726 5789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:57:51.237274 5789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:57:51.267710 5789 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250627 13:57:52.612082 5789 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.302s user 0.004s sys 0.000s
W20250627 13:57:52.612355 5789 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.303s user 0.004s sys 0.000s
W20250627 13:57:53.969347 5804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:57:53.969950 5804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:57:54.000669 5804 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250627 13:57:55.258261 5804 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.215s user 0.465s sys 0.747s
W20250627 13:57:55.258555 5804 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.216s user 0.465s sys 0.747s
I20250627 13:57:56.325961 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 5319
I20250627 13:57:56.351583 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 5452
I20250627 13:57:56.377121 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 5585
I20250627 13:57:56.403441 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 5227
2025-06-27T13:57:56Z chronyd exiting
[ OK ] AdminCliTest.TestDescribeTableColumnFlags (18789 ms)
[ RUN ] AdminCliTest.TestAuthzResetCacheNotAuthorized
I20250627 13:57:56.458606 3765 test_util.cc:276] Using random seed: 279683156
I20250627 13:57:56.462782 3765 ts_itest-base.cc:115] Starting cluster with:
I20250627 13:57:56.462944 3765 ts_itest-base.cc:116] --------------
I20250627 13:57:56.463119 3765 ts_itest-base.cc:117] 3 tablet servers
I20250627 13:57:56.463271 3765 ts_itest-base.cc:118] 3 replicas per TS
I20250627 13:57:56.463411 3765 ts_itest-base.cc:119] --------------
2025-06-27T13:57:56Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-27T13:57:56Z Disabled control of system clock
I20250627 13:57:56.498910 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.173.126:36171
--webserver_interface=127.3.173.126
--webserver_port=0
--builtin_ntp_servers=127.3.173.84:42795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.173.126:36171
--superuser_acl=no-such-user with env {}
W20250627 13:57:56.788642 5826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:57:56.789194 5826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:57:56.789628 5826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:57:56.820317 5826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250627 13:57:56.820669 5826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:57:56.820930 5826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250627 13:57:56.821170 5826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250627 13:57:56.856277 5826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:42795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.173.126:36171
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.173.126:36171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.3.173.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:57:56.857746 5826 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:57:56.859520 5826 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:57:56.875375 5832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:57:56.875432 5835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:57:56.875536 5833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:57:56.876899 5826 server_base.cc:1048] running on GCE node
I20250627 13:57:58.054447 5826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:57:58.057459 5826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:57:58.058867 5826 hybrid_clock.cc:648] HybridClock initialized: now 1751032678058833 us; error 57 us; skew 500 ppm
I20250627 13:57:58.059747 5826 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:57:58.065665 5826 webserver.cc:469] Webserver started at http://127.3.173.126:36909/ using document root <none> and password file <none>
I20250627 13:57:58.066649 5826 fs_manager.cc:362] Metadata directory not provided
I20250627 13:57:58.066838 5826 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:57:58.067297 5826 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:57:58.071565 5826 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "ac62f515150a4ea58e47c02baaa2cfbe"
format_stamp: "Formatted at 2025-06-27 13:57:58 on dist-test-slave-rn80"
I20250627 13:57:58.072626 5826 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "ac62f515150a4ea58e47c02baaa2cfbe"
format_stamp: "Formatted at 2025-06-27 13:57:58 on dist-test-slave-rn80"
I20250627 13:57:58.079519 5826 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20250627 13:57:58.084662 5842 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:58.085660 5826 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250627 13:57:58.085961 5826 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ac62f515150a4ea58e47c02baaa2cfbe"
format_stamp: "Formatted at 2025-06-27 13:57:58 on dist-test-slave-rn80"
I20250627 13:57:58.086308 5826 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:57:58.134949 5826 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:57:58.136387 5826 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:57:58.136833 5826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:57:58.204906 5826 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.126:36171
I20250627 13:57:58.205054 5893 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.126:36171 every 8 connection(s)
I20250627 13:57:58.207541 5826 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250627 13:57:58.212401 5894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:57:58.215664 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 5826
I20250627 13:57:58.216035 3765 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250627 13:57:58.246397 5894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac62f515150a4ea58e47c02baaa2cfbe: Bootstrap starting.
I20250627 13:57:58.251991 5894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac62f515150a4ea58e47c02baaa2cfbe: Neither blocks nor log segments found. Creating new log.
I20250627 13:57:58.253676 5894 log.cc:826] T 00000000000000000000000000000000 P ac62f515150a4ea58e47c02baaa2cfbe: Log is configured to *not* fsync() on all Append() calls
I20250627 13:57:58.257880 5894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac62f515150a4ea58e47c02baaa2cfbe: No bootstrap required, opened a new log
I20250627 13:57:58.274430 5894 raft_consensus.cc:357] T 00000000000000000000000000000000 P ac62f515150a4ea58e47c02baaa2cfbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac62f515150a4ea58e47c02baaa2cfbe" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 36171 } }
I20250627 13:57:58.275089 5894 raft_consensus.cc:383] T 00000000000000000000000000000000 P ac62f515150a4ea58e47c02baaa2cfbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:57:58.275326 5894 raft_consensus.cc:738] T 00000000000000000000000000000000 P ac62f515150a4ea58e47c02baaa2cfbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac62f515150a4ea58e47c02baaa2cfbe, State: Initialized, Role: FOLLOWER
I20250627 13:57:58.275959 5894 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac62f515150a4ea58e47c02baaa2cfbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac62f515150a4ea58e47c02baaa2cfbe" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 36171 } }
I20250627 13:57:58.276469 5894 raft_consensus.cc:397] T 00000000000000000000000000000000 P ac62f515150a4ea58e47c02baaa2cfbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250627 13:57:58.276721 5894 raft_consensus.cc:491] T 00000000000000000000000000000000 P ac62f515150a4ea58e47c02baaa2cfbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250627 13:57:58.277037 5894 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ac62f515150a4ea58e47c02baaa2cfbe [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:57:58.281055 5894 raft_consensus.cc:513] T 00000000000000000000000000000000 P ac62f515150a4ea58e47c02baaa2cfbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac62f515150a4ea58e47c02baaa2cfbe" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 36171 } }
I20250627 13:57:58.281742 5894 leader_election.cc:304] T 00000000000000000000000000000000 P ac62f515150a4ea58e47c02baaa2cfbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac62f515150a4ea58e47c02baaa2cfbe; no voters:
I20250627 13:57:58.283432 5894 leader_election.cc:290] T 00000000000000000000000000000000 P ac62f515150a4ea58e47c02baaa2cfbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20250627 13:57:58.284106 5899 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ac62f515150a4ea58e47c02baaa2cfbe [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:57:58.286291 5899 raft_consensus.cc:695] T 00000000000000000000000000000000 P ac62f515150a4ea58e47c02baaa2cfbe [term 1 LEADER]: Becoming Leader. State: Replica: ac62f515150a4ea58e47c02baaa2cfbe, State: Running, Role: LEADER
I20250627 13:57:58.286967 5899 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac62f515150a4ea58e47c02baaa2cfbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac62f515150a4ea58e47c02baaa2cfbe" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 36171 } }
I20250627 13:57:58.287413 5894 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac62f515150a4ea58e47c02baaa2cfbe [sys.catalog]: configured and running, proceeding with master startup.
I20250627 13:57:58.297806 5900 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac62f515150a4ea58e47c02baaa2cfbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac62f515150a4ea58e47c02baaa2cfbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac62f515150a4ea58e47c02baaa2cfbe" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 36171 } } }
I20250627 13:57:58.298743 5900 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac62f515150a4ea58e47c02baaa2cfbe [sys.catalog]: This master's current role is: LEADER
I20250627 13:57:58.298044 5901 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac62f515150a4ea58e47c02baaa2cfbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac62f515150a4ea58e47c02baaa2cfbe. Latest consensus state: current_term: 1 leader_uuid: "ac62f515150a4ea58e47c02baaa2cfbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac62f515150a4ea58e47c02baaa2cfbe" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 36171 } } }
I20250627 13:57:58.299234 5901 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac62f515150a4ea58e47c02baaa2cfbe [sys.catalog]: This master's current role is: LEADER
I20250627 13:57:58.303913 5908 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250627 13:57:58.314527 5908 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250627 13:57:58.330216 5908 catalog_manager.cc:1349] Generated new cluster ID: 45cd3566a018425c855e293ab068ad86
I20250627 13:57:58.330583 5908 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250627 13:57:58.360466 5908 catalog_manager.cc:1372] Generated new certificate authority record
I20250627 13:57:58.362483 5908 catalog_manager.cc:1506] Loading token signing keys...
I20250627 13:57:58.384712 5908 catalog_manager.cc:5955] T 00000000000000000000000000000000 P ac62f515150a4ea58e47c02baaa2cfbe: Generated new TSK 0
I20250627 13:57:58.385776 5908 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250627 13:57:58.409638 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.173.65:0
--local_ip_for_outbound_sockets=127.3.173.65
--webserver_interface=127.3.173.65
--webserver_port=0
--tserver_master_addrs=127.3.173.126:36171
--builtin_ntp_servers=127.3.173.84:42795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250627 13:57:58.704504 5918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:57:58.705001 5918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:57:58.705564 5918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:57:58.735937 5918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:57:58.736743 5918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.173.65
I20250627 13:57:58.772333 5918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:42795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.173.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.3.173.65
--webserver_port=0
--tserver_master_addrs=127.3.173.126:36171
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.173.65
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:57:58.773572 5918 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:57:58.775135 5918 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:57:58.792471 5924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:57:58.792631 5925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:57:58.793326 5927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:57:59.941679 5926 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250627 13:57:59.941694 5918 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250627 13:57:59.946198 5918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:57:59.948836 5918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:57:59.950212 5918 hybrid_clock.cc:648] HybridClock initialized: now 1751032679950158 us; error 72 us; skew 500 ppm
I20250627 13:57:59.950992 5918 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:57:59.957937 5918 webserver.cc:469] Webserver started at http://127.3.173.65:32873/ using document root <none> and password file <none>
I20250627 13:57:59.958873 5918 fs_manager.cc:362] Metadata directory not provided
I20250627 13:57:59.959077 5918 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:57:59.959509 5918 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:57:59.963815 5918 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "3f3021fd38f24d55a271c09a0a557cc7"
format_stamp: "Formatted at 2025-06-27 13:57:59 on dist-test-slave-rn80"
I20250627 13:57:59.964843 5918 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "3f3021fd38f24d55a271c09a0a557cc7"
format_stamp: "Formatted at 2025-06-27 13:57:59 on dist-test-slave-rn80"
I20250627 13:57:59.972656 5918 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20250627 13:57:59.978739 5934 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:59.979875 5918 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250627 13:57:59.980190 5918 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "3f3021fd38f24d55a271c09a0a557cc7"
format_stamp: "Formatted at 2025-06-27 13:57:59 on dist-test-slave-rn80"
I20250627 13:57:59.980510 5918 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:58:00.034559 5918 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:58:00.035982 5918 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:58:00.036396 5918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:58:00.038880 5918 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:58:00.043326 5918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:58:00.043552 5918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:58:00.043797 5918 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:58:00.043963 5918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:58:00.174916 5918 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.65:45801
I20250627 13:58:00.175017 6046 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.65:45801 every 8 connection(s)
I20250627 13:58:00.177421 5918 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250627 13:58:00.180164 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 5918
I20250627 13:58:00.180673 3765 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250627 13:58:00.187121 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.173.66:0
--local_ip_for_outbound_sockets=127.3.173.66
--webserver_interface=127.3.173.66
--webserver_port=0
--tserver_master_addrs=127.3.173.126:36171
--builtin_ntp_servers=127.3.173.84:42795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250627 13:58:00.198382 6047 heartbeater.cc:344] Connected to a master server at 127.3.173.126:36171
I20250627 13:58:00.198809 6047 heartbeater.cc:461] Registering TS with master...
I20250627 13:58:00.199823 6047 heartbeater.cc:507] Master 127.3.173.126:36171 requested a full tablet report, sending...
I20250627 13:58:00.202690 5859 ts_manager.cc:194] Registered new tserver with Master: 3f3021fd38f24d55a271c09a0a557cc7 (127.3.173.65:45801)
I20250627 13:58:00.205749 5859 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.173.65:41913
W20250627 13:58:00.481621 6051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:58:00.482084 6051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:58:00.482597 6051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:58:00.514241 6051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:58:00.515040 6051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.173.66
I20250627 13:58:00.549024 6051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:42795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.173.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.3.173.66
--webserver_port=0
--tserver_master_addrs=127.3.173.126:36171
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.173.66
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:58:00.550354 6051 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:58:00.551892 6051 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:58:00.567365 6058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:58:01.210160 6047 heartbeater.cc:499] Master 127.3.173.126:36171 was elected leader, sending a full tablet report...
W20250627 13:58:00.567354 6060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:58:00.569669 6051 server_base.cc:1048] running on GCE node
W20250627 13:58:00.567952 6057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:58:01.711597 6051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:58:01.714383 6051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:58:01.715832 6051 hybrid_clock.cc:648] HybridClock initialized: now 1751032681715787 us; error 59 us; skew 500 ppm
I20250627 13:58:01.716619 6051 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:58:01.726449 6051 webserver.cc:469] Webserver started at http://127.3.173.66:36453/ using document root <none> and password file <none>
I20250627 13:58:01.727339 6051 fs_manager.cc:362] Metadata directory not provided
I20250627 13:58:01.727526 6051 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:58:01.727949 6051 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:58:01.732208 6051 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "6d7f78b0d3704e70903eb39e97f23042"
format_stamp: "Formatted at 2025-06-27 13:58:01 on dist-test-slave-rn80"
I20250627 13:58:01.733260 6051 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "6d7f78b0d3704e70903eb39e97f23042"
format_stamp: "Formatted at 2025-06-27 13:58:01 on dist-test-slave-rn80"
I20250627 13:58:01.740211 6051 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20250627 13:58:01.745676 6067 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:58:01.746675 6051 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250627 13:58:01.746991 6051 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "6d7f78b0d3704e70903eb39e97f23042"
format_stamp: "Formatted at 2025-06-27 13:58:01 on dist-test-slave-rn80"
I20250627 13:58:01.747318 6051 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:58:01.799854 6051 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:58:01.801257 6051 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:58:01.801669 6051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:58:01.804096 6051 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:58:01.808073 6051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:58:01.808279 6051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:58:01.808507 6051 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:58:01.808681 6051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:58:01.939844 6051 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.66:38405
I20250627 13:58:01.939947 6179 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.66:38405 every 8 connection(s)
I20250627 13:58:01.942378 6051 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250627 13:58:01.949906 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 6051
I20250627 13:58:01.950331 3765 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250627 13:58:01.955969 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.173.67:0
--local_ip_for_outbound_sockets=127.3.173.67
--webserver_interface=127.3.173.67
--webserver_port=0
--tserver_master_addrs=127.3.173.126:36171
--builtin_ntp_servers=127.3.173.84:42795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250627 13:58:01.963915 6180 heartbeater.cc:344] Connected to a master server at 127.3.173.126:36171
I20250627 13:58:01.964443 6180 heartbeater.cc:461] Registering TS with master...
I20250627 13:58:01.965816 6180 heartbeater.cc:507] Master 127.3.173.126:36171 requested a full tablet report, sending...
I20250627 13:58:01.968320 5859 ts_manager.cc:194] Registered new tserver with Master: 6d7f78b0d3704e70903eb39e97f23042 (127.3.173.66:38405)
I20250627 13:58:01.969540 5859 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.173.66:48483
W20250627 13:58:02.258622 6184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:58:02.259109 6184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:58:02.259598 6184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:58:02.289388 6184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:58:02.290266 6184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.173.67
I20250627 13:58:02.323722 6184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:42795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.173.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.3.173.67
--webserver_port=0
--tserver_master_addrs=127.3.173.126:36171
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.173.67
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:58:02.324944 6184 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:58:02.326545 6184 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:58:02.340180 6190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:58:02.973367 6180 heartbeater.cc:499] Master 127.3.173.126:36171 was elected leader, sending a full tablet report...
W20250627 13:58:02.342875 6193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:58:02.340291 6191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:58:03.502414 6192 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1160 milliseconds
I20250627 13:58:03.502521 6184 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250627 13:58:03.503680 6184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:58:03.506402 6184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:58:03.507828 6184 hybrid_clock.cc:648] HybridClock initialized: now 1751032683507793 us; error 57 us; skew 500 ppm
I20250627 13:58:03.508615 6184 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:58:03.519068 6184 webserver.cc:469] Webserver started at http://127.3.173.67:45653/ using document root <none> and password file <none>
I20250627 13:58:03.519963 6184 fs_manager.cc:362] Metadata directory not provided
I20250627 13:58:03.520169 6184 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:58:03.520614 6184 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:58:03.524950 6184 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "6302322f68c7453bb5f41861e1d469f7"
format_stamp: "Formatted at 2025-06-27 13:58:03 on dist-test-slave-rn80"
I20250627 13:58:03.525993 6184 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "6302322f68c7453bb5f41861e1d469f7"
format_stamp: "Formatted at 2025-06-27 13:58:03 on dist-test-slave-rn80"
I20250627 13:58:03.532912 6184 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250627 13:58:03.538075 6200 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:58:03.539093 6184 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250627 13:58:03.539395 6184 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "6302322f68c7453bb5f41861e1d469f7"
format_stamp: "Formatted at 2025-06-27 13:58:03 on dist-test-slave-rn80"
I20250627 13:58:03.539690 6184 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:58:03.611979 6184 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:58:03.613484 6184 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:58:03.613898 6184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:58:03.616349 6184 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:58:03.620456 6184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:58:03.620649 6184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:58:03.620901 6184 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:58:03.621062 6184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:58:03.753268 6184 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.67:34575
I20250627 13:58:03.753372 6312 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.67:34575 every 8 connection(s)
I20250627 13:58:03.755780 6184 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250627 13:58:03.757151 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 6184
I20250627 13:58:03.757699 3765 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250627 13:58:03.777032 6313 heartbeater.cc:344] Connected to a master server at 127.3.173.126:36171
I20250627 13:58:03.777465 6313 heartbeater.cc:461] Registering TS with master...
I20250627 13:58:03.778751 6313 heartbeater.cc:507] Master 127.3.173.126:36171 requested a full tablet report, sending...
I20250627 13:58:03.780831 5859 ts_manager.cc:194] Registered new tserver with Master: 6302322f68c7453bb5f41861e1d469f7 (127.3.173.67:34575)
I20250627 13:58:03.782040 5859 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.173.67:55511
I20250627 13:58:03.792877 3765 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250627 13:58:03.821910 5859 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46914:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250627 13:58:03.840142 5859 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250627 13:58:03.898980 6115 tablet_service.cc:1468] Processing CreateTablet for tablet 50c51a88e784419ba71e924495c3b2a6 (DEFAULT_TABLE table=TestTable [id=a688a85e387f4e779894db385f205914]), partition=RANGE (key) PARTITION UNBOUNDED
I20250627 13:58:03.899740 5982 tablet_service.cc:1468] Processing CreateTablet for tablet 50c51a88e784419ba71e924495c3b2a6 (DEFAULT_TABLE table=TestTable [id=a688a85e387f4e779894db385f205914]), partition=RANGE (key) PARTITION UNBOUNDED
I20250627 13:58:03.900904 6115 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50c51a88e784419ba71e924495c3b2a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:58:03.900439 6248 tablet_service.cc:1468] Processing CreateTablet for tablet 50c51a88e784419ba71e924495c3b2a6 (DEFAULT_TABLE table=TestTable [id=a688a85e387f4e779894db385f205914]), partition=RANGE (key) PARTITION UNBOUNDED
I20250627 13:58:03.901161 5982 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50c51a88e784419ba71e924495c3b2a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:58:03.902195 6248 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50c51a88e784419ba71e924495c3b2a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:58:03.922665 6332 tablet_bootstrap.cc:492] T 50c51a88e784419ba71e924495c3b2a6 P 3f3021fd38f24d55a271c09a0a557cc7: Bootstrap starting.
I20250627 13:58:03.928608 6332 tablet_bootstrap.cc:654] T 50c51a88e784419ba71e924495c3b2a6 P 3f3021fd38f24d55a271c09a0a557cc7: Neither blocks nor log segments found. Creating new log.
I20250627 13:58:03.929945 6333 tablet_bootstrap.cc:492] T 50c51a88e784419ba71e924495c3b2a6 P 6d7f78b0d3704e70903eb39e97f23042: Bootstrap starting.
I20250627 13:58:03.930068 6334 tablet_bootstrap.cc:492] T 50c51a88e784419ba71e924495c3b2a6 P 6302322f68c7453bb5f41861e1d469f7: Bootstrap starting.
I20250627 13:58:03.932739 6332 log.cc:826] T 50c51a88e784419ba71e924495c3b2a6 P 3f3021fd38f24d55a271c09a0a557cc7: Log is configured to *not* fsync() on all Append() calls
I20250627 13:58:03.936421 6334 tablet_bootstrap.cc:654] T 50c51a88e784419ba71e924495c3b2a6 P 6302322f68c7453bb5f41861e1d469f7: Neither blocks nor log segments found. Creating new log.
I20250627 13:58:03.937575 6333 tablet_bootstrap.cc:654] T 50c51a88e784419ba71e924495c3b2a6 P 6d7f78b0d3704e70903eb39e97f23042: Neither blocks nor log segments found. Creating new log.
I20250627 13:58:03.938664 6332 tablet_bootstrap.cc:492] T 50c51a88e784419ba71e924495c3b2a6 P 3f3021fd38f24d55a271c09a0a557cc7: No bootstrap required, opened a new log
I20250627 13:58:03.938704 6334 log.cc:826] T 50c51a88e784419ba71e924495c3b2a6 P 6302322f68c7453bb5f41861e1d469f7: Log is configured to *not* fsync() on all Append() calls
I20250627 13:58:03.939227 6332 ts_tablet_manager.cc:1397] T 50c51a88e784419ba71e924495c3b2a6 P 3f3021fd38f24d55a271c09a0a557cc7: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.004s
I20250627 13:58:03.939917 6333 log.cc:826] T 50c51a88e784419ba71e924495c3b2a6 P 6d7f78b0d3704e70903eb39e97f23042: Log is configured to *not* fsync() on all Append() calls
I20250627 13:58:03.964663 6333 tablet_bootstrap.cc:492] T 50c51a88e784419ba71e924495c3b2a6 P 6d7f78b0d3704e70903eb39e97f23042: No bootstrap required, opened a new log
I20250627 13:58:03.965323 6333 ts_tablet_manager.cc:1397] T 50c51a88e784419ba71e924495c3b2a6 P 6d7f78b0d3704e70903eb39e97f23042: Time spent bootstrapping tablet: real 0.037s user 0.012s sys 0.022s
I20250627 13:58:03.968310 6334 tablet_bootstrap.cc:492] T 50c51a88e784419ba71e924495c3b2a6 P 6302322f68c7453bb5f41861e1d469f7: No bootstrap required, opened a new log
I20250627 13:58:03.968856 6334 ts_tablet_manager.cc:1397] T 50c51a88e784419ba71e924495c3b2a6 P 6302322f68c7453bb5f41861e1d469f7: Time spent bootstrapping tablet: real 0.039s user 0.014s sys 0.024s
I20250627 13:58:03.970935 6332 raft_consensus.cc:357] T 50c51a88e784419ba71e924495c3b2a6 P 3f3021fd38f24d55a271c09a0a557cc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3021fd38f24d55a271c09a0a557cc7" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 45801 } } peers { permanent_uuid: "6302322f68c7453bb5f41861e1d469f7" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34575 } } peers { permanent_uuid: "6d7f78b0d3704e70903eb39e97f23042" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38405 } }
I20250627 13:58:03.971872 6332 raft_consensus.cc:383] T 50c51a88e784419ba71e924495c3b2a6 P 3f3021fd38f24d55a271c09a0a557cc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:58:03.972199 6332 raft_consensus.cc:738] T 50c51a88e784419ba71e924495c3b2a6 P 3f3021fd38f24d55a271c09a0a557cc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f3021fd38f24d55a271c09a0a557cc7, State: Initialized, Role: FOLLOWER
I20250627 13:58:03.973222 6332 consensus_queue.cc:260] T 50c51a88e784419ba71e924495c3b2a6 P 3f3021fd38f24d55a271c09a0a557cc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3021fd38f24d55a271c09a0a557cc7" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 45801 } } peers { permanent_uuid: "6302322f68c7453bb5f41861e1d469f7" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34575 } } peers { permanent_uuid: "6d7f78b0d3704e70903eb39e97f23042" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38405 } }
I20250627 13:58:03.996500 6332 ts_tablet_manager.cc:1428] T 50c51a88e784419ba71e924495c3b2a6 P 3f3021fd38f24d55a271c09a0a557cc7: Time spent starting tablet: real 0.057s user 0.035s sys 0.019s
I20250627 13:58:03.997537 6334 raft_consensus.cc:357] T 50c51a88e784419ba71e924495c3b2a6 P 6302322f68c7453bb5f41861e1d469f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3021fd38f24d55a271c09a0a557cc7" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 45801 } } peers { permanent_uuid: "6302322f68c7453bb5f41861e1d469f7" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34575 } } peers { permanent_uuid: "6d7f78b0d3704e70903eb39e97f23042" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38405 } }
I20250627 13:58:03.998564 6334 raft_consensus.cc:383] T 50c51a88e784419ba71e924495c3b2a6 P 6302322f68c7453bb5f41861e1d469f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:58:03.998896 6334 raft_consensus.cc:738] T 50c51a88e784419ba71e924495c3b2a6 P 6302322f68c7453bb5f41861e1d469f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6302322f68c7453bb5f41861e1d469f7, State: Initialized, Role: FOLLOWER
I20250627 13:58:03.999954 6334 consensus_queue.cc:260] T 50c51a88e784419ba71e924495c3b2a6 P 6302322f68c7453bb5f41861e1d469f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3021fd38f24d55a271c09a0a557cc7" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 45801 } } peers { permanent_uuid: "6302322f68c7453bb5f41861e1d469f7" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34575 } } peers { permanent_uuid: "6d7f78b0d3704e70903eb39e97f23042" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38405 } }
I20250627 13:58:04.004652 6333 raft_consensus.cc:357] T 50c51a88e784419ba71e924495c3b2a6 P 6d7f78b0d3704e70903eb39e97f23042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3021fd38f24d55a271c09a0a557cc7" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 45801 } } peers { permanent_uuid: "6302322f68c7453bb5f41861e1d469f7" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34575 } } peers { permanent_uuid: "6d7f78b0d3704e70903eb39e97f23042" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38405 } }
I20250627 13:58:04.005627 6333 raft_consensus.cc:383] T 50c51a88e784419ba71e924495c3b2a6 P 6d7f78b0d3704e70903eb39e97f23042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:58:04.005961 6333 raft_consensus.cc:738] T 50c51a88e784419ba71e924495c3b2a6 P 6d7f78b0d3704e70903eb39e97f23042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d7f78b0d3704e70903eb39e97f23042, State: Initialized, Role: FOLLOWER
I20250627 13:58:04.007040 6333 consensus_queue.cc:260] T 50c51a88e784419ba71e924495c3b2a6 P 6d7f78b0d3704e70903eb39e97f23042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3021fd38f24d55a271c09a0a557cc7" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 45801 } } peers { permanent_uuid: "6302322f68c7453bb5f41861e1d469f7" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34575 } } peers { permanent_uuid: "6d7f78b0d3704e70903eb39e97f23042" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38405 } }
I20250627 13:58:04.008189 6313 heartbeater.cc:499] Master 127.3.173.126:36171 was elected leader, sending a full tablet report...
W20250627 13:58:04.010898 6314 tablet.cc:2378] T 50c51a88e784419ba71e924495c3b2a6 P 6302322f68c7453bb5f41861e1d469f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250627 13:58:04.012558 6334 ts_tablet_manager.cc:1428] T 50c51a88e784419ba71e924495c3b2a6 P 6302322f68c7453bb5f41861e1d469f7: Time spent starting tablet: real 0.043s user 0.033s sys 0.004s
I20250627 13:58:04.013839 6333 ts_tablet_manager.cc:1428] T 50c51a88e784419ba71e924495c3b2a6 P 6d7f78b0d3704e70903eb39e97f23042: Time spent starting tablet: real 0.048s user 0.027s sys 0.009s
I20250627 13:58:04.173867 6338 raft_consensus.cc:491] T 50c51a88e784419ba71e924495c3b2a6 P 3f3021fd38f24d55a271c09a0a557cc7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:58:04.174412 6338 raft_consensus.cc:513] T 50c51a88e784419ba71e924495c3b2a6 P 3f3021fd38f24d55a271c09a0a557cc7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3021fd38f24d55a271c09a0a557cc7" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 45801 } } peers { permanent_uuid: "6302322f68c7453bb5f41861e1d469f7" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34575 } } peers { permanent_uuid: "6d7f78b0d3704e70903eb39e97f23042" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38405 } }
I20250627 13:58:04.176782 6338 leader_election.cc:290] T 50c51a88e784419ba71e924495c3b2a6 P 3f3021fd38f24d55a271c09a0a557cc7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6302322f68c7453bb5f41861e1d469f7 (127.3.173.67:34575), 6d7f78b0d3704e70903eb39e97f23042 (127.3.173.66:38405)
W20250627 13:58:04.186424 6048 tablet.cc:2378] T 50c51a88e784419ba71e924495c3b2a6 P 3f3021fd38f24d55a271c09a0a557cc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250627 13:58:04.191444 6268 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c51a88e784419ba71e924495c3b2a6" candidate_uuid: "3f3021fd38f24d55a271c09a0a557cc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6302322f68c7453bb5f41861e1d469f7" is_pre_election: true
I20250627 13:58:04.191380 6135 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c51a88e784419ba71e924495c3b2a6" candidate_uuid: "3f3021fd38f24d55a271c09a0a557cc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d7f78b0d3704e70903eb39e97f23042" is_pre_election: true
I20250627 13:58:04.192121 6268 raft_consensus.cc:2466] T 50c51a88e784419ba71e924495c3b2a6 P 6302322f68c7453bb5f41861e1d469f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f3021fd38f24d55a271c09a0a557cc7 in term 0.
I20250627 13:58:04.192188 6135 raft_consensus.cc:2466] T 50c51a88e784419ba71e924495c3b2a6 P 6d7f78b0d3704e70903eb39e97f23042 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f3021fd38f24d55a271c09a0a557cc7 in term 0.
I20250627 13:58:04.193332 5938 leader_election.cc:304] T 50c51a88e784419ba71e924495c3b2a6 P 3f3021fd38f24d55a271c09a0a557cc7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f3021fd38f24d55a271c09a0a557cc7, 6302322f68c7453bb5f41861e1d469f7; no voters:
I20250627 13:58:04.194079 6338 raft_consensus.cc:2802] T 50c51a88e784419ba71e924495c3b2a6 P 3f3021fd38f24d55a271c09a0a557cc7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:58:04.194448 6338 raft_consensus.cc:491] T 50c51a88e784419ba71e924495c3b2a6 P 3f3021fd38f24d55a271c09a0a557cc7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:58:04.194769 6338 raft_consensus.cc:3058] T 50c51a88e784419ba71e924495c3b2a6 P 3f3021fd38f24d55a271c09a0a557cc7 [term 0 FOLLOWER]: Advancing to term 1
W20250627 13:58:04.199144 6181 tablet.cc:2378] T 50c51a88e784419ba71e924495c3b2a6 P 6d7f78b0d3704e70903eb39e97f23042: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250627 13:58:04.202054 6338 raft_consensus.cc:513] T 50c51a88e784419ba71e924495c3b2a6 P 3f3021fd38f24d55a271c09a0a557cc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3021fd38f24d55a271c09a0a557cc7" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 45801 } } peers { permanent_uuid: "6302322f68c7453bb5f41861e1d469f7" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34575 } } peers { permanent_uuid: "6d7f78b0d3704e70903eb39e97f23042" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38405 } }
I20250627 13:58:04.204990 6135 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c51a88e784419ba71e924495c3b2a6" candidate_uuid: "3f3021fd38f24d55a271c09a0a557cc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d7f78b0d3704e70903eb39e97f23042"
I20250627 13:58:04.205066 6268 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50c51a88e784419ba71e924495c3b2a6" candidate_uuid: "3f3021fd38f24d55a271c09a0a557cc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6302322f68c7453bb5f41861e1d469f7"
I20250627 13:58:04.205523 6135 raft_consensus.cc:3058] T 50c51a88e784419ba71e924495c3b2a6 P 6d7f78b0d3704e70903eb39e97f23042 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:58:04.205677 6268 raft_consensus.cc:3058] T 50c51a88e784419ba71e924495c3b2a6 P 6302322f68c7453bb5f41861e1d469f7 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:58:04.212248 6268 raft_consensus.cc:2466] T 50c51a88e784419ba71e924495c3b2a6 P 6302322f68c7453bb5f41861e1d469f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f3021fd38f24d55a271c09a0a557cc7 in term 1.
I20250627 13:58:04.213229 6135 raft_consensus.cc:2466] T 50c51a88e784419ba71e924495c3b2a6 P 6d7f78b0d3704e70903eb39e97f23042 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f3021fd38f24d55a271c09a0a557cc7 in term 1.
I20250627 13:58:04.213342 5938 leader_election.cc:304] T 50c51a88e784419ba71e924495c3b2a6 P 3f3021fd38f24d55a271c09a0a557cc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f3021fd38f24d55a271c09a0a557cc7, 6302322f68c7453bb5f41861e1d469f7; no voters:
I20250627 13:58:04.214959 6338 leader_election.cc:290] T 50c51a88e784419ba71e924495c3b2a6 P 3f3021fd38f24d55a271c09a0a557cc7 [CANDIDATE]: Term 1 election: Requested vote from peers 6302322f68c7453bb5f41861e1d469f7 (127.3.173.67:34575), 6d7f78b0d3704e70903eb39e97f23042 (127.3.173.66:38405)
I20250627 13:58:04.215529 6338 raft_consensus.cc:2802] T 50c51a88e784419ba71e924495c3b2a6 P 3f3021fd38f24d55a271c09a0a557cc7 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:58:04.216022 6338 raft_consensus.cc:695] T 50c51a88e784419ba71e924495c3b2a6 P 3f3021fd38f24d55a271c09a0a557cc7 [term 1 LEADER]: Becoming Leader. State: Replica: 3f3021fd38f24d55a271c09a0a557cc7, State: Running, Role: LEADER
I20250627 13:58:04.216946 6338 consensus_queue.cc:237] T 50c51a88e784419ba71e924495c3b2a6 P 3f3021fd38f24d55a271c09a0a557cc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3021fd38f24d55a271c09a0a557cc7" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 45801 } } peers { permanent_uuid: "6302322f68c7453bb5f41861e1d469f7" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34575 } } peers { permanent_uuid: "6d7f78b0d3704e70903eb39e97f23042" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38405 } }
I20250627 13:58:04.228924 5858 catalog_manager.cc:5582] T 50c51a88e784419ba71e924495c3b2a6 P 3f3021fd38f24d55a271c09a0a557cc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f3021fd38f24d55a271c09a0a557cc7 (127.3.173.65). New cstate: current_term: 1 leader_uuid: "3f3021fd38f24d55a271c09a0a557cc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f3021fd38f24d55a271c09a0a557cc7" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 45801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6302322f68c7453bb5f41861e1d469f7" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d7f78b0d3704e70903eb39e97f23042" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38405 } health_report { overall_health: UNKNOWN } } }
I20250627 13:58:04.265736 3765 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250627 13:58:04.268952 3765 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 3f3021fd38f24d55a271c09a0a557cc7 to finish bootstrapping
I20250627 13:58:04.281538 3765 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 6d7f78b0d3704e70903eb39e97f23042 to finish bootstrapping
I20250627 13:58:04.291778 3765 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 6302322f68c7453bb5f41861e1d469f7 to finish bootstrapping
I20250627 13:58:04.648741 6338 consensus_queue.cc:1035] T 50c51a88e784419ba71e924495c3b2a6 P 3f3021fd38f24d55a271c09a0a557cc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6302322f68c7453bb5f41861e1d469f7" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:58:04.674774 6338 consensus_queue.cc:1035] T 50c51a88e784419ba71e924495c3b2a6 P 3f3021fd38f24d55a271c09a0a557cc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d7f78b0d3704e70903eb39e97f23042" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250627 13:58:05.909718 5859 server_base.cc:1130] Unauthorized access attempt to method kudu.master.MasterService.RefreshAuthzCache from {username='slave'} at 127.0.0.1:45740
I20250627 13:58:06.939509 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 5918
I20250627 13:58:06.974725 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 6051
I20250627 13:58:07.002885 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 6184
I20250627 13:58:07.026190 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 5826
2025-06-27T13:58:07Z chronyd exiting
[ OK ] AdminCliTest.TestAuthzResetCacheNotAuthorized (10621 ms)
[ RUN ] AdminCliTest.TestRebuildTables
I20250627 13:58:07.081187 3765 test_util.cc:276] Using random seed: 290305734
I20250627 13:58:07.085444 3765 ts_itest-base.cc:115] Starting cluster with:
I20250627 13:58:07.085613 3765 ts_itest-base.cc:116] --------------
I20250627 13:58:07.085786 3765 ts_itest-base.cc:117] 3 tablet servers
I20250627 13:58:07.085929 3765 ts_itest-base.cc:118] 3 replicas per TS
I20250627 13:58:07.086079 3765 ts_itest-base.cc:119] --------------
2025-06-27T13:58:07Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-27T13:58:07Z Disabled control of system clock
I20250627 13:58:07.121821 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.173.126:45811
--webserver_interface=127.3.173.126
--webserver_port=0
--builtin_ntp_servers=127.3.173.84:33333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.173.126:45811 with env {}
W20250627 13:58:07.414723 6381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:58:07.415283 6381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:58:07.415786 6381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:58:07.446287 6381 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250627 13:58:07.446671 6381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:58:07.446934 6381 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250627 13:58:07.447181 6381 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250627 13:58:07.487401 6381 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:33333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.173.126:45811
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.173.126:45811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.3.173.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:58:07.488734 6381 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:58:07.490461 6381 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:58:07.504381 6388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:58:07.504415 6387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:58:08.691207 6390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:58:08.693437 6389 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1186 milliseconds
I20250627 13:58:08.693531 6381 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250627 13:58:08.694793 6381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:58:08.697857 6381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:58:08.699304 6381 hybrid_clock.cc:648] HybridClock initialized: now 1751032688699261 us; error 69 us; skew 500 ppm
I20250627 13:58:08.700115 6381 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:58:08.710448 6381 webserver.cc:469] Webserver started at http://127.3.173.126:40177/ using document root <none> and password file <none>
I20250627 13:58:08.711411 6381 fs_manager.cc:362] Metadata directory not provided
I20250627 13:58:08.711623 6381 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:58:08.712105 6381 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:58:08.716638 6381 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "e909221bbd254a3b9f691b4c8147b32b"
format_stamp: "Formatted at 2025-06-27 13:58:08 on dist-test-slave-rn80"
I20250627 13:58:08.717728 6381 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "e909221bbd254a3b9f691b4c8147b32b"
format_stamp: "Formatted at 2025-06-27 13:58:08 on dist-test-slave-rn80"
I20250627 13:58:08.724764 6381 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20250627 13:58:08.730216 6397 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:58:08.731220 6381 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20250627 13:58:08.731528 6381 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
uuid: "e909221bbd254a3b9f691b4c8147b32b"
format_stamp: "Formatted at 2025-06-27 13:58:08 on dist-test-slave-rn80"
I20250627 13:58:08.731855 6381 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:58:08.800848 6381 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:58:08.802397 6381 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:58:08.802805 6381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:58:08.871572 6381 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.126:45811
I20250627 13:58:08.871647 6448 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.126:45811 every 8 connection(s)
I20250627 13:58:08.874235 6381 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250627 13:58:08.876682 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 6381
I20250627 13:58:08.877182 3765 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250627 13:58:08.880295 6449 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:58:08.903667 6449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b: Bootstrap starting.
I20250627 13:58:08.908900 6449 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b: Neither blocks nor log segments found. Creating new log.
I20250627 13:58:08.910626 6449 log.cc:826] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b: Log is configured to *not* fsync() on all Append() calls
I20250627 13:58:08.914902 6449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b: No bootstrap required, opened a new log
I20250627 13:58:08.932199 6449 raft_consensus.cc:357] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e909221bbd254a3b9f691b4c8147b32b" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 45811 } }
I20250627 13:58:08.932962 6449 raft_consensus.cc:383] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:58:08.933261 6449 raft_consensus.cc:738] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e909221bbd254a3b9f691b4c8147b32b, State: Initialized, Role: FOLLOWER
I20250627 13:58:08.933885 6449 consensus_queue.cc:260] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e909221bbd254a3b9f691b4c8147b32b" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 45811 } }
I20250627 13:58:08.934378 6449 raft_consensus.cc:397] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250627 13:58:08.934597 6449 raft_consensus.cc:491] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250627 13:58:08.934844 6449 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:58:08.938766 6449 raft_consensus.cc:513] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e909221bbd254a3b9f691b4c8147b32b" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 45811 } }
I20250627 13:58:08.939450 6449 leader_election.cc:304] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e909221bbd254a3b9f691b4c8147b32b; no voters:
I20250627 13:58:08.941444 6449 leader_election.cc:290] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250627 13:58:08.942179 6454 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:58:08.944350 6454 raft_consensus.cc:695] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [term 1 LEADER]: Becoming Leader. State: Replica: e909221bbd254a3b9f691b4c8147b32b, State: Running, Role: LEADER
I20250627 13:58:08.945098 6454 consensus_queue.cc:237] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e909221bbd254a3b9f691b4c8147b32b" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 45811 } }
I20250627 13:58:08.946031 6449 sys_catalog.cc:564] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [sys.catalog]: configured and running, proceeding with master startup.
I20250627 13:58:08.955494 6456 sys_catalog.cc:455] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e909221bbd254a3b9f691b4c8147b32b. Latest consensus state: current_term: 1 leader_uuid: "e909221bbd254a3b9f691b4c8147b32b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e909221bbd254a3b9f691b4c8147b32b" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 45811 } } }
I20250627 13:58:08.956400 6456 sys_catalog.cc:458] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [sys.catalog]: This master's current role is: LEADER
I20250627 13:58:08.957213 6455 sys_catalog.cc:455] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e909221bbd254a3b9f691b4c8147b32b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e909221bbd254a3b9f691b4c8147b32b" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 45811 } } }
I20250627 13:58:08.958082 6455 sys_catalog.cc:458] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [sys.catalog]: This master's current role is: LEADER
I20250627 13:58:08.962298 6462 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250627 13:58:08.973857 6462 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250627 13:58:08.989688 6462 catalog_manager.cc:1349] Generated new cluster ID: fe7948cebdfa460e9e6e1c303d722e64
I20250627 13:58:08.990034 6462 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250627 13:58:09.011461 6462 catalog_manager.cc:1372] Generated new certificate authority record
I20250627 13:58:09.012859 6462 catalog_manager.cc:1506] Loading token signing keys...
I20250627 13:58:09.028182 6462 catalog_manager.cc:5955] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b: Generated new TSK 0
I20250627 13:58:09.029043 6462 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250627 13:58:09.042934 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.173.65:0
--local_ip_for_outbound_sockets=127.3.173.65
--webserver_interface=127.3.173.65
--webserver_port=0
--tserver_master_addrs=127.3.173.126:45811
--builtin_ntp_servers=127.3.173.84:33333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250627 13:58:09.339141 6473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:58:09.339671 6473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:58:09.340176 6473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:58:09.371456 6473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:58:09.372335 6473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.173.65
I20250627 13:58:09.407851 6473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:33333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.173.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.3.173.65
--webserver_port=0
--tserver_master_addrs=127.3.173.126:45811
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.173.65
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:58:09.409132 6473 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:58:09.410902 6473 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:58:09.428593 6480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:58:09.430666 6482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:58:09.432567 6473 server_base.cc:1048] running on GCE node
W20250627 13:58:09.430482 6479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:58:10.623203 6473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:58:10.625883 6473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:58:10.627383 6473 hybrid_clock.cc:648] HybridClock initialized: now 1751032690627342 us; error 42 us; skew 500 ppm
I20250627 13:58:10.628472 6473 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:58:10.636018 6473 webserver.cc:469] Webserver started at http://127.3.173.65:35721/ using document root <none> and password file <none>
I20250627 13:58:10.637219 6473 fs_manager.cc:362] Metadata directory not provided
I20250627 13:58:10.637486 6473 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:58:10.638029 6473 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:58:10.644490 6473 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "2d0c422cf8334e25855a5aca8ce93a68"
format_stamp: "Formatted at 2025-06-27 13:58:10 on dist-test-slave-rn80"
I20250627 13:58:10.645926 6473 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "2d0c422cf8334e25855a5aca8ce93a68"
format_stamp: "Formatted at 2025-06-27 13:58:10 on dist-test-slave-rn80"
I20250627 13:58:10.655140 6473 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20250627 13:58:10.663317 6489 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:58:10.664448 6473 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20250627 13:58:10.664831 6473 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "2d0c422cf8334e25855a5aca8ce93a68"
format_stamp: "Formatted at 2025-06-27 13:58:10 on dist-test-slave-rn80"
I20250627 13:58:10.665325 6473 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:58:10.739287 6473 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:58:10.740684 6473 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:58:10.741078 6473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:58:10.743539 6473 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:58:10.747478 6473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:58:10.747666 6473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:58:10.747881 6473 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:58:10.748018 6473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:58:10.879321 6473 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.65:37021
I20250627 13:58:10.879426 6601 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.65:37021 every 8 connection(s)
I20250627 13:58:10.881726 6473 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250627 13:58:10.885999 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 6473
I20250627 13:58:10.886497 3765 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250627 13:58:10.892688 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.173.66:0
--local_ip_for_outbound_sockets=127.3.173.66
--webserver_interface=127.3.173.66
--webserver_port=0
--tserver_master_addrs=127.3.173.126:45811
--builtin_ntp_servers=127.3.173.84:33333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250627 13:58:10.903798 6602 heartbeater.cc:344] Connected to a master server at 127.3.173.126:45811
I20250627 13:58:10.904210 6602 heartbeater.cc:461] Registering TS with master...
I20250627 13:58:10.905184 6602 heartbeater.cc:507] Master 127.3.173.126:45811 requested a full tablet report, sending...
I20250627 13:58:10.907928 6414 ts_manager.cc:194] Registered new tserver with Master: 2d0c422cf8334e25855a5aca8ce93a68 (127.3.173.65:37021)
I20250627 13:58:10.910913 6414 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.173.65:57691
W20250627 13:58:11.186419 6606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:58:11.186889 6606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:58:11.187318 6606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:58:11.217700 6606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:58:11.218536 6606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.173.66
I20250627 13:58:11.252310 6606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:33333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.173.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.3.173.66
--webserver_port=0
--tserver_master_addrs=127.3.173.126:45811
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.173.66
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:58:11.253566 6606 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:58:11.255304 6606 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:58:11.271685 6612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:58:11.914680 6602 heartbeater.cc:499] Master 127.3.173.126:45811 was elected leader, sending a full tablet report...
W20250627 13:58:11.271756 6613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:58:11.272426 6615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:58:11.273670 6606 server_base.cc:1048] running on GCE node
I20250627 13:58:12.422411 6606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:58:12.424487 6606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:58:12.425812 6606 hybrid_clock.cc:648] HybridClock initialized: now 1751032692425776 us; error 55 us; skew 500 ppm
I20250627 13:58:12.426610 6606 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:58:12.432606 6606 webserver.cc:469] Webserver started at http://127.3.173.66:36955/ using document root <none> and password file <none>
I20250627 13:58:12.433561 6606 fs_manager.cc:362] Metadata directory not provided
I20250627 13:58:12.433753 6606 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:58:12.434259 6606 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:58:12.438670 6606 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "96420c429aa34d06b05da75fc24060c6"
format_stamp: "Formatted at 2025-06-27 13:58:12 on dist-test-slave-rn80"
I20250627 13:58:12.439731 6606 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "96420c429aa34d06b05da75fc24060c6"
format_stamp: "Formatted at 2025-06-27 13:58:12 on dist-test-slave-rn80"
I20250627 13:58:12.446486 6606 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20250627 13:58:12.451893 6622 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:58:12.452873 6606 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250627 13:58:12.453182 6606 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "96420c429aa34d06b05da75fc24060c6"
format_stamp: "Formatted at 2025-06-27 13:58:12 on dist-test-slave-rn80"
I20250627 13:58:12.453492 6606 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:58:12.502806 6606 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:58:12.504210 6606 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:58:12.504647 6606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:58:12.507289 6606 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:58:12.511473 6606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:58:12.511659 6606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:58:12.511907 6606 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:58:12.512060 6606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:58:12.640560 6606 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.66:38459
I20250627 13:58:12.640667 6734 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.66:38459 every 8 connection(s)
I20250627 13:58:12.643095 6606 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250627 13:58:12.643821 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 6606
I20250627 13:58:12.644202 3765 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250627 13:58:12.652796 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.173.67:0
--local_ip_for_outbound_sockets=127.3.173.67
--webserver_interface=127.3.173.67
--webserver_port=0
--tserver_master_addrs=127.3.173.126:45811
--builtin_ntp_servers=127.3.173.84:33333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250627 13:58:12.668900 6735 heartbeater.cc:344] Connected to a master server at 127.3.173.126:45811
I20250627 13:58:12.669337 6735 heartbeater.cc:461] Registering TS with master...
I20250627 13:58:12.670382 6735 heartbeater.cc:507] Master 127.3.173.126:45811 requested a full tablet report, sending...
I20250627 13:58:12.672561 6414 ts_manager.cc:194] Registered new tserver with Master: 96420c429aa34d06b05da75fc24060c6 (127.3.173.66:38459)
I20250627 13:58:12.673784 6414 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.173.66:60529
W20250627 13:58:12.978818 6739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:58:12.979336 6739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:58:12.979832 6739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:58:13.010803 6739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:58:13.011647 6739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.173.67
I20250627 13:58:13.045958 6739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:33333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.173.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.3.173.67
--webserver_port=0
--tserver_master_addrs=127.3.173.126:45811
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.173.67
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:58:13.047276 6739 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:58:13.048861 6739 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:58:13.064445 6746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:58:13.676865 6735 heartbeater.cc:499] Master 127.3.173.126:45811 was elected leader, sending a full tablet report...
W20250627 13:58:13.064695 6745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:58:13.066711 6739 server_base.cc:1048] running on GCE node
W20250627 13:58:13.064771 6748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:58:14.233381 6739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:58:14.235687 6739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:58:14.237015 6739 hybrid_clock.cc:648] HybridClock initialized: now 1751032694236976 us; error 47 us; skew 500 ppm
I20250627 13:58:14.237787 6739 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:58:14.249969 6739 webserver.cc:469] Webserver started at http://127.3.173.67:45421/ using document root <none> and password file <none>
I20250627 13:58:14.250959 6739 fs_manager.cc:362] Metadata directory not provided
I20250627 13:58:14.251176 6739 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:58:14.251632 6739 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:58:14.256062 6739 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "8f73a64a53924fe6a9338f131b5db19c"
format_stamp: "Formatted at 2025-06-27 13:58:14 on dist-test-slave-rn80"
I20250627 13:58:14.257124 6739 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "8f73a64a53924fe6a9338f131b5db19c"
format_stamp: "Formatted at 2025-06-27 13:58:14 on dist-test-slave-rn80"
I20250627 13:58:14.264645 6739 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.001s sys 0.008s
I20250627 13:58:14.270243 6755 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:58:14.271194 6739 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250627 13:58:14.271463 6739 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "8f73a64a53924fe6a9338f131b5db19c"
format_stamp: "Formatted at 2025-06-27 13:58:14 on dist-test-slave-rn80"
I20250627 13:58:14.271786 6739 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:58:14.326874 6739 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:58:14.328230 6739 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:58:14.328631 6739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:58:14.331096 6739 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:58:14.335075 6739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:58:14.335269 6739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:58:14.335465 6739 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:58:14.335626 6739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:58:14.467120 6739 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.67:34731
I20250627 13:58:14.467247 6867 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.67:34731 every 8 connection(s)
I20250627 13:58:14.469825 6739 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250627 13:58:14.473886 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 6739
I20250627 13:58:14.474401 3765 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250627 13:58:14.490036 6868 heartbeater.cc:344] Connected to a master server at 127.3.173.126:45811
I20250627 13:58:14.490535 6868 heartbeater.cc:461] Registering TS with master...
I20250627 13:58:14.491463 6868 heartbeater.cc:507] Master 127.3.173.126:45811 requested a full tablet report, sending...
I20250627 13:58:14.493643 6414 ts_manager.cc:194] Registered new tserver with Master: 8f73a64a53924fe6a9338f131b5db19c (127.3.173.67:34731)
I20250627 13:58:14.494896 3765 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250627 13:58:14.495108 6414 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.173.67:49905
I20250627 13:58:14.526216 3765 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250627 13:58:14.526510 3765 test_util.cc:276] Using random seed: 297751070
I20250627 13:58:14.566196 6414 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34728:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250627 13:58:14.605813 6803 tablet_service.cc:1468] Processing CreateTablet for tablet d740546f249747b29b61418e84a952bb (DEFAULT_TABLE table=TestTable [id=175b6e08f2054def91df0b1a2c64b40e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250627 13:58:14.607249 6803 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d740546f249747b29b61418e84a952bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:58:14.625804 6888 tablet_bootstrap.cc:492] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c: Bootstrap starting.
I20250627 13:58:14.631356 6888 tablet_bootstrap.cc:654] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c: Neither blocks nor log segments found. Creating new log.
I20250627 13:58:14.633064 6888 log.cc:826] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c: Log is configured to *not* fsync() on all Append() calls
I20250627 13:58:14.637660 6888 tablet_bootstrap.cc:492] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c: No bootstrap required, opened a new log
I20250627 13:58:14.638202 6888 ts_tablet_manager.cc:1397] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20250627 13:58:14.660809 6888 raft_consensus.cc:357] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } }
I20250627 13:58:14.661471 6888 raft_consensus.cc:383] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:58:14.661748 6888 raft_consensus.cc:738] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f73a64a53924fe6a9338f131b5db19c, State: Initialized, Role: FOLLOWER
I20250627 13:58:14.662503 6888 consensus_queue.cc:260] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } }
I20250627 13:58:14.663009 6888 raft_consensus.cc:397] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250627 13:58:14.663345 6888 raft_consensus.cc:491] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250627 13:58:14.663745 6888 raft_consensus.cc:3058] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:58:14.667965 6888 raft_consensus.cc:513] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } }
I20250627 13:58:14.668602 6888 leader_election.cc:304] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f73a64a53924fe6a9338f131b5db19c; no voters:
I20250627 13:58:14.670226 6888 leader_election.cc:290] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250627 13:58:14.670980 6890 raft_consensus.cc:2802] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:58:14.673171 6890 raft_consensus.cc:695] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [term 1 LEADER]: Becoming Leader. State: Replica: 8f73a64a53924fe6a9338f131b5db19c, State: Running, Role: LEADER
I20250627 13:58:14.673501 6868 heartbeater.cc:499] Master 127.3.173.126:45811 was elected leader, sending a full tablet report...
I20250627 13:58:14.674036 6890 consensus_queue.cc:237] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } }
I20250627 13:58:14.674470 6888 ts_tablet_manager.cc:1428] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c: Time spent starting tablet: real 0.036s user 0.028s sys 0.008s
I20250627 13:58:14.686851 6414 catalog_manager.cc:5582] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f73a64a53924fe6a9338f131b5db19c (127.3.173.67). New cstate: current_term: 1 leader_uuid: "8f73a64a53924fe6a9338f131b5db19c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } health_report { overall_health: HEALTHY } } }
I20250627 13:58:14.888792 3765 test_util.cc:276] Using random seed: 298113202
I20250627 13:58:14.914595 6413 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34732:
name: "TestTable1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250627 13:58:14.946550 6537 tablet_service.cc:1468] Processing CreateTablet for tablet 4ee5162104534a39a5cf9ff9a0ab7a4d (DEFAULT_TABLE table=TestTable1 [id=af09358a541045e1921fd36a8cff7b33]), partition=RANGE (key) PARTITION UNBOUNDED
I20250627 13:58:14.948019 6537 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ee5162104534a39a5cf9ff9a0ab7a4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:58:14.967558 6910 tablet_bootstrap.cc:492] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68: Bootstrap starting.
I20250627 13:58:14.973403 6910 tablet_bootstrap.cc:654] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68: Neither blocks nor log segments found. Creating new log.
I20250627 13:58:14.975135 6910 log.cc:826] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68: Log is configured to *not* fsync() on all Append() calls
I20250627 13:58:14.979400 6910 tablet_bootstrap.cc:492] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68: No bootstrap required, opened a new log
I20250627 13:58:14.979857 6910 ts_tablet_manager.cc:1397] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20250627 13:58:14.997670 6910 raft_consensus.cc:357] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } }
I20250627 13:58:14.998284 6910 raft_consensus.cc:383] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:58:14.998570 6910 raft_consensus.cc:738] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d0c422cf8334e25855a5aca8ce93a68, State: Initialized, Role: FOLLOWER
I20250627 13:58:14.999254 6910 consensus_queue.cc:260] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } }
I20250627 13:58:14.999732 6910 raft_consensus.cc:397] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250627 13:58:14.999954 6910 raft_consensus.cc:491] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250627 13:58:15.000200 6910 raft_consensus.cc:3058] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:58:15.004331 6910 raft_consensus.cc:513] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } }
I20250627 13:58:15.004997 6910 leader_election.cc:304] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d0c422cf8334e25855a5aca8ce93a68; no voters:
I20250627 13:58:15.006995 6910 leader_election.cc:290] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250627 13:58:15.007438 6912 raft_consensus.cc:2802] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:58:15.010041 6912 raft_consensus.cc:695] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [term 1 LEADER]: Becoming Leader. State: Replica: 2d0c422cf8334e25855a5aca8ce93a68, State: Running, Role: LEADER
I20250627 13:58:15.010962 6912 consensus_queue.cc:237] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } }
I20250627 13:58:15.011555 6910 ts_tablet_manager.cc:1428] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68: Time spent starting tablet: real 0.031s user 0.019s sys 0.013s
I20250627 13:58:15.022212 6412 catalog_manager.cc:5582] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d0c422cf8334e25855a5aca8ce93a68 (127.3.173.65). New cstate: current_term: 1 leader_uuid: "2d0c422cf8334e25855a5aca8ce93a68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } health_report { overall_health: HEALTHY } } }
I20250627 13:58:15.221164 3765 test_util.cc:276] Using random seed: 298445712
I20250627 13:58:15.242236 6413 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34740:
name: "TestTable2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250627 13:58:15.270292 6670 tablet_service.cc:1468] Processing CreateTablet for tablet 12047b7d6a5e488c8875843b74f2b661 (DEFAULT_TABLE table=TestTable2 [id=3c10e1f63a5d4e05a6249bf9a108319a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250627 13:58:15.271651 6670 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12047b7d6a5e488c8875843b74f2b661. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:58:15.291239 6931 tablet_bootstrap.cc:492] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6: Bootstrap starting.
I20250627 13:58:15.296828 6931 tablet_bootstrap.cc:654] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6: Neither blocks nor log segments found. Creating new log.
I20250627 13:58:15.298801 6931 log.cc:826] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6: Log is configured to *not* fsync() on all Append() calls
I20250627 13:58:15.303282 6931 tablet_bootstrap.cc:492] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6: No bootstrap required, opened a new log
I20250627 13:58:15.303699 6931 ts_tablet_manager.cc:1397] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20250627 13:58:15.320951 6931 raft_consensus.cc:357] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } }
I20250627 13:58:15.321578 6931 raft_consensus.cc:383] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:58:15.321815 6931 raft_consensus.cc:738] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96420c429aa34d06b05da75fc24060c6, State: Initialized, Role: FOLLOWER
I20250627 13:58:15.322554 6931 consensus_queue.cc:260] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } }
I20250627 13:58:15.323083 6931 raft_consensus.cc:397] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250627 13:58:15.323375 6931 raft_consensus.cc:491] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250627 13:58:15.323704 6931 raft_consensus.cc:3058] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:58:15.327816 6931 raft_consensus.cc:513] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } }
I20250627 13:58:15.328516 6931 leader_election.cc:304] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96420c429aa34d06b05da75fc24060c6; no voters:
I20250627 13:58:15.330539 6931 leader_election.cc:290] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250627 13:58:15.330924 6933 raft_consensus.cc:2802] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:58:15.332976 6933 raft_consensus.cc:695] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6 [term 1 LEADER]: Becoming Leader. State: Replica: 96420c429aa34d06b05da75fc24060c6, State: Running, Role: LEADER
I20250627 13:58:15.334326 6931 ts_tablet_manager.cc:1428] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6: Time spent starting tablet: real 0.030s user 0.028s sys 0.004s
I20250627 13:58:15.333993 6933 consensus_queue.cc:237] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } }
I20250627 13:58:15.345036 6413 catalog_manager.cc:5582] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96420c429aa34d06b05da75fc24060c6 (127.3.173.66). New cstate: current_term: 1 leader_uuid: "96420c429aa34d06b05da75fc24060c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } health_report { overall_health: HEALTHY } } }
I20250627 13:58:15.555271 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 6381
W20250627 13:58:15.701821 6868 heartbeater.cc:646] Failed to heartbeat to 127.3.173.126:45811 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.173.126:45811: connect: Connection refused (error 111)
W20250627 13:58:16.040217 6602 heartbeater.cc:646] Failed to heartbeat to 127.3.173.126:45811 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.173.126:45811: connect: Connection refused (error 111)
W20250627 13:58:16.361151 6735 heartbeater.cc:646] Failed to heartbeat to 127.3.173.126:45811 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.173.126:45811: connect: Connection refused (error 111)
I20250627 13:58:20.337013 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 6473
I20250627 13:58:20.359509 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 6606
I20250627 13:58:20.386519 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 6739
I20250627 13:58:20.413015 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.173.126:45811
--webserver_interface=127.3.173.126
--webserver_port=40177
--builtin_ntp_servers=127.3.173.84:33333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.173.126:45811 with env {}
W20250627 13:58:20.700049 7009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:58:20.700620 7009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:58:20.701030 7009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:58:20.731951 7009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250627 13:58:20.732235 7009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:58:20.732450 7009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250627 13:58:20.732652 7009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250627 13:58:20.766314 7009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:33333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.173.126:45811
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.173.126:45811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.3.173.126
--webserver_port=40177
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:58:20.767560 7009 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:58:20.769073 7009 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:58:20.782847 7016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:58:20.783130 7018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:58:20.784351 7009 server_base.cc:1048] running on GCE node
W20250627 13:58:20.783681 7015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:58:21.953275 7009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:58:21.955828 7009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:58:21.957190 7009 hybrid_clock.cc:648] HybridClock initialized: now 1751032701957156 us; error 61 us; skew 500 ppm
I20250627 13:58:21.957960 7009 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:58:21.969635 7009 webserver.cc:469] Webserver started at http://127.3.173.126:40177/ using document root <none> and password file <none>
I20250627 13:58:21.970602 7009 fs_manager.cc:362] Metadata directory not provided
I20250627 13:58:21.970810 7009 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:58:21.978340 7009 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.001s
I20250627 13:58:21.982817 7025 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:58:21.983843 7009 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250627 13:58:21.984164 7009 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
uuid: "e909221bbd254a3b9f691b4c8147b32b"
format_stamp: "Formatted at 2025-06-27 13:58:08 on dist-test-slave-rn80"
I20250627 13:58:21.986018 7009 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:58:22.042745 7009 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:58:22.044188 7009 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:58:22.044615 7009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:58:22.113659 7009 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.126:45811
I20250627 13:58:22.113734 7076 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.126:45811 every 8 connection(s)
I20250627 13:58:22.116410 7009 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250627 13:58:22.116995 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 7009
I20250627 13:58:22.118469 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.173.65:37021
--local_ip_for_outbound_sockets=127.3.173.65
--tserver_master_addrs=127.3.173.126:45811
--webserver_port=35721
--webserver_interface=127.3.173.65
--builtin_ntp_servers=127.3.173.84:33333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250627 13:58:22.127182 7077 sys_catalog.cc:263] Verifying existing consensus state
I20250627 13:58:22.131713 7077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b: Bootstrap starting.
I20250627 13:58:22.141666 7077 log.cc:826] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b: Log is configured to *not* fsync() on all Append() calls
I20250627 13:58:22.188299 7077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=13 ignored=0} mutations{seen=10 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:58:22.189183 7077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b: Bootstrap complete.
I20250627 13:58:22.209618 7077 raft_consensus.cc:357] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e909221bbd254a3b9f691b4c8147b32b" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 45811 } }
I20250627 13:58:22.211743 7077 raft_consensus.cc:738] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e909221bbd254a3b9f691b4c8147b32b, State: Initialized, Role: FOLLOWER
I20250627 13:58:22.212568 7077 consensus_queue.cc:260] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e909221bbd254a3b9f691b4c8147b32b" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 45811 } }
I20250627 13:58:22.213073 7077 raft_consensus.cc:397] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250627 13:58:22.213335 7077 raft_consensus.cc:491] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250627 13:58:22.213617 7077 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [term 2 FOLLOWER]: Advancing to term 3
I20250627 13:58:22.219205 7077 raft_consensus.cc:513] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e909221bbd254a3b9f691b4c8147b32b" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 45811 } }
I20250627 13:58:22.219827 7077 leader_election.cc:304] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e909221bbd254a3b9f691b4c8147b32b; no voters:
I20250627 13:58:22.221913 7077 leader_election.cc:290] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [CANDIDATE]: Term 3 election: Requested vote from peers
I20250627 13:58:22.222417 7081 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [term 3 FOLLOWER]: Leader election won for term 3
I20250627 13:58:22.225731 7081 raft_consensus.cc:695] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [term 3 LEADER]: Becoming Leader. State: Replica: e909221bbd254a3b9f691b4c8147b32b, State: Running, Role: LEADER
I20250627 13:58:22.226625 7081 consensus_queue.cc:237] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e909221bbd254a3b9f691b4c8147b32b" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 45811 } }
I20250627 13:58:22.226994 7077 sys_catalog.cc:564] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [sys.catalog]: configured and running, proceeding with master startup.
I20250627 13:58:22.237965 7082 sys_catalog.cc:455] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e909221bbd254a3b9f691b4c8147b32b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e909221bbd254a3b9f691b4c8147b32b" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 45811 } } }
I20250627 13:58:22.237916 7083 sys_catalog.cc:455] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e909221bbd254a3b9f691b4c8147b32b. Latest consensus state: current_term: 3 leader_uuid: "e909221bbd254a3b9f691b4c8147b32b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e909221bbd254a3b9f691b4c8147b32b" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 45811 } } }
I20250627 13:58:22.238956 7082 sys_catalog.cc:458] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [sys.catalog]: This master's current role is: LEADER
I20250627 13:58:22.239114 7083 sys_catalog.cc:458] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [sys.catalog]: This master's current role is: LEADER
I20250627 13:58:22.246086 7088 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250627 13:58:22.262056 7088 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=3c10e1f63a5d4e05a6249bf9a108319a]
I20250627 13:58:22.265166 7088 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=478b9e28b731427baf55d53330dc687c]
I20250627 13:58:22.267326 7088 catalog_manager.cc:671] Loaded metadata for table TestTable [id=e7fd25830df045de9a471ea8d92c851a]
I20250627 13:58:22.277297 7088 tablet_loader.cc:96] loaded metadata for tablet 12047b7d6a5e488c8875843b74f2b661 (table TestTable2 [id=3c10e1f63a5d4e05a6249bf9a108319a])
I20250627 13:58:22.279074 7088 tablet_loader.cc:96] loaded metadata for tablet 4ee5162104534a39a5cf9ff9a0ab7a4d (table TestTable1 [id=478b9e28b731427baf55d53330dc687c])
I20250627 13:58:22.280882 7088 tablet_loader.cc:96] loaded metadata for tablet d740546f249747b29b61418e84a952bb (table TestTable [id=e7fd25830df045de9a471ea8d92c851a])
I20250627 13:58:22.283066 7088 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250627 13:58:22.289086 7088 catalog_manager.cc:1261] Loaded cluster ID: fe7948cebdfa460e9e6e1c303d722e64
I20250627 13:58:22.289674 7088 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250627 13:58:22.300076 7088 catalog_manager.cc:1506] Loading token signing keys...
I20250627 13:58:22.307291 7088 catalog_manager.cc:5966] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b: Loaded TSK: 0
I20250627 13:58:22.309222 7088 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250627 13:58:22.462740 7079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:58:22.463250 7079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:58:22.463790 7079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:58:22.495172 7079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:58:22.496017 7079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.173.65
I20250627 13:58:22.530431 7079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:33333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.173.65:37021
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.3.173.65
--webserver_port=35721
--tserver_master_addrs=127.3.173.126:45811
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.173.65
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:58:22.531785 7079 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:58:22.533401 7079 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:58:22.550668 7106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:58:22.552889 7105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:58:22.553359 7079 server_base.cc:1048] running on GCE node
W20250627 13:58:22.552361 7108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:58:23.740947 7079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:58:23.743827 7079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:58:23.745260 7079 hybrid_clock.cc:648] HybridClock initialized: now 1751032703745202 us; error 75 us; skew 500 ppm
I20250627 13:58:23.746055 7079 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:58:23.753427 7079 webserver.cc:469] Webserver started at http://127.3.173.65:35721/ using document root <none> and password file <none>
I20250627 13:58:23.754426 7079 fs_manager.cc:362] Metadata directory not provided
I20250627 13:58:23.754647 7079 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:58:23.762717 7079 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20250627 13:58:23.767464 7115 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:58:23.768563 7079 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20250627 13:58:23.768852 7079 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "2d0c422cf8334e25855a5aca8ce93a68"
format_stamp: "Formatted at 2025-06-27 13:58:10 on dist-test-slave-rn80"
I20250627 13:58:23.770804 7079 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:58:23.816092 7079 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:58:23.817430 7079 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:58:23.817816 7079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:58:23.820863 7079 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:58:23.827039 7122 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250627 13:58:23.834506 7079 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250627 13:58:23.834728 7079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.002s
I20250627 13:58:23.835016 7079 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250627 13:58:23.839572 7079 ts_tablet_manager.cc:610] Registered 1 tablets
I20250627 13:58:23.839813 7079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.003s sys 0.001s
I20250627 13:58:23.840169 7122 tablet_bootstrap.cc:492] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68: Bootstrap starting.
I20250627 13:58:23.917259 7122 log.cc:826] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68: Log is configured to *not* fsync() on all Append() calls
I20250627 13:58:24.019835 7079 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.65:37021
I20250627 13:58:24.020002 7229 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.65:37021 every 8 connection(s)
I20250627 13:58:24.023380 7079 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250627 13:58:24.027170 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 7079
I20250627 13:58:24.028857 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.173.66:38459
--local_ip_for_outbound_sockets=127.3.173.66
--tserver_master_addrs=127.3.173.126:45811
--webserver_port=36955
--webserver_interface=127.3.173.66
--builtin_ntp_servers=127.3.173.84:33333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250627 13:58:24.039414 7122 tablet_bootstrap.cc:492] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:58:24.040545 7122 tablet_bootstrap.cc:492] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68: Bootstrap complete.
I20250627 13:58:24.042428 7122 ts_tablet_manager.cc:1397] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68: Time spent bootstrapping tablet: real 0.203s user 0.154s sys 0.047s
I20250627 13:58:24.066418 7122 raft_consensus.cc:357] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } }
I20250627 13:58:24.069468 7122 raft_consensus.cc:738] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d0c422cf8334e25855a5aca8ce93a68, State: Initialized, Role: FOLLOWER
I20250627 13:58:24.070567 7122 consensus_queue.cc:260] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } }
I20250627 13:58:24.071264 7122 raft_consensus.cc:397] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250627 13:58:24.071630 7122 raft_consensus.cc:491] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250627 13:58:24.072010 7122 raft_consensus.cc:3058] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:58:24.079703 7122 raft_consensus.cc:513] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } }
I20250627 13:58:24.080614 7122 leader_election.cc:304] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d0c422cf8334e25855a5aca8ce93a68; no voters:
I20250627 13:58:24.084321 7230 heartbeater.cc:344] Connected to a master server at 127.3.173.126:45811
I20250627 13:58:24.084798 7230 heartbeater.cc:461] Registering TS with master...
I20250627 13:58:24.085938 7230 heartbeater.cc:507] Master 127.3.173.126:45811 requested a full tablet report, sending...
I20250627 13:58:24.088611 7235 raft_consensus.cc:2802] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [term 2 FOLLOWER]: Leader election won for term 2
I20250627 13:58:24.089072 7122 leader_election.cc:290] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250627 13:58:24.090700 7042 ts_manager.cc:194] Registered new tserver with Master: 2d0c422cf8334e25855a5aca8ce93a68 (127.3.173.65:37021)
I20250627 13:58:24.098635 7042 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.173.65:44129
I20250627 13:58:24.100342 7235 raft_consensus.cc:695] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [term 2 LEADER]: Becoming Leader. State: Replica: 2d0c422cf8334e25855a5aca8ce93a68, State: Running, Role: LEADER
I20250627 13:58:24.101878 7235 consensus_queue.cc:237] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } }
I20250627 13:58:24.105370 7230 heartbeater.cc:499] Master 127.3.173.126:45811 was elected leader, sending a full tablet report...
I20250627 13:58:24.106031 7122 ts_tablet_manager.cc:1428] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68: Time spent starting tablet: real 0.063s user 0.037s sys 0.016s
I20250627 13:58:24.118820 7042 catalog_manager.cc:5582] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 reported cstate change: term changed from 0 to 2, leader changed from <none> to 2d0c422cf8334e25855a5aca8ce93a68 (127.3.173.65), VOTER 2d0c422cf8334e25855a5aca8ce93a68 (127.3.173.65) added. New cstate: current_term: 2 leader_uuid: "2d0c422cf8334e25855a5aca8ce93a68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } health_report { overall_health: HEALTHY } } }
W20250627 13:58:24.170586 7042 catalog_manager.cc:5260] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4ee5162104534a39a5cf9ff9a0ab7a4d with cas_config_opid_index -1: no extra replica candidate found for tablet 4ee5162104534a39a5cf9ff9a0ab7a4d (table TestTable1 [id=478b9e28b731427baf55d53330dc687c]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (1) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
W20250627 13:58:24.373131 7232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:58:24.373590 7232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:58:24.374040 7232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:58:24.404664 7232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:58:24.405449 7232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.173.66
I20250627 13:58:24.439339 7232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:33333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.173.66:38459
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.3.173.66
--webserver_port=36955
--tserver_master_addrs=127.3.173.126:45811
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.173.66
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:58:24.440528 7232 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:58:24.442164 7232 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:58:24.456471 7250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:58:24.456468 7249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:58:24.457553 7252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:58:25.677845 7251 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1219 milliseconds
I20250627 13:58:25.677956 7232 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250627 13:58:25.679169 7232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:58:25.682478 7232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:58:25.683862 7232 hybrid_clock.cc:648] HybridClock initialized: now 1751032705683822 us; error 54 us; skew 500 ppm
I20250627 13:58:25.684648 7232 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:58:25.690935 7232 webserver.cc:469] Webserver started at http://127.3.173.66:36955/ using document root <none> and password file <none>
I20250627 13:58:25.691850 7232 fs_manager.cc:362] Metadata directory not provided
I20250627 13:58:25.692081 7232 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:58:25.699656 7232 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20250627 13:58:25.704351 7260 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:58:25.705355 7232 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250627 13:58:25.705672 7232 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "96420c429aa34d06b05da75fc24060c6"
format_stamp: "Formatted at 2025-06-27 13:58:12 on dist-test-slave-rn80"
I20250627 13:58:25.707697 7232 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:58:25.753808 7232 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:58:25.755277 7232 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:58:25.755713 7232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:58:25.758138 7232 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:58:25.763736 7267 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250627 13:58:25.771377 7232 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250627 13:58:25.771647 7232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.001s sys 0.000s
I20250627 13:58:25.771930 7232 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250627 13:58:25.776496 7232 ts_tablet_manager.cc:610] Registered 1 tablets
I20250627 13:58:25.776686 7232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.003s sys 0.000s
I20250627 13:58:25.777060 7267 tablet_bootstrap.cc:492] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6: Bootstrap starting.
I20250627 13:58:25.831786 7267 log.cc:826] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6: Log is configured to *not* fsync() on all Append() calls
I20250627 13:58:25.916764 7267 tablet_bootstrap.cc:492] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:58:25.917831 7267 tablet_bootstrap.cc:492] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6: Bootstrap complete.
I20250627 13:58:25.919335 7267 ts_tablet_manager.cc:1397] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6: Time spent bootstrapping tablet: real 0.143s user 0.112s sys 0.028s
I20250627 13:58:25.934567 7267 raft_consensus.cc:357] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } }
I20250627 13:58:25.937156 7267 raft_consensus.cc:738] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96420c429aa34d06b05da75fc24060c6, State: Initialized, Role: FOLLOWER
I20250627 13:58:25.938055 7267 consensus_queue.cc:260] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } }
I20250627 13:58:25.938592 7267 raft_consensus.cc:397] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250627 13:58:25.938846 7267 raft_consensus.cc:491] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250627 13:58:25.939226 7267 raft_consensus.cc:3058] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:58:25.945686 7267 raft_consensus.cc:513] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } }
I20250627 13:58:25.946409 7267 leader_election.cc:304] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96420c429aa34d06b05da75fc24060c6; no voters:
I20250627 13:58:25.948858 7267 leader_election.cc:290] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250627 13:58:25.949281 7372 raft_consensus.cc:2802] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6 [term 2 FOLLOWER]: Leader election won for term 2
I20250627 13:58:25.951803 7372 raft_consensus.cc:695] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6 [term 2 LEADER]: Becoming Leader. State: Replica: 96420c429aa34d06b05da75fc24060c6, State: Running, Role: LEADER
I20250627 13:58:25.952911 7372 consensus_queue.cc:237] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } }
I20250627 13:58:25.954854 7267 ts_tablet_manager.cc:1428] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6: Time spent starting tablet: real 0.035s user 0.031s sys 0.004s
I20250627 13:58:25.957481 7232 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.66:38459
I20250627 13:58:25.958326 7378 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.66:38459 every 8 connection(s)
I20250627 13:58:25.960592 7232 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250627 13:58:25.969213 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 7232
I20250627 13:58:25.971141 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.173.67:34731
--local_ip_for_outbound_sockets=127.3.173.67
--tserver_master_addrs=127.3.173.126:45811
--webserver_port=45421
--webserver_interface=127.3.173.67
--builtin_ntp_servers=127.3.173.84:33333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250627 13:58:25.978585 7380 heartbeater.cc:344] Connected to a master server at 127.3.173.126:45811
I20250627 13:58:25.979087 7380 heartbeater.cc:461] Registering TS with master...
I20250627 13:58:25.980343 7380 heartbeater.cc:507] Master 127.3.173.126:45811 requested a full tablet report, sending...
I20250627 13:58:25.984299 7042 ts_manager.cc:194] Registered new tserver with Master: 96420c429aa34d06b05da75fc24060c6 (127.3.173.66:38459)
I20250627 13:58:25.985248 7042 catalog_manager.cc:5582] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "96420c429aa34d06b05da75fc24060c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } health_report { overall_health: HEALTHY } } }
I20250627 13:58:25.995724 7042 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.173.66:32789
I20250627 13:58:25.999540 7380 heartbeater.cc:499] Master 127.3.173.126:45811 was elected leader, sending a full tablet report...
W20250627 13:58:26.273252 7384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:58:26.273718 7384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:58:26.274216 7384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:58:26.305269 7384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:58:26.306195 7384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.173.67
I20250627 13:58:26.340236 7384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:33333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.173.67:34731
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.3.173.67
--webserver_port=45421
--tserver_master_addrs=127.3.173.126:45811
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.173.67
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:58:26.341545 7384 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:58:26.343128 7384 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:58:26.359683 7397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:58:26.488765 7185 consensus_queue.cc:237] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: NON_VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: true } }
I20250627 13:58:26.498205 7404 raft_consensus.cc:2953] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index -1 to 8, NON_VOTER 96420c429aa34d06b05da75fc24060c6 (127.3.173.66) added. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: NON_VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: true } } }
I20250627 13:58:26.512243 7027 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4ee5162104534a39a5cf9ff9a0ab7a4d with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 8)
I20250627 13:58:26.518721 7042 catalog_manager.cc:5582] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 reported cstate change: config changed from index -1 to 8, NON_VOTER 96420c429aa34d06b05da75fc24060c6 (127.3.173.66) added. New cstate: current_term: 2 leader_uuid: "2d0c422cf8334e25855a5aca8ce93a68" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: NON_VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250627 13:58:26.528841 7117 consensus_peers.cc:487] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 -> Peer 96420c429aa34d06b05da75fc24060c6 (127.3.173.66:38459): Couldn't send request to peer 96420c429aa34d06b05da75fc24060c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4ee5162104534a39a5cf9ff9a0ab7a4d. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:58:26.945292 7410 ts_tablet_manager.cc:927] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6: Initiating tablet copy from peer 2d0c422cf8334e25855a5aca8ce93a68 (127.3.173.65:37021)
I20250627 13:58:26.947408 7410 tablet_copy_client.cc:323] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6: tablet copy: Beginning tablet copy session from remote peer at address 127.3.173.65:37021
I20250627 13:58:26.962241 7205 tablet_copy_service.cc:140] P 2d0c422cf8334e25855a5aca8ce93a68: Received BeginTabletCopySession request for tablet 4ee5162104534a39a5cf9ff9a0ab7a4d from peer 96420c429aa34d06b05da75fc24060c6 ({username='slave'} at 127.3.173.66:50523)
I20250627 13:58:26.962878 7205 tablet_copy_service.cc:161] P 2d0c422cf8334e25855a5aca8ce93a68: Beginning new tablet copy session on tablet 4ee5162104534a39a5cf9ff9a0ab7a4d from peer 96420c429aa34d06b05da75fc24060c6 at {username='slave'} at 127.3.173.66:50523: session id = 96420c429aa34d06b05da75fc24060c6-4ee5162104534a39a5cf9ff9a0ab7a4d
I20250627 13:58:26.970633 7205 tablet_copy_source_session.cc:215] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:58:26.975894 7410 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ee5162104534a39a5cf9ff9a0ab7a4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:58:27.005604 7410 tablet_copy_client.cc:806] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6: tablet copy: Starting download of 0 data blocks...
I20250627 13:58:27.006573 7410 tablet_copy_client.cc:670] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6: tablet copy: Starting download of 1 WAL segments...
I20250627 13:58:27.010921 7410 tablet_copy_client.cc:538] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:58:27.023386 7410 tablet_bootstrap.cc:492] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6: Bootstrap starting.
I20250627 13:58:27.181078 7410 tablet_bootstrap.cc:492] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:58:27.182277 7410 tablet_bootstrap.cc:492] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6: Bootstrap complete.
I20250627 13:58:27.183075 7410 ts_tablet_manager.cc:1397] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6: Time spent bootstrapping tablet: real 0.160s user 0.143s sys 0.017s
I20250627 13:58:27.186452 7410 raft_consensus.cc:357] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: NON_VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: true } }
I20250627 13:58:27.187284 7410 raft_consensus.cc:738] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 96420c429aa34d06b05da75fc24060c6, State: Initialized, Role: LEARNER
I20250627 13:58:27.188024 7410 consensus_queue.cc:260] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: NON_VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: true } }
I20250627 13:58:27.192237 7410 ts_tablet_manager.cc:1428] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20250627 13:58:27.194531 7205 tablet_copy_service.cc:342] P 2d0c422cf8334e25855a5aca8ce93a68: Request end of tablet copy session 96420c429aa34d06b05da75fc24060c6-4ee5162104534a39a5cf9ff9a0ab7a4d received from {username='slave'} at 127.3.173.66:50523
I20250627 13:58:27.194979 7205 tablet_copy_service.cc:434] P 2d0c422cf8334e25855a5aca8ce93a68: ending tablet copy session 96420c429aa34d06b05da75fc24060c6-4ee5162104534a39a5cf9ff9a0ab7a4d on tablet 4ee5162104534a39a5cf9ff9a0ab7a4d with peer 96420c429aa34d06b05da75fc24060c6
I20250627 13:58:27.387344 7330 raft_consensus.cc:1215] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
W20250627 13:58:26.360554 7394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:58:26.359730 7395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:58:27.572899 7396 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250627 13:58:27.573036 7384 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250627 13:58:27.576884 7384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:58:27.579054 7384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:58:27.580395 7384 hybrid_clock.cc:648] HybridClock initialized: now 1751032707580377 us; error 48 us; skew 500 ppm
I20250627 13:58:27.581171 7384 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:58:27.587284 7384 webserver.cc:469] Webserver started at http://127.3.173.67:45421/ using document root <none> and password file <none>
I20250627 13:58:27.588182 7384 fs_manager.cc:362] Metadata directory not provided
I20250627 13:58:27.588411 7384 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:58:27.595875 7384 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20250627 13:58:27.600164 7421 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:58:27.601130 7384 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20250627 13:58:27.601445 7384 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "8f73a64a53924fe6a9338f131b5db19c"
format_stamp: "Formatted at 2025-06-27 13:58:14 on dist-test-slave-rn80"
I20250627 13:58:27.603379 7384 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:58:27.681576 7384 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:58:27.683097 7384 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:58:27.683552 7384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:58:27.686064 7384 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:58:27.691591 7428 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250627 13:58:27.699033 7384 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250627 13:58:27.699294 7384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.001s sys 0.000s
I20250627 13:58:27.699559 7384 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250627 13:58:27.704123 7384 ts_tablet_manager.cc:610] Registered 1 tablets
I20250627 13:58:27.704320 7384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.000s sys 0.003s
I20250627 13:58:27.704680 7428 tablet_bootstrap.cc:492] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c: Bootstrap starting.
I20250627 13:58:27.762681 7428 log.cc:826] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c: Log is configured to *not* fsync() on all Append() calls
I20250627 13:58:27.847043 7428 tablet_bootstrap.cc:492] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:58:27.847889 7428 tablet_bootstrap.cc:492] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c: Bootstrap complete.
I20250627 13:58:27.849897 7428 ts_tablet_manager.cc:1397] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c: Time spent bootstrapping tablet: real 0.146s user 0.120s sys 0.020s
I20250627 13:58:27.867514 7428 raft_consensus.cc:357] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } }
I20250627 13:58:27.870422 7428 raft_consensus.cc:738] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f73a64a53924fe6a9338f131b5db19c, State: Initialized, Role: FOLLOWER
I20250627 13:58:27.871233 7428 consensus_queue.cc:260] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } }
I20250627 13:58:27.871706 7428 raft_consensus.cc:397] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250627 13:58:27.871960 7428 raft_consensus.cc:491] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250627 13:58:27.872236 7428 raft_consensus.cc:3058] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:58:27.877692 7428 raft_consensus.cc:513] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } }
I20250627 13:58:27.878396 7428 leader_election.cc:304] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f73a64a53924fe6a9338f131b5db19c; no voters:
I20250627 13:58:27.880609 7428 leader_election.cc:290] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [CANDIDATE]: Term 2 election: Requested vote from peers
I20250627 13:58:27.880965 7535 raft_consensus.cc:2802] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [term 2 FOLLOWER]: Leader election won for term 2
I20250627 13:58:27.882256 7384 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.67:34731
I20250627 13:58:27.884323 7535 raft_consensus.cc:695] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [term 2 LEADER]: Becoming Leader. State: Replica: 8f73a64a53924fe6a9338f131b5db19c, State: Running, Role: LEADER
I20250627 13:58:27.885703 7428 ts_tablet_manager.cc:1428] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c: Time spent starting tablet: real 0.035s user 0.033s sys 0.001s
I20250627 13:58:27.884995 7536 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.67:34731 every 8 connection(s)
I20250627 13:58:27.885907 7535 consensus_queue.cc:237] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } }
I20250627 13:58:27.887951 7384 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250627 13:58:27.892741 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 7384
I20250627 13:58:27.912709 7540 heartbeater.cc:344] Connected to a master server at 127.3.173.126:45811
I20250627 13:58:27.913100 7540 heartbeater.cc:461] Registering TS with master...
I20250627 13:58:27.913980 7540 heartbeater.cc:507] Master 127.3.173.126:45811 requested a full tablet report, sending...
I20250627 13:58:27.917129 7041 ts_manager.cc:194] Registered new tserver with Master: 8f73a64a53924fe6a9338f131b5db19c (127.3.173.67:34731)
I20250627 13:58:27.918079 7041 catalog_manager.cc:5582] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c reported cstate change: term changed from 0 to 2, leader changed from <none> to 8f73a64a53924fe6a9338f131b5db19c (127.3.173.67), VOTER 8f73a64a53924fe6a9338f131b5db19c (127.3.173.67) added. New cstate: current_term: 2 leader_uuid: "8f73a64a53924fe6a9338f131b5db19c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } health_report { overall_health: HEALTHY } } }
I20250627 13:58:27.924719 3765 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250627 13:58:27.929437 3765 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250627 13:58:27.931808 7041 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.173.67:43215
W20250627 13:58:27.933406 3765 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d740546f249747b29b61418e84a952bb: tablet_id: "d740546f249747b29b61418e84a952bb" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
I20250627 13:58:27.936941 7540 heartbeater.cc:499] Master 127.3.173.126:45811 was elected leader, sending a full tablet report...
I20250627 13:58:27.938496 7551 raft_consensus.cc:1062] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68: attempting to promote NON_VOTER 96420c429aa34d06b05da75fc24060c6 to VOTER
I20250627 13:58:27.940227 7551 consensus_queue.cc:237] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } }
I20250627 13:58:27.945286 7330 raft_consensus.cc:1273] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6 [term 2 LEARNER]: Refusing update from remote peer 2d0c422cf8334e25855a5aca8ce93a68: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20250627 13:58:27.946578 7552 consensus_queue.cc:1035] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:58:27.948977 7491 consensus_queue.cc:237] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } } peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: NON_VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: true } }
I20250627 13:58:27.952754 7538 raft_consensus.cc:2953] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index -1 to 8, NON_VOTER 2d0c422cf8334e25855a5aca8ce93a68 (127.3.173.65) added. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } } peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: NON_VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: true } } }
I20250627 13:58:27.959887 7027 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d740546f249747b29b61418e84a952bb with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250627 13:58:27.960850 7551 raft_consensus.cc:2953] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 96420c429aa34d06b05da75fc24060c6 (127.3.173.66) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } } }
I20250627 13:58:27.960937 7330 raft_consensus.cc:2953] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 96420c429aa34d06b05da75fc24060c6 (127.3.173.66) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } } }
I20250627 13:58:27.964097 7041 catalog_manager.cc:5582] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c reported cstate change: config changed from index -1 to 8, NON_VOTER 2d0c422cf8334e25855a5aca8ce93a68 (127.3.173.65) added. New cstate: current_term: 2 leader_uuid: "8f73a64a53924fe6a9338f131b5db19c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: NON_VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:58:27.977514 7491 consensus_queue.cc:237] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } } peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: NON_VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: true } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: NON_VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: true } }
I20250627 13:58:27.980912 7538 raft_consensus.cc:2953] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 96420c429aa34d06b05da75fc24060c6 (127.3.173.66) added. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } } peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: NON_VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: true } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: NON_VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: true } } }
I20250627 13:58:27.986389 7041 catalog_manager.cc:5582] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6 reported cstate change: config changed from index 8 to 9, 96420c429aa34d06b05da75fc24060c6 (127.3.173.66) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "2d0c422cf8334e25855a5aca8ce93a68" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } } }
I20250627 13:58:27.993487 7027 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d740546f249747b29b61418e84a952bb with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250627 13:58:28.003175 7041 catalog_manager.cc:5582] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c reported cstate change: config changed from index 8 to 9, NON_VOTER 96420c429aa34d06b05da75fc24060c6 (127.3.173.66) added. New cstate: current_term: 2 leader_uuid: "8f73a64a53924fe6a9338f131b5db19c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: NON_VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: NON_VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:58:28.008450 7185 consensus_queue.cc:237] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } } peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: NON_VOTER last_known_addr { host: "127.3.173.67" port: 34731 } attrs { promote: true } }
W20250627 13:58:28.014626 7423 consensus_peers.cc:487] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c -> Peer 2d0c422cf8334e25855a5aca8ce93a68 (127.3.173.65:37021): Couldn't send request to peer 2d0c422cf8334e25855a5aca8ce93a68. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d740546f249747b29b61418e84a952bb. This is attempt 1: this message will repeat every 5th retry.
W20250627 13:58:28.015555 7423 consensus_peers.cc:487] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c -> Peer 96420c429aa34d06b05da75fc24060c6 (127.3.173.66:38459): Couldn't send request to peer 96420c429aa34d06b05da75fc24060c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d740546f249747b29b61418e84a952bb. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:58:28.017310 7330 raft_consensus.cc:1273] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6 [term 2 FOLLOWER]: Refusing update from remote peer 2d0c422cf8334e25855a5aca8ce93a68: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250627 13:58:28.018484 7551 consensus_queue.cc:1035] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20250627 13:58:28.026340 7117 consensus_peers.cc:487] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 -> Peer 8f73a64a53924fe6a9338f131b5db19c (127.3.173.67:34731): Couldn't send request to peer 8f73a64a53924fe6a9338f131b5db19c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4ee5162104534a39a5cf9ff9a0ab7a4d. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:58:28.025583 7559 raft_consensus.cc:2953] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, NON_VOTER 8f73a64a53924fe6a9338f131b5db19c (127.3.173.67) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } } peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: NON_VOTER last_known_addr { host: "127.3.173.67" port: 34731 } attrs { promote: true } } }
I20250627 13:58:28.029258 7330 raft_consensus.cc:2953] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, NON_VOTER 8f73a64a53924fe6a9338f131b5db19c (127.3.173.67) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } } peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: NON_VOTER last_known_addr { host: "127.3.173.67" port: 34731 } attrs { promote: true } } }
I20250627 13:58:28.034904 7027 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4ee5162104534a39a5cf9ff9a0ab7a4d with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250627 13:58:28.038301 7041 catalog_manager.cc:5582] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 reported cstate change: config changed from index 9 to 10, NON_VOTER 8f73a64a53924fe6a9338f131b5db19c (127.3.173.67) added. New cstate: current_term: 2 leader_uuid: "2d0c422cf8334e25855a5aca8ce93a68" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: NON_VOTER last_known_addr { host: "127.3.173.67" port: 34731 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:58:28.425273 7563 ts_tablet_manager.cc:927] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 8f73a64a53924fe6a9338f131b5db19c: Initiating tablet copy from peer 2d0c422cf8334e25855a5aca8ce93a68 (127.3.173.65:37021)
I20250627 13:58:28.427158 7563 tablet_copy_client.cc:323] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 8f73a64a53924fe6a9338f131b5db19c: tablet copy: Beginning tablet copy session from remote peer at address 127.3.173.65:37021
I20250627 13:58:28.428604 7205 tablet_copy_service.cc:140] P 2d0c422cf8334e25855a5aca8ce93a68: Received BeginTabletCopySession request for tablet 4ee5162104534a39a5cf9ff9a0ab7a4d from peer 8f73a64a53924fe6a9338f131b5db19c ({username='slave'} at 127.3.173.67:36999)
I20250627 13:58:28.429016 7205 tablet_copy_service.cc:161] P 2d0c422cf8334e25855a5aca8ce93a68: Beginning new tablet copy session on tablet 4ee5162104534a39a5cf9ff9a0ab7a4d from peer 8f73a64a53924fe6a9338f131b5db19c at {username='slave'} at 127.3.173.67:36999: session id = 8f73a64a53924fe6a9338f131b5db19c-4ee5162104534a39a5cf9ff9a0ab7a4d
I20250627 13:58:28.433058 7205 tablet_copy_source_session.cc:215] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:58:28.435806 7563 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ee5162104534a39a5cf9ff9a0ab7a4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:58:28.448014 7563 tablet_copy_client.cc:806] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 8f73a64a53924fe6a9338f131b5db19c: tablet copy: Starting download of 0 data blocks...
I20250627 13:58:28.448586 7563 tablet_copy_client.cc:670] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 8f73a64a53924fe6a9338f131b5db19c: tablet copy: Starting download of 1 WAL segments...
I20250627 13:58:28.449263 7565 ts_tablet_manager.cc:927] T d740546f249747b29b61418e84a952bb P 96420c429aa34d06b05da75fc24060c6: Initiating tablet copy from peer 8f73a64a53924fe6a9338f131b5db19c (127.3.173.67:34731)
I20250627 13:58:28.450943 7565 tablet_copy_client.cc:323] T d740546f249747b29b61418e84a952bb P 96420c429aa34d06b05da75fc24060c6: tablet copy: Beginning tablet copy session from remote peer at address 127.3.173.67:34731
I20250627 13:58:28.452594 7563 tablet_copy_client.cc:538] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 8f73a64a53924fe6a9338f131b5db19c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:58:28.459720 7563 tablet_bootstrap.cc:492] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 8f73a64a53924fe6a9338f131b5db19c: Bootstrap starting.
I20250627 13:58:28.461354 7511 tablet_copy_service.cc:140] P 8f73a64a53924fe6a9338f131b5db19c: Received BeginTabletCopySession request for tablet d740546f249747b29b61418e84a952bb from peer 96420c429aa34d06b05da75fc24060c6 ({username='slave'} at 127.3.173.66:47311)
I20250627 13:58:28.461966 7511 tablet_copy_service.cc:161] P 8f73a64a53924fe6a9338f131b5db19c: Beginning new tablet copy session on tablet d740546f249747b29b61418e84a952bb from peer 96420c429aa34d06b05da75fc24060c6 at {username='slave'} at 127.3.173.66:47311: session id = 96420c429aa34d06b05da75fc24060c6-d740546f249747b29b61418e84a952bb
I20250627 13:58:28.469766 7511 tablet_copy_source_session.cc:215] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:58:28.473160 7565 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d740546f249747b29b61418e84a952bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:58:28.483762 7565 tablet_copy_client.cc:806] T d740546f249747b29b61418e84a952bb P 96420c429aa34d06b05da75fc24060c6: tablet copy: Starting download of 0 data blocks...
I20250627 13:58:28.484285 7565 tablet_copy_client.cc:670] T d740546f249747b29b61418e84a952bb P 96420c429aa34d06b05da75fc24060c6: tablet copy: Starting download of 1 WAL segments...
I20250627 13:58:28.488209 7565 tablet_copy_client.cc:538] T d740546f249747b29b61418e84a952bb P 96420c429aa34d06b05da75fc24060c6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:58:28.493544 7565 tablet_bootstrap.cc:492] T d740546f249747b29b61418e84a952bb P 96420c429aa34d06b05da75fc24060c6: Bootstrap starting.
I20250627 13:58:28.539347 7563 tablet_bootstrap.cc:492] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 8f73a64a53924fe6a9338f131b5db19c: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:58:28.539994 7563 tablet_bootstrap.cc:492] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 8f73a64a53924fe6a9338f131b5db19c: Bootstrap complete.
I20250627 13:58:28.540445 7563 ts_tablet_manager.cc:1397] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 8f73a64a53924fe6a9338f131b5db19c: Time spent bootstrapping tablet: real 0.081s user 0.067s sys 0.012s
I20250627 13:58:28.542081 7563 raft_consensus.cc:357] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 8f73a64a53924fe6a9338f131b5db19c [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } } peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: NON_VOTER last_known_addr { host: "127.3.173.67" port: 34731 } attrs { promote: true } }
I20250627 13:58:28.542726 7563 raft_consensus.cc:738] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 8f73a64a53924fe6a9338f131b5db19c [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 8f73a64a53924fe6a9338f131b5db19c, State: Initialized, Role: LEARNER
I20250627 13:58:28.543190 7563 consensus_queue.cc:260] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 8f73a64a53924fe6a9338f131b5db19c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } } peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: NON_VOTER last_known_addr { host: "127.3.173.67" port: 34731 } attrs { promote: true } }
I20250627 13:58:28.544592 7563 ts_tablet_manager.cc:1428] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 8f73a64a53924fe6a9338f131b5db19c: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250627 13:58:28.546227 7205 tablet_copy_service.cc:342] P 2d0c422cf8334e25855a5aca8ce93a68: Request end of tablet copy session 8f73a64a53924fe6a9338f131b5db19c-4ee5162104534a39a5cf9ff9a0ab7a4d received from {username='slave'} at 127.3.173.67:36999
I20250627 13:58:28.546530 7205 tablet_copy_service.cc:434] P 2d0c422cf8334e25855a5aca8ce93a68: ending tablet copy session 8f73a64a53924fe6a9338f131b5db19c-4ee5162104534a39a5cf9ff9a0ab7a4d on tablet 4ee5162104534a39a5cf9ff9a0ab7a4d with peer 8f73a64a53924fe6a9338f131b5db19c
I20250627 13:58:28.569597 7565 tablet_bootstrap.cc:492] T d740546f249747b29b61418e84a952bb P 96420c429aa34d06b05da75fc24060c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:58:28.570236 7565 tablet_bootstrap.cc:492] T d740546f249747b29b61418e84a952bb P 96420c429aa34d06b05da75fc24060c6: Bootstrap complete.
I20250627 13:58:28.570653 7565 ts_tablet_manager.cc:1397] T d740546f249747b29b61418e84a952bb P 96420c429aa34d06b05da75fc24060c6: Time spent bootstrapping tablet: real 0.077s user 0.060s sys 0.020s
I20250627 13:58:28.572046 7565 raft_consensus.cc:357] T d740546f249747b29b61418e84a952bb P 96420c429aa34d06b05da75fc24060c6 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } } peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: NON_VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: true } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: NON_VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: true } }
I20250627 13:58:28.572486 7565 raft_consensus.cc:738] T d740546f249747b29b61418e84a952bb P 96420c429aa34d06b05da75fc24060c6 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 96420c429aa34d06b05da75fc24060c6, State: Initialized, Role: LEARNER
I20250627 13:58:28.572896 7565 consensus_queue.cc:260] T d740546f249747b29b61418e84a952bb P 96420c429aa34d06b05da75fc24060c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } } peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: NON_VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: true } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: NON_VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: true } }
I20250627 13:58:28.574590 7565 ts_tablet_manager.cc:1428] T d740546f249747b29b61418e84a952bb P 96420c429aa34d06b05da75fc24060c6: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20250627 13:58:28.576143 7511 tablet_copy_service.cc:342] P 8f73a64a53924fe6a9338f131b5db19c: Request end of tablet copy session 96420c429aa34d06b05da75fc24060c6-d740546f249747b29b61418e84a952bb received from {username='slave'} at 127.3.173.66:47311
I20250627 13:58:28.576553 7511 tablet_copy_service.cc:434] P 8f73a64a53924fe6a9338f131b5db19c: ending tablet copy session 96420c429aa34d06b05da75fc24060c6-d740546f249747b29b61418e84a952bb on tablet d740546f249747b29b61418e84a952bb with peer 96420c429aa34d06b05da75fc24060c6
I20250627 13:58:28.612298 7571 ts_tablet_manager.cc:927] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68: Initiating tablet copy from peer 8f73a64a53924fe6a9338f131b5db19c (127.3.173.67:34731)
I20250627 13:58:28.613577 7571 tablet_copy_client.cc:323] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68: tablet copy: Beginning tablet copy session from remote peer at address 127.3.173.67:34731
I20250627 13:58:28.614878 7511 tablet_copy_service.cc:140] P 8f73a64a53924fe6a9338f131b5db19c: Received BeginTabletCopySession request for tablet d740546f249747b29b61418e84a952bb from peer 2d0c422cf8334e25855a5aca8ce93a68 ({username='slave'} at 127.3.173.65:36333)
I20250627 13:58:28.615300 7511 tablet_copy_service.cc:161] P 8f73a64a53924fe6a9338f131b5db19c: Beginning new tablet copy session on tablet d740546f249747b29b61418e84a952bb from peer 2d0c422cf8334e25855a5aca8ce93a68 at {username='slave'} at 127.3.173.65:36333: session id = 2d0c422cf8334e25855a5aca8ce93a68-d740546f249747b29b61418e84a952bb
I20250627 13:58:28.619338 7511 tablet_copy_source_session.cc:215] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:58:28.621760 7571 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d740546f249747b29b61418e84a952bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:58:28.631292 7571 tablet_copy_client.cc:806] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68: tablet copy: Starting download of 0 data blocks...
I20250627 13:58:28.631770 7571 tablet_copy_client.cc:670] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68: tablet copy: Starting download of 1 WAL segments...
I20250627 13:58:28.634913 7571 tablet_copy_client.cc:538] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:58:28.640265 7571 tablet_bootstrap.cc:492] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68: Bootstrap starting.
I20250627 13:58:28.709589 7571 tablet_bootstrap.cc:492] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:58:28.710216 7571 tablet_bootstrap.cc:492] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68: Bootstrap complete.
I20250627 13:58:28.710626 7571 ts_tablet_manager.cc:1397] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68: Time spent bootstrapping tablet: real 0.071s user 0.064s sys 0.008s
I20250627 13:58:28.712218 7571 raft_consensus.cc:357] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } } peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: NON_VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: true } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: NON_VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: true } }
I20250627 13:58:28.712666 7571 raft_consensus.cc:738] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 2d0c422cf8334e25855a5aca8ce93a68, State: Initialized, Role: LEARNER
I20250627 13:58:28.713022 7571 consensus_queue.cc:260] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } } peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: NON_VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: true } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: NON_VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: true } }
I20250627 13:58:28.714331 7571 ts_tablet_manager.cc:1428] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250627 13:58:28.715713 7511 tablet_copy_service.cc:342] P 8f73a64a53924fe6a9338f131b5db19c: Request end of tablet copy session 2d0c422cf8334e25855a5aca8ce93a68-d740546f249747b29b61418e84a952bb received from {username='slave'} at 127.3.173.65:36333
I20250627 13:58:28.716003 7511 tablet_copy_service.cc:434] P 8f73a64a53924fe6a9338f131b5db19c: ending tablet copy session 2d0c422cf8334e25855a5aca8ce93a68-d740546f249747b29b61418e84a952bb on tablet d740546f249747b29b61418e84a952bb with peer 2d0c422cf8334e25855a5aca8ce93a68
I20250627 13:58:28.749161 7028 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4ee5162104534a39a5cf9ff9a0ab7a4d with cas_config_opid_index 8: aborting the task: latest config opid_index 10; task opid_index 8
I20250627 13:58:28.938088 3765 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 2d0c422cf8334e25855a5aca8ce93a68 to finish bootstrapping
I20250627 13:58:28.952378 3765 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 96420c429aa34d06b05da75fc24060c6 to finish bootstrapping
I20250627 13:58:28.953321 7330 raft_consensus.cc:1215] T d740546f249747b29b61418e84a952bb P 96420c429aa34d06b05da75fc24060c6 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20250627 13:58:28.979549 3765 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 8f73a64a53924fe6a9338f131b5db19c to finish bootstrapping
I20250627 13:58:28.991217 7491 raft_consensus.cc:1215] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 8f73a64a53924fe6a9338f131b5db19c [term 2 LEARNER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20250627 13:58:29.170765 7185 raft_consensus.cc:1215] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20250627 13:58:29.267828 7164 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:58:29.270994 7471 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:58:29.274003 7310 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:58:29.424718 7581 raft_consensus.cc:1062] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68: attempting to promote NON_VOTER 8f73a64a53924fe6a9338f131b5db19c to VOTER
I20250627 13:58:29.426688 7581 consensus_queue.cc:237] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } } peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } attrs { promote: false } }
I20250627 13:58:29.432451 7491 raft_consensus.cc:1273] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 8f73a64a53924fe6a9338f131b5db19c [term 2 LEARNER]: Refusing update from remote peer 2d0c422cf8334e25855a5aca8ce93a68: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250627 13:58:29.433455 7330 raft_consensus.cc:1273] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6 [term 2 FOLLOWER]: Refusing update from remote peer 2d0c422cf8334e25855a5aca8ce93a68: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250627 13:58:29.434269 7580 consensus_queue.cc:1035] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20250627 13:58:29.435130 7581 consensus_queue.cc:1035] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:58:29.451133 7580 raft_consensus.cc:2953] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 8f73a64a53924fe6a9338f131b5db19c (127.3.173.67) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } } peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } attrs { promote: false } } }
I20250627 13:58:29.458330 7330 raft_consensus.cc:2953] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 8f73a64a53924fe6a9338f131b5db19c (127.3.173.67) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } } peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } attrs { promote: false } } }
I20250627 13:58:29.461433 7490 raft_consensus.cc:2953] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 8f73a64a53924fe6a9338f131b5db19c [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 8f73a64a53924fe6a9338f131b5db19c (127.3.173.67) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } } peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } attrs { promote: false } } }
I20250627 13:58:29.478144 7040 catalog_manager.cc:5582] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6 reported cstate change: config changed from index 10 to 11, 8f73a64a53924fe6a9338f131b5db19c (127.3.173.67) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "2d0c422cf8334e25855a5aca8ce93a68" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } } peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } attrs { promote: false } } }
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
e909221bbd254a3b9f691b4c8147b32b | 127.3.173.126:45811 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.3.173.84:33333 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
2d0c422cf8334e25855a5aca8ce93a68 | 127.3.173.65:37021 | HEALTHY | <none> | 1 | 0
8f73a64a53924fe6a9338f131b5db19c | 127.3.173.67:34731 | HEALTHY | <none> | 1 | 0
96420c429aa34d06b05da75fc24060c6 | 127.3.173.66:38459 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.3.173.65 | experimental | 127.3.173.65:37021
local_ip_for_outbound_sockets | 127.3.173.66 | experimental | 127.3.173.66:38459
local_ip_for_outbound_sockets | 127.3.173.67 | experimental | 127.3.173.67:34731
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.3.173.65:37021
server_dump_info_path | /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.173.66:38459
server_dump_info_path | /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.173.67:34731
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.3.173.84:33333 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable1 | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable2 | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 3
Tablets | 3
Replicas | 7
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250627 13:58:29.523804 3765 log_verifier.cc:126] Checking tablet 12047b7d6a5e488c8875843b74f2b661
I20250627 13:58:29.554563 3765 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 12047b7d6a5e488c8875843b74f2b661
I20250627 13:58:29.554831 3765 log_verifier.cc:126] Checking tablet 4ee5162104534a39a5cf9ff9a0ab7a4d
I20250627 13:58:29.563710 7577 raft_consensus.cc:1062] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c: attempting to promote NON_VOTER 96420c429aa34d06b05da75fc24060c6 to VOTER
I20250627 13:58:29.566187 7577 consensus_queue.cc:237] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } } peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: NON_VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: true } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } }
I20250627 13:58:29.572242 7330 raft_consensus.cc:1273] T d740546f249747b29b61418e84a952bb P 96420c429aa34d06b05da75fc24060c6 [term 2 LEARNER]: Refusing update from remote peer 8f73a64a53924fe6a9338f131b5db19c: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250627 13:58:29.572474 7185 raft_consensus.cc:1273] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 [term 2 LEARNER]: Refusing update from remote peer 8f73a64a53924fe6a9338f131b5db19c: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250627 13:58:29.573526 7577 consensus_queue.cc:1035] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [LEADER]: Connected to new peer: Peer: permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:58:29.574162 7538 consensus_queue.cc:1035] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: NON_VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:58:29.580854 7577 raft_consensus.cc:1025] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [term 2 LEADER]: attempt to promote peer 2d0c422cf8334e25855a5aca8ce93a68: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250627 13:58:29.582399 7538 raft_consensus.cc:2953] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 96420c429aa34d06b05da75fc24060c6 (127.3.173.66) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } } peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: NON_VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: true } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } } }
I20250627 13:58:29.586022 7330 raft_consensus.cc:2953] T d740546f249747b29b61418e84a952bb P 96420c429aa34d06b05da75fc24060c6 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 96420c429aa34d06b05da75fc24060c6 (127.3.173.66) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } } peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: NON_VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: true } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } } }
I20250627 13:58:29.591468 7185 raft_consensus.cc:2953] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 [term 2 LEARNER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 96420c429aa34d06b05da75fc24060c6 (127.3.173.66) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } } peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: NON_VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: true } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } } }
I20250627 13:58:29.594149 7041 catalog_manager.cc:5582] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c reported cstate change: config changed from index 9 to 10, 96420c429aa34d06b05da75fc24060c6 (127.3.173.66) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "8f73a64a53924fe6a9338f131b5db19c" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: NON_VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:58:29.602465 7576 raft_consensus.cc:1062] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c: attempting to promote NON_VOTER 2d0c422cf8334e25855a5aca8ce93a68 to VOTER
I20250627 13:58:29.604586 7576 consensus_queue.cc:237] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } } peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: false } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } }
I20250627 13:58:29.611645 7330 raft_consensus.cc:1273] T d740546f249747b29b61418e84a952bb P 96420c429aa34d06b05da75fc24060c6 [term 2 FOLLOWER]: Refusing update from remote peer 8f73a64a53924fe6a9338f131b5db19c: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250627 13:58:29.612098 7185 raft_consensus.cc:1273] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 [term 2 LEARNER]: Refusing update from remote peer 8f73a64a53924fe6a9338f131b5db19c: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250627 13:58:29.613310 7576 consensus_queue.cc:1035] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [LEADER]: Connected to new peer: Peer: permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:58:29.614198 7538 consensus_queue.cc:1035] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:58:29.619992 7538 raft_consensus.cc:2953] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 2d0c422cf8334e25855a5aca8ce93a68 (127.3.173.65) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } } peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: false } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } } }
I20250627 13:58:29.622087 7185 raft_consensus.cc:2953] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 2d0c422cf8334e25855a5aca8ce93a68 (127.3.173.65) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } } peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: false } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } } }
I20250627 13:58:29.626613 7330 raft_consensus.cc:2953] T d740546f249747b29b61418e84a952bb P 96420c429aa34d06b05da75fc24060c6 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 2d0c422cf8334e25855a5aca8ce93a68 (127.3.173.65) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } } peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: false } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } } }
I20250627 13:58:29.637351 7041 catalog_manager.cc:5582] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 reported cstate change: config changed from index 10 to 11, 2d0c422cf8334e25855a5aca8ce93a68 (127.3.173.65) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "8f73a64a53924fe6a9338f131b5db19c" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } } peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: false } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } } }
I20250627 13:58:29.670481 3765 log_verifier.cc:177] Verified matching terms for 11 ops in tablet 4ee5162104534a39a5cf9ff9a0ab7a4d
I20250627 13:58:29.670753 3765 log_verifier.cc:126] Checking tablet d740546f249747b29b61418e84a952bb
I20250627 13:58:29.753535 3765 log_verifier.cc:177] Verified matching terms for 11 ops in tablet d740546f249747b29b61418e84a952bb
I20250627 13:58:29.753973 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 7009
I20250627 13:58:29.783661 3765 minidump.cc:252] Setting minidump size limit to 20M
I20250627 13:58:29.785822 3765 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:58:29.786837 3765 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:58:29.796653 7620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:58:29.798008 7623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:58:29.796828 7621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:58:29.924629 3765 server_base.cc:1048] running on GCE node
I20250627 13:58:29.925757 3765 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250627 13:58:29.925981 3765 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250627 13:58:29.926169 3765 hybrid_clock.cc:648] HybridClock initialized: now 1751032709926149 us; error 0 us; skew 500 ppm
I20250627 13:58:29.926800 3765 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:58:29.930236 3765 webserver.cc:469] Webserver started at http://0.0.0.0:38031/ using document root <none> and password file <none>
I20250627 13:58:29.931026 3765 fs_manager.cc:362] Metadata directory not provided
I20250627 13:58:29.931231 3765 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:58:29.936358 3765 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.001s
I20250627 13:58:29.939950 7628 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:58:29.940927 3765 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20250627 13:58:29.941303 3765 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
uuid: "e909221bbd254a3b9f691b4c8147b32b"
format_stamp: "Formatted at 2025-06-27 13:58:08 on dist-test-slave-rn80"
I20250627 13:58:29.942932 3765 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:58:29.979024 3765 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:58:29.980383 3765 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:58:29.980798 3765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:58:29.990345 3765 sys_catalog.cc:263] Verifying existing consensus state
W20250627 13:58:29.993716 3765 sys_catalog.cc:243] For a single master config, on-disk Raft master: 127.3.173.126:45811 exists but no master address supplied!
I20250627 13:58:29.995676 3765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b: Bootstrap starting.
I20250627 13:58:30.038604 3765 log.cc:826] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b: Log is configured to *not* fsync() on all Append() calls
I20250627 13:58:30.101073 3765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=13 ignored=0} mutations{seen=21 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:58:30.101856 3765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b: Bootstrap complete.
I20250627 13:58:30.114701 3765 raft_consensus.cc:357] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e909221bbd254a3b9f691b4c8147b32b" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 45811 } }
I20250627 13:58:30.115269 3765 raft_consensus.cc:738] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: e909221bbd254a3b9f691b4c8147b32b, State: Initialized, Role: FOLLOWER
I20250627 13:58:30.116009 3765 consensus_queue.cc:260] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e909221bbd254a3b9f691b4c8147b32b" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 45811 } }
I20250627 13:58:30.116480 3765 raft_consensus.cc:397] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [term 3 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250627 13:58:30.116721 3765 raft_consensus.cc:491] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [term 3 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250627 13:58:30.117019 3765 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [term 3 FOLLOWER]: Advancing to term 4
I20250627 13:58:30.122195 3765 raft_consensus.cc:513] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e909221bbd254a3b9f691b4c8147b32b" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 45811 } }
I20250627 13:58:30.122843 3765 leader_election.cc:304] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e909221bbd254a3b9f691b4c8147b32b; no voters:
I20250627 13:58:30.123945 3765 leader_election.cc:290] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [CANDIDATE]: Term 4 election: Requested vote from peers
I20250627 13:58:30.124219 7635 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [term 4 FOLLOWER]: Leader election won for term 4
I20250627 13:58:30.125980 7635 raft_consensus.cc:695] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [term 4 LEADER]: Becoming Leader. State: Replica: e909221bbd254a3b9f691b4c8147b32b, State: Running, Role: LEADER
I20250627 13:58:30.126776 7635 consensus_queue.cc:237] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e909221bbd254a3b9f691b4c8147b32b" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 45811 } }
I20250627 13:58:30.133685 7636 sys_catalog.cc:455] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 leader_uuid: "e909221bbd254a3b9f691b4c8147b32b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e909221bbd254a3b9f691b4c8147b32b" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 45811 } } }
I20250627 13:58:30.134207 7636 sys_catalog.cc:458] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [sys.catalog]: This master's current role is: LEADER
I20250627 13:58:30.134446 7637 sys_catalog.cc:455] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e909221bbd254a3b9f691b4c8147b32b. Latest consensus state: current_term: 4 leader_uuid: "e909221bbd254a3b9f691b4c8147b32b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e909221bbd254a3b9f691b4c8147b32b" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 45811 } } }
I20250627 13:58:30.134975 7637 sys_catalog.cc:458] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [sys.catalog]: This master's current role is: LEADER
I20250627 13:58:30.160741 3765 tablet_replica.cc:331] stopping tablet replica
I20250627 13:58:30.161321 3765 raft_consensus.cc:2241] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [term 4 LEADER]: Raft consensus shutting down.
I20250627 13:58:30.161752 3765 raft_consensus.cc:2270] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [term 4 FOLLOWER]: Raft consensus is shut down!
I20250627 13:58:30.163964 3765 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250627 13:58:30.164426 3765 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250627 13:58:30.188650 3765 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
W20250627 13:58:30.664142 7380 heartbeater.cc:646] Failed to heartbeat to 127.3.173.126:45811 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.173.126:45811: connect: Connection refused (error 111)
W20250627 13:58:30.683723 7540 heartbeater.cc:646] Failed to heartbeat to 127.3.173.126:45811 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.173.126:45811: connect: Connection refused (error 111)
W20250627 13:58:30.686679 7230 heartbeater.cc:646] Failed to heartbeat to 127.3.173.126:45811 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.173.126:45811: connect: Connection refused (error 111)
I20250627 13:58:35.338742 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 7079
I20250627 13:58:35.364771 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 7232
I20250627 13:58:35.390712 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 7384
I20250627 13:58:35.420500 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.173.126:45811
--webserver_interface=127.3.173.126
--webserver_port=40177
--builtin_ntp_servers=127.3.173.84:33333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.173.126:45811 with env {}
W20250627 13:58:35.718423 7711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:58:35.718959 7711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:58:35.719355 7711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:58:35.749854 7711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250627 13:58:35.750190 7711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:58:35.750417 7711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250627 13:58:35.750612 7711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250627 13:58:35.785517 7711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:33333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.173.126:45811
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.173.126:45811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.3.173.126
--webserver_port=40177
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:58:35.786762 7711 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:58:35.788363 7711 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:58:35.803515 7720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:58:35.804019 7717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:58:35.805878 7711 server_base.cc:1048] running on GCE node
W20250627 13:58:35.803550 7718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:58:36.973547 7711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:58:36.976720 7711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:58:36.978219 7711 hybrid_clock.cc:648] HybridClock initialized: now 1751032716978178 us; error 57 us; skew 500 ppm
I20250627 13:58:36.979032 7711 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:58:36.985610 7711 webserver.cc:469] Webserver started at http://127.3.173.126:40177/ using document root <none> and password file <none>
I20250627 13:58:36.986573 7711 fs_manager.cc:362] Metadata directory not provided
I20250627 13:58:36.986766 7711 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:58:36.994374 7711 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.003s
I20250627 13:58:36.998907 7727 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:58:36.999917 7711 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250627 13:58:37.000226 7711 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
uuid: "e909221bbd254a3b9f691b4c8147b32b"
format_stamp: "Formatted at 2025-06-27 13:58:08 on dist-test-slave-rn80"
I20250627 13:58:37.002091 7711 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:58:37.060966 7711 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:58:37.062422 7711 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:58:37.062850 7711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:58:37.132104 7711 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.126:45811
I20250627 13:58:37.132192 7778 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.126:45811 every 8 connection(s)
I20250627 13:58:37.134853 7711 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250627 13:58:37.138688 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 7711
I20250627 13:58:37.140689 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.173.65:37021
--local_ip_for_outbound_sockets=127.3.173.65
--tserver_master_addrs=127.3.173.126:45811
--webserver_port=35721
--webserver_interface=127.3.173.65
--builtin_ntp_servers=127.3.173.84:33333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250627 13:58:37.149333 7779 sys_catalog.cc:263] Verifying existing consensus state
I20250627 13:58:37.154655 7779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b: Bootstrap starting.
I20250627 13:58:37.164453 7779 log.cc:826] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b: Log is configured to *not* fsync() on all Append() calls
I20250627 13:58:37.241075 7779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=15 ignored=0} mutations{seen=23 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:58:37.241912 7779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b: Bootstrap complete.
I20250627 13:58:37.260097 7779 raft_consensus.cc:357] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e909221bbd254a3b9f691b4c8147b32b" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 45811 } }
I20250627 13:58:37.261986 7779 raft_consensus.cc:738] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: e909221bbd254a3b9f691b4c8147b32b, State: Initialized, Role: FOLLOWER
I20250627 13:58:37.262822 7779 consensus_queue.cc:260] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e909221bbd254a3b9f691b4c8147b32b" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 45811 } }
I20250627 13:58:37.263266 7779 raft_consensus.cc:397] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [term 5 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250627 13:58:37.263516 7779 raft_consensus.cc:491] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [term 5 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250627 13:58:37.263797 7779 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [term 5 FOLLOWER]: Advancing to term 6
I20250627 13:58:37.268712 7779 raft_consensus.cc:513] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e909221bbd254a3b9f691b4c8147b32b" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 45811 } }
I20250627 13:58:37.269402 7779 leader_election.cc:304] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e909221bbd254a3b9f691b4c8147b32b; no voters:
I20250627 13:58:37.271503 7779 leader_election.cc:290] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [CANDIDATE]: Term 6 election: Requested vote from peers
I20250627 13:58:37.271986 7783 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [term 6 FOLLOWER]: Leader election won for term 6
I20250627 13:58:37.275130 7783 raft_consensus.cc:695] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [term 6 LEADER]: Becoming Leader. State: Replica: e909221bbd254a3b9f691b4c8147b32b, State: Running, Role: LEADER
I20250627 13:58:37.275990 7783 consensus_queue.cc:237] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 6, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e909221bbd254a3b9f691b4c8147b32b" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 45811 } }
I20250627 13:58:37.276495 7779 sys_catalog.cc:564] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [sys.catalog]: configured and running, proceeding with master startup.
I20250627 13:58:37.287099 7784 sys_catalog.cc:455] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 leader_uuid: "e909221bbd254a3b9f691b4c8147b32b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e909221bbd254a3b9f691b4c8147b32b" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 45811 } } }
I20250627 13:58:37.288792 7784 sys_catalog.cc:458] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [sys.catalog]: This master's current role is: LEADER
I20250627 13:58:37.288211 7785 sys_catalog.cc:455] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e909221bbd254a3b9f691b4c8147b32b. Latest consensus state: current_term: 6 leader_uuid: "e909221bbd254a3b9f691b4c8147b32b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e909221bbd254a3b9f691b4c8147b32b" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 45811 } } }
I20250627 13:58:37.290645 7785 sys_catalog.cc:458] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b [sys.catalog]: This master's current role is: LEADER
I20250627 13:58:37.296476 7790 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250627 13:58:37.312790 7790 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=3c10e1f63a5d4e05a6249bf9a108319a]
I20250627 13:58:37.314944 7790 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=478b9e28b731427baf55d53330dc687c]
I20250627 13:58:37.317070 7790 catalog_manager.cc:671] Loaded metadata for table TestTable [id=cde5c8eb28a843db8f1b71e44a803f6c]
I20250627 13:58:37.327502 7790 tablet_loader.cc:96] loaded metadata for tablet 12047b7d6a5e488c8875843b74f2b661 (table TestTable2 [id=3c10e1f63a5d4e05a6249bf9a108319a])
I20250627 13:58:37.329478 7790 tablet_loader.cc:96] loaded metadata for tablet 4ee5162104534a39a5cf9ff9a0ab7a4d (table TestTable1 [id=478b9e28b731427baf55d53330dc687c])
I20250627 13:58:37.331162 7790 tablet_loader.cc:96] loaded metadata for tablet d740546f249747b29b61418e84a952bb (table TestTable [id=cde5c8eb28a843db8f1b71e44a803f6c])
I20250627 13:58:37.333176 7790 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250627 13:58:37.341078 7790 catalog_manager.cc:1261] Loaded cluster ID: fe7948cebdfa460e9e6e1c303d722e64
I20250627 13:58:37.341537 7790 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250627 13:58:37.353107 7790 catalog_manager.cc:1506] Loading token signing keys...
I20250627 13:58:37.361949 7790 catalog_manager.cc:5966] T 00000000000000000000000000000000 P e909221bbd254a3b9f691b4c8147b32b: Loaded TSK: 0
I20250627 13:58:37.364830 7790 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250627 13:58:37.493921 7781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:58:37.494589 7781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:58:37.495153 7781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:58:37.526211 7781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:58:37.527067 7781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.173.65
I20250627 13:58:37.561383 7781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:33333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.173.65:37021
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.3.173.65
--webserver_port=35721
--tserver_master_addrs=127.3.173.126:45811
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.173.65
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:58:37.562769 7781 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:58:37.564383 7781 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:58:37.582060 7807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:58:37.584499 7806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:58:37.586045 7781 server_base.cc:1048] running on GCE node
W20250627 13:58:37.585531 7809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:58:38.754484 7781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:58:38.757230 7781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:58:38.758726 7781 hybrid_clock.cc:648] HybridClock initialized: now 1751032718758690 us; error 36 us; skew 500 ppm
I20250627 13:58:38.759791 7781 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:58:38.771023 7781 webserver.cc:469] Webserver started at http://127.3.173.65:35721/ using document root <none> and password file <none>
I20250627 13:58:38.772270 7781 fs_manager.cc:362] Metadata directory not provided
I20250627 13:58:38.772552 7781 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:58:38.782764 7781 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.001s
I20250627 13:58:38.787556 7816 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:58:38.788578 7781 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250627 13:58:38.788892 7781 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "2d0c422cf8334e25855a5aca8ce93a68"
format_stamp: "Formatted at 2025-06-27 13:58:10 on dist-test-slave-rn80"
I20250627 13:58:38.790920 7781 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:58:38.842536 7781 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:58:38.843961 7781 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:58:38.844413 7781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:58:38.847419 7781 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:58:38.853710 7823 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250627 13:58:38.865281 7781 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250627 13:58:38.865513 7781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.002s sys 0.001s
I20250627 13:58:38.865808 7781 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250627 13:58:38.871413 7823 tablet_bootstrap.cc:492] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68: Bootstrap starting.
I20250627 13:58:38.876252 7781 ts_tablet_manager.cc:610] Registered 2 tablets
I20250627 13:58:38.876551 7781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.011s user 0.009s sys 0.000s
I20250627 13:58:38.944463 7823 log.cc:826] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68: Log is configured to *not* fsync() on all Append() calls
I20250627 13:58:39.060886 7781 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.65:37021
I20250627 13:58:39.061070 7930 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.65:37021 every 8 connection(s)
I20250627 13:58:39.064558 7781 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250627 13:58:39.075212 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 7781
I20250627 13:58:39.077006 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.173.66:38459
--local_ip_for_outbound_sockets=127.3.173.66
--tserver_master_addrs=127.3.173.126:45811
--webserver_port=36955
--webserver_interface=127.3.173.66
--builtin_ntp_servers=127.3.173.84:33333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250627 13:58:39.081739 7823 tablet_bootstrap.cc:492] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:58:39.083178 7823 tablet_bootstrap.cc:492] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68: Bootstrap complete.
I20250627 13:58:39.085546 7823 ts_tablet_manager.cc:1397] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68: Time spent bootstrapping tablet: real 0.215s user 0.165s sys 0.041s
I20250627 13:58:39.115056 7823 raft_consensus.cc:357] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } } peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: false } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } }
I20250627 13:58:39.118991 7823 raft_consensus.cc:738] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d0c422cf8334e25855a5aca8ce93a68, State: Initialized, Role: FOLLOWER
I20250627 13:58:39.120072 7823 consensus_queue.cc:260] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } } peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: false } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } }
I20250627 13:58:39.143141 7931 heartbeater.cc:344] Connected to a master server at 127.3.173.126:45811
I20250627 13:58:39.143702 7931 heartbeater.cc:461] Registering TS with master...
I20250627 13:58:39.143950 7823 ts_tablet_manager.cc:1428] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68: Time spent starting tablet: real 0.058s user 0.037s sys 0.012s
I20250627 13:58:39.144730 7823 tablet_bootstrap.cc:492] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68: Bootstrap starting.
I20250627 13:58:39.145022 7931 heartbeater.cc:507] Master 127.3.173.126:45811 requested a full tablet report, sending...
I20250627 13:58:39.150509 7744 ts_manager.cc:194] Registered new tserver with Master: 2d0c422cf8334e25855a5aca8ce93a68 (127.3.173.65:37021)
I20250627 13:58:39.156164 7744 catalog_manager.cc:5582] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 reported cstate change: config changed from index -1 to 11, term changed from 0 to 2, VOTER 2d0c422cf8334e25855a5aca8ce93a68 (127.3.173.65) added, VOTER 8f73a64a53924fe6a9338f131b5db19c (127.3.173.67) added, VOTER 96420c429aa34d06b05da75fc24060c6 (127.3.173.66) added. New cstate: current_term: 2 committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } } peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: false } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } } }
I20250627 13:58:39.218281 7744 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.173.65:47247
I20250627 13:58:39.221808 7931 heartbeater.cc:499] Master 127.3.173.126:45811 was elected leader, sending a full tablet report...
I20250627 13:58:39.297506 7823 tablet_bootstrap.cc:492] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:58:39.298455 7823 tablet_bootstrap.cc:492] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68: Bootstrap complete.
I20250627 13:58:39.300074 7823 ts_tablet_manager.cc:1397] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68: Time spent bootstrapping tablet: real 0.156s user 0.120s sys 0.032s
I20250627 13:58:39.302423 7823 raft_consensus.cc:357] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } } peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } attrs { promote: false } }
I20250627 13:58:39.303154 7823 raft_consensus.cc:738] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d0c422cf8334e25855a5aca8ce93a68, State: Initialized, Role: FOLLOWER
I20250627 13:58:39.303776 7823 consensus_queue.cc:260] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } } peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } attrs { promote: false } }
I20250627 13:58:39.305878 7823 ts_tablet_manager.cc:1428] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20250627 13:58:39.459187 7935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:58:39.459638 7935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:58:39.460084 7935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:58:39.493588 7935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:58:39.494448 7935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.173.66
I20250627 13:58:39.529654 7935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:33333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.173.66:38459
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.3.173.66
--webserver_port=36955
--tserver_master_addrs=127.3.173.126:45811
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.173.66
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:58:39.531029 7935 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:58:39.532582 7935 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:58:39.547473 7947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:58:40.435073 7953 raft_consensus.cc:491] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:58:40.435623 7953 raft_consensus.cc:513] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } } peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: false } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } }
I20250627 13:58:40.438225 7953 leader_election.cc:290] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f73a64a53924fe6a9338f131b5db19c (127.3.173.67:34731), 96420c429aa34d06b05da75fc24060c6 (127.3.173.66:38459)
W20250627 13:58:40.443604 7818 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.173.67:34731: connect: Connection refused (error 111)
W20250627 13:58:40.447791 7818 leader_election.cc:336] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8f73a64a53924fe6a9338f131b5db19c (127.3.173.67:34731): Network error: Client connection negotiation failed: client connection to 127.3.173.67:34731: connect: Connection refused (error 111)
W20250627 13:58:40.448884 7818 leader_election.cc:336] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 96420c429aa34d06b05da75fc24060c6 (127.3.173.66:38459): Network error: Client connection negotiation failed: client connection to 127.3.173.66:38459: connect: Connection refused (error 111)
I20250627 13:58:40.449198 7818 leader_election.cc:304] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d0c422cf8334e25855a5aca8ce93a68; no voters: 8f73a64a53924fe6a9338f131b5db19c, 96420c429aa34d06b05da75fc24060c6
I20250627 13:58:40.449826 7953 raft_consensus.cc:2747] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250627 13:58:39.548161 7948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:58:39.550189 7935 server_base.cc:1048] running on GCE node
W20250627 13:58:39.548882 7950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:58:40.704941 7935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:58:40.707194 7935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:58:40.708525 7935 hybrid_clock.cc:648] HybridClock initialized: now 1751032720708505 us; error 43 us; skew 500 ppm
I20250627 13:58:40.709280 7935 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:58:40.715803 7935 webserver.cc:469] Webserver started at http://127.3.173.66:36955/ using document root <none> and password file <none>
I20250627 13:58:40.716688 7935 fs_manager.cc:362] Metadata directory not provided
I20250627 13:58:40.716879 7935 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:58:40.724493 7935 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.001s sys 0.004s
I20250627 13:58:40.729274 7961 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:58:40.730321 7935 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20250627 13:58:40.730645 7935 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "96420c429aa34d06b05da75fc24060c6"
format_stamp: "Formatted at 2025-06-27 13:58:12 on dist-test-slave-rn80"
I20250627 13:58:40.732550 7935 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:58:40.777979 7935 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:58:40.779436 7935 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:58:40.779860 7935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:58:40.782384 7935 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:58:40.787910 7968 ts_tablet_manager.cc:542] Loading tablet metadata (0/3 complete)
I20250627 13:58:40.803529 7935 ts_tablet_manager.cc:579] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20250627 13:58:40.803783 7935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.017s user 0.002s sys 0.000s
I20250627 13:58:40.804069 7935 ts_tablet_manager.cc:594] Registering tablets (0/3 complete)
I20250627 13:58:40.809034 7968 tablet_bootstrap.cc:492] T d740546f249747b29b61418e84a952bb P 96420c429aa34d06b05da75fc24060c6: Bootstrap starting.
I20250627 13:58:40.814218 7935 ts_tablet_manager.cc:610] Registered 3 tablets
I20250627 13:58:40.814407 7935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.010s user 0.008s sys 0.003s
I20250627 13:58:40.860543 7968 log.cc:826] T d740546f249747b29b61418e84a952bb P 96420c429aa34d06b05da75fc24060c6: Log is configured to *not* fsync() on all Append() calls
I20250627 13:58:40.916687 7953 raft_consensus.cc:491] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:58:40.917155 7953 raft_consensus.cc:513] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } } peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } attrs { promote: false } }
I20250627 13:58:40.919050 7953 leader_election.cc:290] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 96420c429aa34d06b05da75fc24060c6 (127.3.173.66:38459), 8f73a64a53924fe6a9338f131b5db19c (127.3.173.67:34731)
W20250627 13:58:40.926136 7818 leader_election.cc:336] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 96420c429aa34d06b05da75fc24060c6 (127.3.173.66:38459): Network error: Client connection negotiation failed: client connection to 127.3.173.66:38459: connect: Connection refused (error 111)
W20250627 13:58:40.926950 7818 leader_election.cc:336] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8f73a64a53924fe6a9338f131b5db19c (127.3.173.67:34731): Network error: Client connection negotiation failed: client connection to 127.3.173.67:34731: connect: Connection refused (error 111)
I20250627 13:58:40.927482 7818 leader_election.cc:304] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d0c422cf8334e25855a5aca8ce93a68; no voters: 8f73a64a53924fe6a9338f131b5db19c, 96420c429aa34d06b05da75fc24060c6
I20250627 13:58:40.928369 7953 raft_consensus.cc:2747] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250627 13:58:40.943938 7968 tablet_bootstrap.cc:492] T d740546f249747b29b61418e84a952bb P 96420c429aa34d06b05da75fc24060c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:58:40.944708 7968 tablet_bootstrap.cc:492] T d740546f249747b29b61418e84a952bb P 96420c429aa34d06b05da75fc24060c6: Bootstrap complete.
I20250627 13:58:40.946398 7968 ts_tablet_manager.cc:1397] T d740546f249747b29b61418e84a952bb P 96420c429aa34d06b05da75fc24060c6: Time spent bootstrapping tablet: real 0.138s user 0.105s sys 0.031s
I20250627 13:58:40.962821 7968 raft_consensus.cc:357] T d740546f249747b29b61418e84a952bb P 96420c429aa34d06b05da75fc24060c6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } } peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: false } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } }
I20250627 13:58:40.965435 7968 raft_consensus.cc:738] T d740546f249747b29b61418e84a952bb P 96420c429aa34d06b05da75fc24060c6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96420c429aa34d06b05da75fc24060c6, State: Initialized, Role: FOLLOWER
I20250627 13:58:40.966392 7968 consensus_queue.cc:260] T d740546f249747b29b61418e84a952bb P 96420c429aa34d06b05da75fc24060c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } } peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: false } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } }
I20250627 13:58:40.970299 7968 ts_tablet_manager.cc:1428] T d740546f249747b29b61418e84a952bb P 96420c429aa34d06b05da75fc24060c6: Time spent starting tablet: real 0.024s user 0.023s sys 0.001s
I20250627 13:58:40.970925 7968 tablet_bootstrap.cc:492] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6: Bootstrap starting.
I20250627 13:58:41.006572 7935 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.66:38459
I20250627 13:58:41.006744 8076 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.66:38459 every 8 connection(s)
I20250627 13:58:41.009189 7935 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250627 13:58:41.011968 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 7935
I20250627 13:58:41.013876 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.173.67:34731
--local_ip_for_outbound_sockets=127.3.173.67
--tserver_master_addrs=127.3.173.126:45811
--webserver_port=45421
--webserver_interface=127.3.173.67
--builtin_ntp_servers=127.3.173.84:33333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250627 13:58:41.034538 8077 heartbeater.cc:344] Connected to a master server at 127.3.173.126:45811
I20250627 13:58:41.034998 8077 heartbeater.cc:461] Registering TS with master...
I20250627 13:58:41.036041 8077 heartbeater.cc:507] Master 127.3.173.126:45811 requested a full tablet report, sending...
I20250627 13:58:41.039708 7744 ts_manager.cc:194] Registered new tserver with Master: 96420c429aa34d06b05da75fc24060c6 (127.3.173.66:38459)
I20250627 13:58:41.043279 7744 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.173.66:40267
I20250627 13:58:41.099406 7968 tablet_bootstrap.cc:492] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:58:41.100278 7968 tablet_bootstrap.cc:492] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6: Bootstrap complete.
I20250627 13:58:41.101773 7968 ts_tablet_manager.cc:1397] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6: Time spent bootstrapping tablet: real 0.131s user 0.098s sys 0.027s
I20250627 13:58:41.103994 7968 raft_consensus.cc:357] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } }
I20250627 13:58:41.104470 7968 raft_consensus.cc:738] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96420c429aa34d06b05da75fc24060c6, State: Initialized, Role: FOLLOWER
I20250627 13:58:41.105010 7968 consensus_queue.cc:260] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } }
I20250627 13:58:41.105504 7968 raft_consensus.cc:397] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250627 13:58:41.105804 7968 raft_consensus.cc:491] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250627 13:58:41.106163 7968 raft_consensus.cc:3058] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6 [term 2 FOLLOWER]: Advancing to term 3
I20250627 13:58:41.114213 7968 raft_consensus.cc:513] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } }
I20250627 13:58:41.115012 7968 leader_election.cc:304] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96420c429aa34d06b05da75fc24060c6; no voters:
I20250627 13:58:41.115669 7968 leader_election.cc:290] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250627 13:58:41.115932 8055 raft_consensus.cc:2802] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6 [term 3 FOLLOWER]: Leader election won for term 3
I20250627 13:58:41.118827 8077 heartbeater.cc:499] Master 127.3.173.126:45811 was elected leader, sending a full tablet report...
I20250627 13:58:41.119117 7968 ts_tablet_manager.cc:1428] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6: Time spent starting tablet: real 0.017s user 0.015s sys 0.001s
I20250627 13:58:41.119824 7968 tablet_bootstrap.cc:492] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6: Bootstrap starting.
I20250627 13:58:41.120869 8055 raft_consensus.cc:695] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6 [term 3 LEADER]: Becoming Leader. State: Replica: 96420c429aa34d06b05da75fc24060c6, State: Running, Role: LEADER
I20250627 13:58:41.121635 8055 consensus_queue.cc:237] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } }
I20250627 13:58:41.139665 7744 catalog_manager.cc:5582] T 12047b7d6a5e488c8875843b74f2b661 P 96420c429aa34d06b05da75fc24060c6 reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "96420c429aa34d06b05da75fc24060c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } health_report { overall_health: HEALTHY } } }
I20250627 13:58:41.235671 7968 tablet_bootstrap.cc:492] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:58:41.236340 7968 tablet_bootstrap.cc:492] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6: Bootstrap complete.
I20250627 13:58:41.237471 7968 ts_tablet_manager.cc:1397] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6: Time spent bootstrapping tablet: real 0.118s user 0.100s sys 0.012s
I20250627 13:58:41.239073 7968 raft_consensus.cc:357] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } } peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } attrs { promote: false } }
I20250627 13:58:41.239496 7968 raft_consensus.cc:738] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96420c429aa34d06b05da75fc24060c6, State: Initialized, Role: FOLLOWER
I20250627 13:58:41.239993 7968 consensus_queue.cc:260] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } } peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } attrs { promote: false } }
I20250627 13:58:41.241604 7968 ts_tablet_manager.cc:1428] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20250627 13:58:41.381031 8082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:58:41.381538 8082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:58:41.382019 8082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:58:41.414278 8082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:58:41.415105 8082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.173.67
I20250627 13:58:41.450089 8082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:33333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.173.67:34731
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.3.173.67
--webserver_port=45421
--tserver_master_addrs=127.3.173.126:45811
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.173.67
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:58:41.451351 8082 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:58:41.452920 8082 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:58:41.477649 8100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:58:42.336511 8103 raft_consensus.cc:491] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:58:42.337153 8103 raft_consensus.cc:513] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } } peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: false } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } }
I20250627 13:58:42.339689 8103 leader_election.cc:290] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8f73a64a53924fe6a9338f131b5db19c (127.3.173.67:34731), 96420c429aa34d06b05da75fc24060c6 (127.3.173.66:38459)
W20250627 13:58:42.364506 7818 leader_election.cc:336] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8f73a64a53924fe6a9338f131b5db19c (127.3.173.67:34731): Network error: Client connection negotiation failed: client connection to 127.3.173.67:34731: connect: Connection refused (error 111)
I20250627 13:58:42.371285 8031 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d740546f249747b29b61418e84a952bb" candidate_uuid: "2d0c422cf8334e25855a5aca8ce93a68" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "96420c429aa34d06b05da75fc24060c6" is_pre_election: true
I20250627 13:58:42.371976 8031 raft_consensus.cc:2466] T d740546f249747b29b61418e84a952bb P 96420c429aa34d06b05da75fc24060c6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d0c422cf8334e25855a5aca8ce93a68 in term 2.
I20250627 13:58:42.373327 7818 leader_election.cc:304] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2d0c422cf8334e25855a5aca8ce93a68, 96420c429aa34d06b05da75fc24060c6; no voters: 8f73a64a53924fe6a9338f131b5db19c
I20250627 13:58:42.373942 8103 raft_consensus.cc:2802] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250627 13:58:42.374279 8103 raft_consensus.cc:491] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:58:42.374542 8103 raft_consensus.cc:3058] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 [term 2 FOLLOWER]: Advancing to term 3
I20250627 13:58:42.380991 8103 raft_consensus.cc:513] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } } peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: false } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } }
I20250627 13:58:42.382463 8103 leader_election.cc:290] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 [CANDIDATE]: Term 3 election: Requested vote from peers 8f73a64a53924fe6a9338f131b5db19c (127.3.173.67:34731), 96420c429aa34d06b05da75fc24060c6 (127.3.173.66:38459)
I20250627 13:58:42.385210 8031 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d740546f249747b29b61418e84a952bb" candidate_uuid: "2d0c422cf8334e25855a5aca8ce93a68" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "96420c429aa34d06b05da75fc24060c6"
I20250627 13:58:42.385753 8031 raft_consensus.cc:3058] T d740546f249747b29b61418e84a952bb P 96420c429aa34d06b05da75fc24060c6 [term 2 FOLLOWER]: Advancing to term 3
W20250627 13:58:42.386344 7818 leader_election.cc:336] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 8f73a64a53924fe6a9338f131b5db19c (127.3.173.67:34731): Network error: Client connection negotiation failed: client connection to 127.3.173.67:34731: connect: Connection refused (error 111)
I20250627 13:58:42.390313 8031 raft_consensus.cc:2466] T d740546f249747b29b61418e84a952bb P 96420c429aa34d06b05da75fc24060c6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d0c422cf8334e25855a5aca8ce93a68 in term 3.
I20250627 13:58:42.391105 7818 leader_election.cc:304] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2d0c422cf8334e25855a5aca8ce93a68, 96420c429aa34d06b05da75fc24060c6; no voters: 8f73a64a53924fe6a9338f131b5db19c
I20250627 13:58:42.391685 8103 raft_consensus.cc:2802] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 [term 3 FOLLOWER]: Leader election won for term 3
I20250627 13:58:42.393596 8103 raft_consensus.cc:695] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 [term 3 LEADER]: Becoming Leader. State: Replica: 2d0c422cf8334e25855a5aca8ce93a68, State: Running, Role: LEADER
I20250627 13:58:42.394616 8103 consensus_queue.cc:237] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } } peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: false } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } }
I20250627 13:58:42.406409 7744 catalog_manager.cc:5582] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 reported cstate change: term changed from 2 to 3, leader changed from <none> to 2d0c422cf8334e25855a5aca8ce93a68 (127.3.173.65). New cstate: current_term: 3 leader_uuid: "2d0c422cf8334e25855a5aca8ce93a68" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:58:42.518031 8114 raft_consensus.cc:491] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:58:42.518510 8114 raft_consensus.cc:513] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } } peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } attrs { promote: false } }
I20250627 13:58:42.520634 8114 leader_election.cc:290] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2d0c422cf8334e25855a5aca8ce93a68 (127.3.173.65:37021), 8f73a64a53924fe6a9338f131b5db19c (127.3.173.67:34731)
W20250627 13:58:42.546721 7963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.173.67:34731: connect: Connection refused (error 111)
I20250627 13:58:42.553295 7886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ee5162104534a39a5cf9ff9a0ab7a4d" candidate_uuid: "96420c429aa34d06b05da75fc24060c6" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "2d0c422cf8334e25855a5aca8ce93a68" is_pre_election: true
I20250627 13:58:42.554174 7886 raft_consensus.cc:2466] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96420c429aa34d06b05da75fc24060c6 in term 2.
W20250627 13:58:42.554648 7963 leader_election.cc:336] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8f73a64a53924fe6a9338f131b5db19c (127.3.173.67:34731): Network error: Client connection negotiation failed: client connection to 127.3.173.67:34731: connect: Connection refused (error 111)
I20250627 13:58:42.555713 7963 leader_election.cc:304] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2d0c422cf8334e25855a5aca8ce93a68, 96420c429aa34d06b05da75fc24060c6; no voters: 8f73a64a53924fe6a9338f131b5db19c
I20250627 13:58:42.556300 8114 raft_consensus.cc:2802] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250627 13:58:42.556617 8114 raft_consensus.cc:491] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:58:42.556862 8114 raft_consensus.cc:3058] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6 [term 2 FOLLOWER]: Advancing to term 3
I20250627 13:58:42.561573 8114 raft_consensus.cc:513] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } } peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } attrs { promote: false } }
I20250627 13:58:42.563040 8114 leader_election.cc:290] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6 [CANDIDATE]: Term 3 election: Requested vote from peers 2d0c422cf8334e25855a5aca8ce93a68 (127.3.173.65:37021), 8f73a64a53924fe6a9338f131b5db19c (127.3.173.67:34731)
I20250627 13:58:42.563805 7886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ee5162104534a39a5cf9ff9a0ab7a4d" candidate_uuid: "96420c429aa34d06b05da75fc24060c6" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "2d0c422cf8334e25855a5aca8ce93a68"
I20250627 13:58:42.564436 7886 raft_consensus.cc:3058] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [term 2 FOLLOWER]: Advancing to term 3
W20250627 13:58:42.568287 7963 leader_election.cc:336] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 8f73a64a53924fe6a9338f131b5db19c (127.3.173.67:34731): Network error: Client connection negotiation failed: client connection to 127.3.173.67:34731: connect: Connection refused (error 111)
I20250627 13:58:42.570573 7886 raft_consensus.cc:2466] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96420c429aa34d06b05da75fc24060c6 in term 3.
I20250627 13:58:42.571395 7963 leader_election.cc:304] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2d0c422cf8334e25855a5aca8ce93a68, 96420c429aa34d06b05da75fc24060c6; no voters: 8f73a64a53924fe6a9338f131b5db19c
I20250627 13:58:42.571955 8114 raft_consensus.cc:2802] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6 [term 3 FOLLOWER]: Leader election won for term 3
I20250627 13:58:42.573848 8114 raft_consensus.cc:695] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6 [term 3 LEADER]: Becoming Leader. State: Replica: 96420c429aa34d06b05da75fc24060c6, State: Running, Role: LEADER
I20250627 13:58:42.574746 8114 consensus_queue.cc:237] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } } peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } attrs { promote: false } }
I20250627 13:58:42.582942 7744 catalog_manager.cc:5582] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6 reported cstate change: term changed from 2 to 3, leader changed from 2d0c422cf8334e25855a5aca8ce93a68 (127.3.173.65) to 96420c429aa34d06b05da75fc24060c6 (127.3.173.66). New cstate: current_term: 3 leader_uuid: "96420c429aa34d06b05da75fc24060c6" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250627 13:58:41.478919 8098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:58:41.477646 8097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:58:42.759297 8099 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1289 milliseconds
I20250627 13:58:42.759436 8082 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250627 13:58:42.760635 8082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:58:42.762764 8082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:58:42.764117 8082 hybrid_clock.cc:648] HybridClock initialized: now 1751032722764065 us; error 60 us; skew 500 ppm
I20250627 13:58:42.764889 8082 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:58:42.771361 8082 webserver.cc:469] Webserver started at http://127.3.173.67:45421/ using document root <none> and password file <none>
I20250627 13:58:42.772233 8082 fs_manager.cc:362] Metadata directory not provided
I20250627 13:58:42.772436 8082 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:58:42.780010 8082 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20250627 13:58:42.784616 8125 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:58:42.785569 8082 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250627 13:58:42.785873 8082 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "8f73a64a53924fe6a9338f131b5db19c"
format_stamp: "Formatted at 2025-06-27 13:58:14 on dist-test-slave-rn80"
I20250627 13:58:42.787853 8082 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:58:42.866132 8082 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:58:42.867491 8082 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:58:42.867874 8082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:58:42.870276 8082 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
W20250627 13:58:42.875797 7818 consensus_peers.cc:487] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 -> Peer 8f73a64a53924fe6a9338f131b5db19c (127.3.173.67:34731): Couldn't send request to peer 8f73a64a53924fe6a9338f131b5db19c. Status: Network error: Client connection negotiation failed: client connection to 127.3.173.67:34731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250627 13:58:42.877315 8132 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250627 13:58:42.888624 8082 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250627 13:58:42.888839 8082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.002s sys 0.001s
I20250627 13:58:42.889073 8082 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250627 13:58:42.894062 8132 tablet_bootstrap.cc:492] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c: Bootstrap starting.
I20250627 13:58:42.898851 8082 ts_tablet_manager.cc:610] Registered 2 tablets
I20250627 13:58:42.899153 8082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.010s user 0.010s sys 0.000s
I20250627 13:58:42.945406 8132 log.cc:826] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c: Log is configured to *not* fsync() on all Append() calls
W20250627 13:58:42.957718 7963 consensus_peers.cc:487] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6 -> Peer 8f73a64a53924fe6a9338f131b5db19c (127.3.173.67:34731): Couldn't send request to peer 8f73a64a53924fe6a9338f131b5db19c. Status: Network error: Client connection negotiation failed: client connection to 127.3.173.67:34731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250627 13:58:43.001086 8031 raft_consensus.cc:1273] T d740546f249747b29b61418e84a952bb P 96420c429aa34d06b05da75fc24060c6 [term 3 FOLLOWER]: Refusing update from remote peer 2d0c422cf8334e25855a5aca8ce93a68: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20250627 13:58:43.002938 8103 consensus_queue.cc:1035] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:58:43.076794 8132 tablet_bootstrap.cc:492] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:58:43.078086 8132 tablet_bootstrap.cc:492] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c: Bootstrap complete.
I20250627 13:58:43.086884 8132 ts_tablet_manager.cc:1397] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c: Time spent bootstrapping tablet: real 0.193s user 0.148s sys 0.024s
I20250627 13:58:43.096611 7886 consensus_queue.cc:237] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: false } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } }
I20250627 13:58:43.102416 8031 raft_consensus.cc:1273] T d740546f249747b29b61418e84a952bb P 96420c429aa34d06b05da75fc24060c6 [term 3 FOLLOWER]: Refusing update from remote peer 2d0c422cf8334e25855a5aca8ce93a68: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20250627 13:58:43.104127 8193 consensus_queue.cc:1035] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20250627 13:58:43.111434 8193 raft_consensus.cc:2953] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 [term 3 LEADER]: Committing config change with OpId 3.13: config changed from index 11 to 13, VOTER 8f73a64a53924fe6a9338f131b5db19c (127.3.173.67) evicted. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: false } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } } }
I20250627 13:58:43.109521 8132 raft_consensus.cc:357] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } } peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: false } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } }
I20250627 13:58:43.112675 8132 raft_consensus.cc:738] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f73a64a53924fe6a9338f131b5db19c, State: Initialized, Role: FOLLOWER
I20250627 13:58:43.113643 8132 consensus_queue.cc:260] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } } peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: false } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } }
I20250627 13:58:43.114493 8031 raft_consensus.cc:2953] T d740546f249747b29b61418e84a952bb P 96420c429aa34d06b05da75fc24060c6 [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 11 to 13, VOTER 8f73a64a53924fe6a9338f131b5db19c (127.3.173.67) evicted. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: false } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } } }
I20250627 13:58:43.124560 7729 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet d740546f249747b29b61418e84a952bb with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:58:43.128481 7744 catalog_manager.cc:5582] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 reported cstate change: config changed from index 11 to 13, VOTER 8f73a64a53924fe6a9338f131b5db19c (127.3.173.67) evicted. New cstate: current_term: 3 leader_uuid: "2d0c422cf8334e25855a5aca8ce93a68" committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:58:43.134040 8132 ts_tablet_manager.cc:1428] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c: Time spent starting tablet: real 0.046s user 0.027s sys 0.008s
I20250627 13:58:43.134810 8132 tablet_bootstrap.cc:492] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 8f73a64a53924fe6a9338f131b5db19c: Bootstrap starting.
I20250627 13:58:43.156167 7886 raft_consensus.cc:1273] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 2d0c422cf8334e25855a5aca8ce93a68 [term 3 FOLLOWER]: Refusing update from remote peer 96420c429aa34d06b05da75fc24060c6: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20250627 13:58:43.157996 8114 consensus_queue.cc:1035] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 96420c429aa34d06b05da75fc24060c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20250627 13:58:43.167791 7744 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet d740546f249747b29b61418e84a952bb on TS 8f73a64a53924fe6a9338f131b5db19c: Not found: failed to reset TS proxy: Could not find TS for UUID 8f73a64a53924fe6a9338f131b5db19c
I20250627 13:58:43.178232 7886 consensus_queue.cc:237] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 11, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: false } }
I20250627 13:58:43.183413 8103 raft_consensus.cc:2953] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 13 to 14, VOTER 96420c429aa34d06b05da75fc24060c6 (127.3.173.66) evicted. New config: { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: false } } }
I20250627 13:58:43.193689 7729 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet d740546f249747b29b61418e84a952bb with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:58:43.198455 7744 catalog_manager.cc:5582] T d740546f249747b29b61418e84a952bb P 2d0c422cf8334e25855a5aca8ce93a68 reported cstate change: config changed from index 13 to 14, VOTER 96420c429aa34d06b05da75fc24060c6 (127.3.173.66) evicted. New cstate: current_term: 3 leader_uuid: "2d0c422cf8334e25855a5aca8ce93a68" committed_config { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250627 13:58:43.221537 7729 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet d740546f249747b29b61418e84a952bb on TS 8f73a64a53924fe6a9338f131b5db19c failed: Not found: failed to reset TS proxy: Could not find TS for UUID 8f73a64a53924fe6a9338f131b5db19c
I20250627 13:58:43.245549 8011 tablet_service.cc:1515] Processing DeleteTablet for tablet d740546f249747b29b61418e84a952bb with delete_type TABLET_DATA_TOMBSTONED (TS 96420c429aa34d06b05da75fc24060c6 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:59056
I20250627 13:58:43.248831 8226 tablet_replica.cc:331] stopping tablet replica
I20250627 13:58:43.249917 8226 raft_consensus.cc:2241] T d740546f249747b29b61418e84a952bb P 96420c429aa34d06b05da75fc24060c6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250627 13:58:43.250772 8226 raft_consensus.cc:2270] T d740546f249747b29b61418e84a952bb P 96420c429aa34d06b05da75fc24060c6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250627 13:58:43.255230 8226 ts_tablet_manager.cc:1905] T d740546f249747b29b61418e84a952bb P 96420c429aa34d06b05da75fc24060c6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:58:43.274690 8226 ts_tablet_manager.cc:1918] T d740546f249747b29b61418e84a952bb P 96420c429aa34d06b05da75fc24060c6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.13
I20250627 13:58:43.275182 8226 log.cc:1199] T d740546f249747b29b61418e84a952bb P 96420c429aa34d06b05da75fc24060c6: Deleting WAL directory at /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal/wals/d740546f249747b29b61418e84a952bb
I20250627 13:58:43.277367 7729 catalog_manager.cc:4928] TS 96420c429aa34d06b05da75fc24060c6 (127.3.173.66:38459): tablet d740546f249747b29b61418e84a952bb (table TestTable [id=cde5c8eb28a843db8f1b71e44a803f6c]) successfully deleted
I20250627 13:58:43.286990 8132 tablet_bootstrap.cc:492] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 8f73a64a53924fe6a9338f131b5db19c: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:58:43.287636 8132 tablet_bootstrap.cc:492] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 8f73a64a53924fe6a9338f131b5db19c: Bootstrap complete.
I20250627 13:58:43.288791 8132 ts_tablet_manager.cc:1397] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 8f73a64a53924fe6a9338f131b5db19c: Time spent bootstrapping tablet: real 0.154s user 0.104s sys 0.033s
I20250627 13:58:43.290518 8132 raft_consensus.cc:357] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 8f73a64a53924fe6a9338f131b5db19c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } } peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } attrs { promote: false } }
I20250627 13:58:43.291013 8132 raft_consensus.cc:738] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 8f73a64a53924fe6a9338f131b5db19c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f73a64a53924fe6a9338f131b5db19c, State: Initialized, Role: FOLLOWER
I20250627 13:58:43.291615 8132 consensus_queue.cc:260] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 8f73a64a53924fe6a9338f131b5db19c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2d0c422cf8334e25855a5aca8ce93a68" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 37021 } } peers { permanent_uuid: "96420c429aa34d06b05da75fc24060c6" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 38459 } attrs { promote: false } } peers { permanent_uuid: "8f73a64a53924fe6a9338f131b5db19c" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 34731 } attrs { promote: false } }
I20250627 13:58:43.293140 8132 ts_tablet_manager.cc:1428] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 8f73a64a53924fe6a9338f131b5db19c: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250627 13:58:43.302603 8082 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.67:34731
I20250627 13:58:43.302733 8255 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.67:34731 every 8 connection(s)
I20250627 13:58:43.305368 8082 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250627 13:58:43.307471 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 8082
I20250627 13:58:43.326728 8256 heartbeater.cc:344] Connected to a master server at 127.3.173.126:45811
I20250627 13:58:43.327210 8256 heartbeater.cc:461] Registering TS with master...
I20250627 13:58:43.328214 8256 heartbeater.cc:507] Master 127.3.173.126:45811 requested a full tablet report, sending...
I20250627 13:58:43.331501 7743 ts_manager.cc:194] Registered new tserver with Master: 8f73a64a53924fe6a9338f131b5db19c (127.3.173.67:34731)
I20250627 13:58:43.334636 7743 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.173.67:47943
I20250627 13:58:43.342770 3765 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250627 13:58:43.346568 3765 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
W20250627 13:58:43.349467 3765 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d740546f249747b29b61418e84a952bb: tablet_id: "d740546f249747b29b61418e84a952bb" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
I20250627 13:58:43.388816 8175 tablet_service.cc:1515] Processing DeleteTablet for tablet d740546f249747b29b61418e84a952bb with delete_type TABLET_DATA_TOMBSTONED (TS 8f73a64a53924fe6a9338f131b5db19c not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:35162
I20250627 13:58:43.393565 8263 tablet_replica.cc:331] stopping tablet replica
I20250627 13:58:43.394234 8263 raft_consensus.cc:2241] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [term 2 FOLLOWER]: Raft consensus shutting down.
I20250627 13:58:43.394651 8263 raft_consensus.cc:2270] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c [term 2 FOLLOWER]: Raft consensus is shut down!
I20250627 13:58:43.396834 8263 ts_tablet_manager.cc:1905] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:58:43.408349 8263 ts_tablet_manager.cc:1918] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.11
I20250627 13:58:43.408761 8263 log.cc:1199] T d740546f249747b29b61418e84a952bb P 8f73a64a53924fe6a9338f131b5db19c: Deleting WAL directory at /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal/wals/d740546f249747b29b61418e84a952bb
I20250627 13:58:43.410533 7729 catalog_manager.cc:4928] TS 8f73a64a53924fe6a9338f131b5db19c (127.3.173.67:34731): tablet d740546f249747b29b61418e84a952bb (table TestTable [id=cde5c8eb28a843db8f1b71e44a803f6c]) successfully deleted
I20250627 13:58:43.504761 8201 raft_consensus.cc:3058] T 4ee5162104534a39a5cf9ff9a0ab7a4d P 8f73a64a53924fe6a9338f131b5db19c [term 2 FOLLOWER]: Advancing to term 3
I20250627 13:58:43.509788 8256 heartbeater.cc:499] Master 127.3.173.126:45811 was elected leader, sending a full tablet report...
W20250627 13:58:44.353062 3765 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d740546f249747b29b61418e84a952bb: tablet_id: "d740546f249747b29b61418e84a952bb" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250627 13:58:45.357343 3765 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d740546f249747b29b61418e84a952bb: tablet_id: "d740546f249747b29b61418e84a952bb" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250627 13:58:46.361492 3765 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d740546f249747b29b61418e84a952bb: tablet_id: "d740546f249747b29b61418e84a952bb" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250627 13:58:47.365047 3765 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d740546f249747b29b61418e84a952bb: tablet_id: "d740546f249747b29b61418e84a952bb" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250627 13:58:48.368487 3765 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d740546f249747b29b61418e84a952bb: tablet_id: "d740546f249747b29b61418e84a952bb" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250627 13:58:49.372128 3765 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d740546f249747b29b61418e84a952bb: tablet_id: "d740546f249747b29b61418e84a952bb" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250627 13:58:50.375609 3765 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d740546f249747b29b61418e84a952bb: tablet_id: "d740546f249747b29b61418e84a952bb" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250627 13:58:51.379103 3765 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d740546f249747b29b61418e84a952bb: tablet_id: "d740546f249747b29b61418e84a952bb" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250627 13:58:52.382522 3765 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d740546f249747b29b61418e84a952bb: tablet_id: "d740546f249747b29b61418e84a952bb" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250627 13:58:53.385725 3765 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d740546f249747b29b61418e84a952bb: tablet_id: "d740546f249747b29b61418e84a952bb" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250627 13:58:54.389045 3765 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d740546f249747b29b61418e84a952bb: tablet_id: "d740546f249747b29b61418e84a952bb" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250627 13:58:55.393165 3765 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d740546f249747b29b61418e84a952bb: tablet_id: "d740546f249747b29b61418e84a952bb" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250627 13:58:56.397271 3765 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d740546f249747b29b61418e84a952bb: tablet_id: "d740546f249747b29b61418e84a952bb" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250627 13:58:57.400668 3765 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d740546f249747b29b61418e84a952bb: tablet_id: "d740546f249747b29b61418e84a952bb" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250627 13:58:58.404202 3765 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d740546f249747b29b61418e84a952bb: tablet_id: "d740546f249747b29b61418e84a952bb" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250627 13:58:59.407621 3765 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d740546f249747b29b61418e84a952bb: tablet_id: "d740546f249747b29b61418e84a952bb" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250627 13:59:00.411054 3765 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d740546f249747b29b61418e84a952bb: tablet_id: "d740546f249747b29b61418e84a952bb" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250627 13:59:01.414044 3765 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d740546f249747b29b61418e84a952bb: tablet_id: "d740546f249747b29b61418e84a952bb" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250627 13:59:02.417371 3765 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet d740546f249747b29b61418e84a952bb: tablet_id: "d740546f249747b29b61418e84a952bb" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/kudu-admin-test.cc:3914: Failure
Failed
Bad status: Not found: not all replicas of tablets comprising table TestTable are registered yet
I20250627 13:59:03.420666 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 7781
I20250627 13:59:03.446015 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 7935
I20250627 13:59:03.469821 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 8082
I20250627 13:59:03.496964 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 7711
2025-06-27T13:59:03Z chronyd exiting
I20250627 13:59:03.545869 3765 test_util.cc:183] -----------------------------------------------
I20250627 13:59:03.546068 3765 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751032624583798-3765-0
[ FAILED ] AdminCliTest.TestRebuildTables (56468 ms)
[----------] 5 tests from AdminCliTest (118895 ms total)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest
[ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4
I20250627 13:59:03.549876 3765 test_util.cc:276] Using random seed: 346774425
I20250627 13:59:03.554170 3765 ts_itest-base.cc:115] Starting cluster with:
I20250627 13:59:03.554327 3765 ts_itest-base.cc:116] --------------
I20250627 13:59:03.554484 3765 ts_itest-base.cc:117] 5 tablet servers
I20250627 13:59:03.554631 3765 ts_itest-base.cc:118] 3 replicas per TS
I20250627 13:59:03.554785 3765 ts_itest-base.cc:119] --------------
2025-06-27T13:59:03Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-27T13:59:03Z Disabled control of system clock
I20250627 13:59:03.596809 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.173.126:45601
--webserver_interface=127.3.173.126
--webserver_port=0
--builtin_ntp_servers=127.3.173.84:43569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.173.126:45601
--raft_prepare_replacement_before_eviction=true with env {}
W20250627 13:59:03.903961 8289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:59:03.904506 8289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:59:03.904891 8289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:59:03.935783 8289 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:59:03.936157 8289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250627 13:59:03.936365 8289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:59:03.936561 8289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250627 13:59:03.936751 8289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250627 13:59:03.972076 8289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:43569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.173.126:45601
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.173.126:45601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.3.173.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:59:03.973305 8289 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:59:03.974967 8289 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:59:03.990402 8296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:59:03.991544 8298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:59:03.991201 8295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:59:03.993448 8289 server_base.cc:1048] running on GCE node
I20250627 13:59:05.172857 8289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:59:05.175485 8289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:59:05.176841 8289 hybrid_clock.cc:648] HybridClock initialized: now 1751032745176802 us; error 60 us; skew 500 ppm
I20250627 13:59:05.177650 8289 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:59:05.184563 8289 webserver.cc:469] Webserver started at http://127.3.173.126:46413/ using document root <none> and password file <none>
I20250627 13:59:05.185531 8289 fs_manager.cc:362] Metadata directory not provided
I20250627 13:59:05.185736 8289 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:59:05.186252 8289 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:59:05.190739 8289 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "4d55884672494c49a6e44332583dbe00"
format_stamp: "Formatted at 2025-06-27 13:59:05 on dist-test-slave-rn80"
I20250627 13:59:05.192137 8289 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "4d55884672494c49a6e44332583dbe00"
format_stamp: "Formatted at 2025-06-27 13:59:05 on dist-test-slave-rn80"
I20250627 13:59:05.199187 8289 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20250627 13:59:05.204725 8306 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:59:05.205762 8289 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20250627 13:59:05.206092 8289 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
uuid: "4d55884672494c49a6e44332583dbe00"
format_stamp: "Formatted at 2025-06-27 13:59:05 on dist-test-slave-rn80"
I20250627 13:59:05.206490 8289 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:59:05.251013 8289 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:59:05.252514 8289 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:59:05.252955 8289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:59:05.322921 8289 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.126:45601
I20250627 13:59:05.322968 8357 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.126:45601 every 8 connection(s)
I20250627 13:59:05.325570 8289 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250627 13:59:05.330576 8358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:59:05.333496 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 8289
I20250627 13:59:05.334061 3765 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250627 13:59:05.354081 8358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d55884672494c49a6e44332583dbe00: Bootstrap starting.
I20250627 13:59:05.360498 8358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d55884672494c49a6e44332583dbe00: Neither blocks nor log segments found. Creating new log.
I20250627 13:59:05.362190 8358 log.cc:826] T 00000000000000000000000000000000 P 4d55884672494c49a6e44332583dbe00: Log is configured to *not* fsync() on all Append() calls
I20250627 13:59:05.366513 8358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d55884672494c49a6e44332583dbe00: No bootstrap required, opened a new log
I20250627 13:59:05.385224 8358 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4d55884672494c49a6e44332583dbe00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d55884672494c49a6e44332583dbe00" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 45601 } }
I20250627 13:59:05.385913 8358 raft_consensus.cc:383] T 00000000000000000000000000000000 P 4d55884672494c49a6e44332583dbe00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:59:05.386179 8358 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4d55884672494c49a6e44332583dbe00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d55884672494c49a6e44332583dbe00, State: Initialized, Role: FOLLOWER
I20250627 13:59:05.386833 8358 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d55884672494c49a6e44332583dbe00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d55884672494c49a6e44332583dbe00" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 45601 } }
I20250627 13:59:05.387351 8358 raft_consensus.cc:397] T 00000000000000000000000000000000 P 4d55884672494c49a6e44332583dbe00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250627 13:59:05.387606 8358 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4d55884672494c49a6e44332583dbe00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250627 13:59:05.387883 8358 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4d55884672494c49a6e44332583dbe00 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:59:05.392256 8358 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4d55884672494c49a6e44332583dbe00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d55884672494c49a6e44332583dbe00" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 45601 } }
I20250627 13:59:05.393152 8358 leader_election.cc:304] T 00000000000000000000000000000000 P 4d55884672494c49a6e44332583dbe00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d55884672494c49a6e44332583dbe00; no voters:
I20250627 13:59:05.394872 8358 leader_election.cc:290] T 00000000000000000000000000000000 P 4d55884672494c49a6e44332583dbe00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250627 13:59:05.395602 8363 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4d55884672494c49a6e44332583dbe00 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:59:05.397727 8363 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4d55884672494c49a6e44332583dbe00 [term 1 LEADER]: Becoming Leader. State: Replica: 4d55884672494c49a6e44332583dbe00, State: Running, Role: LEADER
I20250627 13:59:05.398502 8363 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d55884672494c49a6e44332583dbe00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d55884672494c49a6e44332583dbe00" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 45601 } }
I20250627 13:59:05.399459 8358 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d55884672494c49a6e44332583dbe00 [sys.catalog]: configured and running, proceeding with master startup.
I20250627 13:59:05.408674 8365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d55884672494c49a6e44332583dbe00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d55884672494c49a6e44332583dbe00. Latest consensus state: current_term: 1 leader_uuid: "4d55884672494c49a6e44332583dbe00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d55884672494c49a6e44332583dbe00" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 45601 } } }
I20250627 13:59:05.409296 8365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d55884672494c49a6e44332583dbe00 [sys.catalog]: This master's current role is: LEADER
I20250627 13:59:05.411248 8364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d55884672494c49a6e44332583dbe00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d55884672494c49a6e44332583dbe00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d55884672494c49a6e44332583dbe00" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 45601 } } }
I20250627 13:59:05.411855 8364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d55884672494c49a6e44332583dbe00 [sys.catalog]: This master's current role is: LEADER
I20250627 13:59:05.415555 8372 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250627 13:59:05.426679 8372 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250627 13:59:05.442870 8372 catalog_manager.cc:1349] Generated new cluster ID: e53ab1d4e7d24694be1dbb6196ec622d
I20250627 13:59:05.443207 8372 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250627 13:59:05.495214 8372 catalog_manager.cc:1372] Generated new certificate authority record
I20250627 13:59:05.496673 8372 catalog_manager.cc:1506] Loading token signing keys...
I20250627 13:59:05.520763 8372 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 4d55884672494c49a6e44332583dbe00: Generated new TSK 0
I20250627 13:59:05.521682 8372 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250627 13:59:05.534911 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.173.65:0
--local_ip_for_outbound_sockets=127.3.173.65
--webserver_interface=127.3.173.65
--webserver_port=0
--tserver_master_addrs=127.3.173.126:45601
--builtin_ntp_servers=127.3.173.84:43569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250627 13:59:05.831928 8382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:59:05.832441 8382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:59:05.832957 8382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:59:05.863874 8382 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:59:05.864310 8382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:59:05.865279 8382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.173.65
I20250627 13:59:05.899534 8382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:43569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.173.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.3.173.65
--webserver_port=0
--tserver_master_addrs=127.3.173.126:45601
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.173.65
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:59:05.900875 8382 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:59:05.902593 8382 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:59:05.919777 8389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:59:07.317966 8387 debug-util.cc:398] Leaking SignalData structure 0x7b0800000a80 after lost signal to thread 8382
W20250627 13:59:07.422798 8387 kernel_stack_watchdog.cc:198] Thread 8382 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:641 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250627 13:59:05.920322 8388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:59:07.423450 8382 thread.cc:640] OpenStack (cloud detector) Time spent creating pthread: real 1.508s user 0.474s sys 1.030s
W20250627 13:59:07.423918 8382 thread.cc:606] OpenStack (cloud detector) Time spent starting thread: real 1.509s user 0.474s sys 1.031s
W20250627 13:59:07.423981 8390 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1505 milliseconds
W20250627 13:59:07.425554 8391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:59:07.425518 8382 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250627 13:59:07.428555 8382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:59:07.430931 8382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:59:07.432374 8382 hybrid_clock.cc:648] HybridClock initialized: now 1751032747432311 us; error 72 us; skew 500 ppm
I20250627 13:59:07.433472 8382 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:59:07.441110 8382 webserver.cc:469] Webserver started at http://127.3.173.65:44245/ using document root <none> and password file <none>
I20250627 13:59:07.442397 8382 fs_manager.cc:362] Metadata directory not provided
I20250627 13:59:07.442683 8382 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:59:07.443415 8382 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:59:07.450035 8382 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "446b26061b9148dba78b4411bc7c6c0f"
format_stamp: "Formatted at 2025-06-27 13:59:07 on dist-test-slave-rn80"
I20250627 13:59:07.451637 8382 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "446b26061b9148dba78b4411bc7c6c0f"
format_stamp: "Formatted at 2025-06-27 13:59:07 on dist-test-slave-rn80"
I20250627 13:59:07.460686 8382 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20250627 13:59:07.468567 8398 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:59:07.470161 8382 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20250627 13:59:07.470659 8382 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "446b26061b9148dba78b4411bc7c6c0f"
format_stamp: "Formatted at 2025-06-27 13:59:07 on dist-test-slave-rn80"
I20250627 13:59:07.471259 8382 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:59:07.542641 8382 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:59:07.544487 8382 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:59:07.545032 8382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:59:07.548223 8382 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:59:07.553673 8382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:59:07.553958 8382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:59:07.554286 8382 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:59:07.554493 8382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:59:07.707043 8382 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.65:33387
I20250627 13:59:07.707115 8510 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.65:33387 every 8 connection(s)
I20250627 13:59:07.710636 8382 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250627 13:59:07.719165 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 8382
I20250627 13:59:07.719570 3765 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250627 13:59:07.725373 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.173.66:0
--local_ip_for_outbound_sockets=127.3.173.66
--webserver_interface=127.3.173.66
--webserver_port=0
--tserver_master_addrs=127.3.173.126:45601
--builtin_ntp_servers=127.3.173.84:43569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250627 13:59:07.736713 8511 heartbeater.cc:344] Connected to a master server at 127.3.173.126:45601
I20250627 13:59:07.737154 8511 heartbeater.cc:461] Registering TS with master...
I20250627 13:59:07.738221 8511 heartbeater.cc:507] Master 127.3.173.126:45601 requested a full tablet report, sending...
I20250627 13:59:07.740721 8323 ts_manager.cc:194] Registered new tserver with Master: 446b26061b9148dba78b4411bc7c6c0f (127.3.173.65:33387)
I20250627 13:59:07.742694 8323 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.173.65:44091
W20250627 13:59:08.041497 8515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:59:08.041968 8515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:59:08.042469 8515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:59:08.073429 8515 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:59:08.073855 8515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:59:08.074781 8515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.173.66
I20250627 13:59:08.110800 8515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:43569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.173.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.3.173.66
--webserver_port=0
--tserver_master_addrs=127.3.173.126:45601
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.173.66
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:59:08.112097 8515 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:59:08.113704 8515 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:59:08.131258 8522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:59:08.746155 8511 heartbeater.cc:499] Master 127.3.173.126:45601 was elected leader, sending a full tablet report...
I20250627 13:59:08.132964 8515 server_base.cc:1048] running on GCE node
W20250627 13:59:08.133431 8521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:59:08.133654 8524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:59:09.283512 8515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:59:09.285686 8515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:59:09.287029 8515 hybrid_clock.cc:648] HybridClock initialized: now 1751032749287008 us; error 39 us; skew 500 ppm
I20250627 13:59:09.287815 8515 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:59:09.299212 8515 webserver.cc:469] Webserver started at http://127.3.173.66:39903/ using document root <none> and password file <none>
I20250627 13:59:09.300177 8515 fs_manager.cc:362] Metadata directory not provided
I20250627 13:59:09.300383 8515 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:59:09.300868 8515 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:59:09.305418 8515 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "68e82d6d969e464e84b4fbd14a973ecf"
format_stamp: "Formatted at 2025-06-27 13:59:09 on dist-test-slave-rn80"
I20250627 13:59:09.306586 8515 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "68e82d6d969e464e84b4fbd14a973ecf"
format_stamp: "Formatted at 2025-06-27 13:59:09 on dist-test-slave-rn80"
I20250627 13:59:09.313577 8515 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250627 13:59:09.319255 8531 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:59:09.320351 8515 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250627 13:59:09.320681 8515 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "68e82d6d969e464e84b4fbd14a973ecf"
format_stamp: "Formatted at 2025-06-27 13:59:09 on dist-test-slave-rn80"
I20250627 13:59:09.321022 8515 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:59:09.371150 8515 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:59:09.372570 8515 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:59:09.373004 8515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:59:09.375473 8515 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:59:09.379504 8515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:59:09.379752 8515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:59:09.379994 8515 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:59:09.380152 8515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:59:09.514896 8515 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.66:37589
I20250627 13:59:09.515026 8643 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.66:37589 every 8 connection(s)
I20250627 13:59:09.517378 8515 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250627 13:59:09.524428 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 8515
I20250627 13:59:09.524883 3765 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250627 13:59:09.530720 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.173.67:0
--local_ip_for_outbound_sockets=127.3.173.67
--webserver_interface=127.3.173.67
--webserver_port=0
--tserver_master_addrs=127.3.173.126:45601
--builtin_ntp_servers=127.3.173.84:43569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250627 13:59:09.539026 8644 heartbeater.cc:344] Connected to a master server at 127.3.173.126:45601
I20250627 13:59:09.539463 8644 heartbeater.cc:461] Registering TS with master...
I20250627 13:59:09.540401 8644 heartbeater.cc:507] Master 127.3.173.126:45601 requested a full tablet report, sending...
I20250627 13:59:09.542527 8323 ts_manager.cc:194] Registered new tserver with Master: 68e82d6d969e464e84b4fbd14a973ecf (127.3.173.66:37589)
I20250627 13:59:09.543756 8323 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.173.66:52925
W20250627 13:59:09.820818 8648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:59:09.821247 8648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:59:09.821801 8648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:59:09.852075 8648 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:59:09.852492 8648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:59:09.853259 8648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.173.67
I20250627 13:59:09.887619 8648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:43569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.173.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.3.173.67
--webserver_port=0
--tserver_master_addrs=127.3.173.126:45601
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.173.67
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:59:09.888885 8648 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:59:09.890479 8648 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:59:09.907384 8655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:59:10.546630 8644 heartbeater.cc:499] Master 127.3.173.126:45601 was elected leader, sending a full tablet report...
W20250627 13:59:09.907459 8654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:59:09.907380 8657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:59:11.062543 8656 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250627 13:59:11.062644 8648 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250627 13:59:11.067045 8648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:59:11.069749 8648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:59:11.071166 8648 hybrid_clock.cc:648] HybridClock initialized: now 1751032751071122 us; error 61 us; skew 500 ppm
I20250627 13:59:11.071921 8648 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:59:11.078686 8648 webserver.cc:469] Webserver started at http://127.3.173.67:40115/ using document root <none> and password file <none>
I20250627 13:59:11.079634 8648 fs_manager.cc:362] Metadata directory not provided
I20250627 13:59:11.079836 8648 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:59:11.080278 8648 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:59:11.084478 8648 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "64832ff33e914a3789ba59159c816e2a"
format_stamp: "Formatted at 2025-06-27 13:59:11 on dist-test-slave-rn80"
I20250627 13:59:11.085484 8648 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "64832ff33e914a3789ba59159c816e2a"
format_stamp: "Formatted at 2025-06-27 13:59:11 on dist-test-slave-rn80"
I20250627 13:59:11.092154 8648 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20250627 13:59:11.097486 8664 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:59:11.098486 8648 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250627 13:59:11.098798 8648 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "64832ff33e914a3789ba59159c816e2a"
format_stamp: "Formatted at 2025-06-27 13:59:11 on dist-test-slave-rn80"
I20250627 13:59:11.099128 8648 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:59:11.160672 8648 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:59:11.162066 8648 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:59:11.162526 8648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:59:11.164974 8648 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:59:11.168890 8648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:59:11.169106 8648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:59:11.169353 8648 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:59:11.169513 8648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:59:11.298712 8648 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.67:38495
I20250627 13:59:11.298802 8776 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.67:38495 every 8 connection(s)
I20250627 13:59:11.301200 8648 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250627 13:59:11.303719 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 8648
I20250627 13:59:11.304253 3765 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250627 13:59:11.312264 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.173.68:0
--local_ip_for_outbound_sockets=127.3.173.68
--webserver_interface=127.3.173.68
--webserver_port=0
--tserver_master_addrs=127.3.173.126:45601
--builtin_ntp_servers=127.3.173.84:43569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250627 13:59:11.326467 8777 heartbeater.cc:344] Connected to a master server at 127.3.173.126:45601
I20250627 13:59:11.326910 8777 heartbeater.cc:461] Registering TS with master...
I20250627 13:59:11.327920 8777 heartbeater.cc:507] Master 127.3.173.126:45601 requested a full tablet report, sending...
I20250627 13:59:11.329911 8323 ts_manager.cc:194] Registered new tserver with Master: 64832ff33e914a3789ba59159c816e2a (127.3.173.67:38495)
I20250627 13:59:11.331131 8323 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.173.67:35993
W20250627 13:59:11.621862 8781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:59:11.622402 8781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:59:11.622895 8781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:59:11.653615 8781 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:59:11.653978 8781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:59:11.654716 8781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.173.68
I20250627 13:59:11.688107 8781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:43569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.173.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.3.173.68
--webserver_port=0
--tserver_master_addrs=127.3.173.126:45601
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.173.68
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:59:11.689391 8781 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:59:11.690933 8781 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:59:11.708073 8790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:59:12.334128 8777 heartbeater.cc:499] Master 127.3.173.126:45601 was elected leader, sending a full tablet report...
W20250627 13:59:11.708081 8787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:59:11.710152 8781 server_base.cc:1048] running on GCE node
W20250627 13:59:11.709009 8788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:59:12.860420 8781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:59:12.863327 8781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:59:12.864861 8781 hybrid_clock.cc:648] HybridClock initialized: now 1751032752864807 us; error 48 us; skew 500 ppm
I20250627 13:59:12.866046 8781 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:59:12.874173 8781 webserver.cc:469] Webserver started at http://127.3.173.68:39223/ using document root <none> and password file <none>
I20250627 13:59:12.875515 8781 fs_manager.cc:362] Metadata directory not provided
I20250627 13:59:12.875797 8781 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:59:12.876438 8781 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:59:12.882710 8781 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "2ecd783a8f22438887e4262680671253"
format_stamp: "Formatted at 2025-06-27 13:59:12 on dist-test-slave-rn80"
I20250627 13:59:12.883747 8781 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "2ecd783a8f22438887e4262680671253"
format_stamp: "Formatted at 2025-06-27 13:59:12 on dist-test-slave-rn80"
I20250627 13:59:12.890547 8781 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250627 13:59:12.895766 8797 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:59:12.896737 8781 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250627 13:59:12.897043 8781 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "2ecd783a8f22438887e4262680671253"
format_stamp: "Formatted at 2025-06-27 13:59:12 on dist-test-slave-rn80"
I20250627 13:59:12.897357 8781 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:59:12.946383 8781 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:59:12.947778 8781 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:59:12.948200 8781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:59:12.950977 8781 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:59:12.954890 8781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:59:12.955097 8781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:59:12.955335 8781 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:59:12.955485 8781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:59:13.086440 8781 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.68:39775
I20250627 13:59:13.086539 8909 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.68:39775 every 8 connection(s)
I20250627 13:59:13.088866 8781 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250627 13:59:13.091686 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 8781
I20250627 13:59:13.092211 3765 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250627 13:59:13.099444 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.173.69:0
--local_ip_for_outbound_sockets=127.3.173.69
--webserver_interface=127.3.173.69
--webserver_port=0
--tserver_master_addrs=127.3.173.126:45601
--builtin_ntp_servers=127.3.173.84:43569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250627 13:59:13.109726 8910 heartbeater.cc:344] Connected to a master server at 127.3.173.126:45601
I20250627 13:59:13.110293 8910 heartbeater.cc:461] Registering TS with master...
I20250627 13:59:13.111575 8910 heartbeater.cc:507] Master 127.3.173.126:45601 requested a full tablet report, sending...
I20250627 13:59:13.113942 8323 ts_manager.cc:194] Registered new tserver with Master: 2ecd783a8f22438887e4262680671253 (127.3.173.68:39775)
I20250627 13:59:13.115248 8323 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.173.68:40879
W20250627 13:59:13.407464 8914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:59:13.407943 8914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:59:13.408427 8914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:59:13.437943 8914 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:59:13.438418 8914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:59:13.439241 8914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.173.69
I20250627 13:59:13.475793 8914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:43569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.173.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.3.173.69
--webserver_port=0
--tserver_master_addrs=127.3.173.126:45601
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.173.69
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:59:13.477133 8914 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:59:13.478791 8914 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:59:13.495436 8921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:59:14.118898 8910 heartbeater.cc:499] Master 127.3.173.126:45601 was elected leader, sending a full tablet report...
W20250627 13:59:13.496341 8920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:59:13.495471 8923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:59:14.651919 8922 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250627 13:59:14.652114 8914 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250627 13:59:14.656404 8914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:59:14.659116 8914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:59:14.660586 8914 hybrid_clock.cc:648] HybridClock initialized: now 1751032754660556 us; error 60 us; skew 500 ppm
I20250627 13:59:14.661367 8914 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:59:14.676394 8914 webserver.cc:469] Webserver started at http://127.3.173.69:43617/ using document root <none> and password file <none>
I20250627 13:59:14.677320 8914 fs_manager.cc:362] Metadata directory not provided
I20250627 13:59:14.677524 8914 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:59:14.677994 8914 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:59:14.682083 8914 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "07285654123141ec9b8b9bcc1ae30716"
format_stamp: "Formatted at 2025-06-27 13:59:14 on dist-test-slave-rn80"
I20250627 13:59:14.683216 8914 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "07285654123141ec9b8b9bcc1ae30716"
format_stamp: "Formatted at 2025-06-27 13:59:14 on dist-test-slave-rn80"
I20250627 13:59:14.689945 8914 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250627 13:59:14.695246 8931 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:59:14.696218 8914 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.005s
I20250627 13:59:14.696524 8914 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "07285654123141ec9b8b9bcc1ae30716"
format_stamp: "Formatted at 2025-06-27 13:59:14 on dist-test-slave-rn80"
I20250627 13:59:14.696838 8914 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:59:14.745236 8914 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:59:14.746701 8914 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:59:14.747138 8914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:59:14.750165 8914 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:59:14.754137 8914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:59:14.754365 8914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250627 13:59:14.754612 8914 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:59:14.754771 8914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:59:14.903862 8914 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.69:32899
I20250627 13:59:14.903970 9044 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.69:32899 every 8 connection(s)
I20250627 13:59:14.906646 8914 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250627 13:59:14.908505 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 8914
I20250627 13:59:14.909286 3765 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250627 13:59:14.931653 9045 heartbeater.cc:344] Connected to a master server at 127.3.173.126:45601
I20250627 13:59:14.932201 9045 heartbeater.cc:461] Registering TS with master...
I20250627 13:59:14.933460 9045 heartbeater.cc:507] Master 127.3.173.126:45601 requested a full tablet report, sending...
I20250627 13:59:14.935472 8323 ts_manager.cc:194] Registered new tserver with Master: 07285654123141ec9b8b9bcc1ae30716 (127.3.173.69:32899)
I20250627 13:59:14.937222 8323 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.173.69:41481
I20250627 13:59:14.947423 3765 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250627 13:59:14.984036 8323 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48574:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250627 13:59:15.072983 8979 tablet_service.cc:1468] Processing CreateTablet for tablet 385eb017c5734053b4d4b703b69de506 (DEFAULT_TABLE table=TestTable [id=0b3d4897af7340f189ed75abd63c9fe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250627 13:59:15.074873 8979 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 385eb017c5734053b4d4b703b69de506. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:59:15.078531 8446 tablet_service.cc:1468] Processing CreateTablet for tablet 385eb017c5734053b4d4b703b69de506 (DEFAULT_TABLE table=TestTable [id=0b3d4897af7340f189ed75abd63c9fe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250627 13:59:15.081239 8446 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 385eb017c5734053b4d4b703b69de506. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:59:15.081678 8712 tablet_service.cc:1468] Processing CreateTablet for tablet 385eb017c5734053b4d4b703b69de506 (DEFAULT_TABLE table=TestTable [id=0b3d4897af7340f189ed75abd63c9fe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250627 13:59:15.083097 8712 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 385eb017c5734053b4d4b703b69de506. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:59:15.105113 9064 tablet_bootstrap.cc:492] T 385eb017c5734053b4d4b703b69de506 P 07285654123141ec9b8b9bcc1ae30716: Bootstrap starting.
I20250627 13:59:15.114523 9064 tablet_bootstrap.cc:654] T 385eb017c5734053b4d4b703b69de506 P 07285654123141ec9b8b9bcc1ae30716: Neither blocks nor log segments found. Creating new log.
I20250627 13:59:15.115320 9065 tablet_bootstrap.cc:492] T 385eb017c5734053b4d4b703b69de506 P 446b26061b9148dba78b4411bc7c6c0f: Bootstrap starting.
I20250627 13:59:15.118008 9064 log.cc:826] T 385eb017c5734053b4d4b703b69de506 P 07285654123141ec9b8b9bcc1ae30716: Log is configured to *not* fsync() on all Append() calls
I20250627 13:59:15.119990 9066 tablet_bootstrap.cc:492] T 385eb017c5734053b4d4b703b69de506 P 64832ff33e914a3789ba59159c816e2a: Bootstrap starting.
I20250627 13:59:15.123910 9065 tablet_bootstrap.cc:654] T 385eb017c5734053b4d4b703b69de506 P 446b26061b9148dba78b4411bc7c6c0f: Neither blocks nor log segments found. Creating new log.
I20250627 13:59:15.126498 9064 tablet_bootstrap.cc:492] T 385eb017c5734053b4d4b703b69de506 P 07285654123141ec9b8b9bcc1ae30716: No bootstrap required, opened a new log
I20250627 13:59:15.127059 9064 ts_tablet_manager.cc:1397] T 385eb017c5734053b4d4b703b69de506 P 07285654123141ec9b8b9bcc1ae30716: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.011s
I20250627 13:59:15.126672 9065 log.cc:826] T 385eb017c5734053b4d4b703b69de506 P 446b26061b9148dba78b4411bc7c6c0f: Log is configured to *not* fsync() on all Append() calls
I20250627 13:59:15.129673 9066 tablet_bootstrap.cc:654] T 385eb017c5734053b4d4b703b69de506 P 64832ff33e914a3789ba59159c816e2a: Neither blocks nor log segments found. Creating new log.
I20250627 13:59:15.132084 9066 log.cc:826] T 385eb017c5734053b4d4b703b69de506 P 64832ff33e914a3789ba59159c816e2a: Log is configured to *not* fsync() on all Append() calls
I20250627 13:59:15.134565 9065 tablet_bootstrap.cc:492] T 385eb017c5734053b4d4b703b69de506 P 446b26061b9148dba78b4411bc7c6c0f: No bootstrap required, opened a new log
I20250627 13:59:15.135098 9065 ts_tablet_manager.cc:1397] T 385eb017c5734053b4d4b703b69de506 P 446b26061b9148dba78b4411bc7c6c0f: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.012s
I20250627 13:59:15.137017 9066 tablet_bootstrap.cc:492] T 385eb017c5734053b4d4b703b69de506 P 64832ff33e914a3789ba59159c816e2a: No bootstrap required, opened a new log
I20250627 13:59:15.137388 9066 ts_tablet_manager.cc:1397] T 385eb017c5734053b4d4b703b69de506 P 64832ff33e914a3789ba59159c816e2a: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.005s
I20250627 13:59:15.154871 9064 raft_consensus.cc:357] T 385eb017c5734053b4d4b703b69de506 P 07285654123141ec9b8b9bcc1ae30716 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07285654123141ec9b8b9bcc1ae30716" member_type: VOTER last_known_addr { host: "127.3.173.69" port: 32899 } } peers { permanent_uuid: "446b26061b9148dba78b4411bc7c6c0f" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 33387 } } peers { permanent_uuid: "64832ff33e914a3789ba59159c816e2a" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 38495 } }
I20250627 13:59:15.155834 9064 raft_consensus.cc:383] T 385eb017c5734053b4d4b703b69de506 P 07285654123141ec9b8b9bcc1ae30716 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:59:15.156181 9064 raft_consensus.cc:738] T 385eb017c5734053b4d4b703b69de506 P 07285654123141ec9b8b9bcc1ae30716 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07285654123141ec9b8b9bcc1ae30716, State: Initialized, Role: FOLLOWER
I20250627 13:59:15.157123 9064 consensus_queue.cc:260] T 385eb017c5734053b4d4b703b69de506 P 07285654123141ec9b8b9bcc1ae30716 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07285654123141ec9b8b9bcc1ae30716" member_type: VOTER last_known_addr { host: "127.3.173.69" port: 32899 } } peers { permanent_uuid: "446b26061b9148dba78b4411bc7c6c0f" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 33387 } } peers { permanent_uuid: "64832ff33e914a3789ba59159c816e2a" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 38495 } }
I20250627 13:59:15.157482 9066 raft_consensus.cc:357] T 385eb017c5734053b4d4b703b69de506 P 64832ff33e914a3789ba59159c816e2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07285654123141ec9b8b9bcc1ae30716" member_type: VOTER last_known_addr { host: "127.3.173.69" port: 32899 } } peers { permanent_uuid: "446b26061b9148dba78b4411bc7c6c0f" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 33387 } } peers { permanent_uuid: "64832ff33e914a3789ba59159c816e2a" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 38495 } }
I20250627 13:59:15.158215 9066 raft_consensus.cc:383] T 385eb017c5734053b4d4b703b69de506 P 64832ff33e914a3789ba59159c816e2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:59:15.158540 9066 raft_consensus.cc:738] T 385eb017c5734053b4d4b703b69de506 P 64832ff33e914a3789ba59159c816e2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64832ff33e914a3789ba59159c816e2a, State: Initialized, Role: FOLLOWER
I20250627 13:59:15.159718 9066 consensus_queue.cc:260] T 385eb017c5734053b4d4b703b69de506 P 64832ff33e914a3789ba59159c816e2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07285654123141ec9b8b9bcc1ae30716" member_type: VOTER last_known_addr { host: "127.3.173.69" port: 32899 } } peers { permanent_uuid: "446b26061b9148dba78b4411bc7c6c0f" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 33387 } } peers { permanent_uuid: "64832ff33e914a3789ba59159c816e2a" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 38495 } }
I20250627 13:59:15.160468 9045 heartbeater.cc:499] Master 127.3.173.126:45601 was elected leader, sending a full tablet report...
I20250627 13:59:15.162515 9065 raft_consensus.cc:357] T 385eb017c5734053b4d4b703b69de506 P 446b26061b9148dba78b4411bc7c6c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07285654123141ec9b8b9bcc1ae30716" member_type: VOTER last_known_addr { host: "127.3.173.69" port: 32899 } } peers { permanent_uuid: "446b26061b9148dba78b4411bc7c6c0f" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 33387 } } peers { permanent_uuid: "64832ff33e914a3789ba59159c816e2a" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 38495 } }
I20250627 13:59:15.163522 9065 raft_consensus.cc:383] T 385eb017c5734053b4d4b703b69de506 P 446b26061b9148dba78b4411bc7c6c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:59:15.163882 9066 ts_tablet_manager.cc:1428] T 385eb017c5734053b4d4b703b69de506 P 64832ff33e914a3789ba59159c816e2a: Time spent starting tablet: real 0.026s user 0.020s sys 0.007s
I20250627 13:59:15.163924 9065 raft_consensus.cc:738] T 385eb017c5734053b4d4b703b69de506 P 446b26061b9148dba78b4411bc7c6c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 446b26061b9148dba78b4411bc7c6c0f, State: Initialized, Role: FOLLOWER
I20250627 13:59:15.165014 9064 ts_tablet_manager.cc:1428] T 385eb017c5734053b4d4b703b69de506 P 07285654123141ec9b8b9bcc1ae30716: Time spent starting tablet: real 0.038s user 0.029s sys 0.005s
I20250627 13:59:15.165911 9065 consensus_queue.cc:260] T 385eb017c5734053b4d4b703b69de506 P 446b26061b9148dba78b4411bc7c6c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07285654123141ec9b8b9bcc1ae30716" member_type: VOTER last_known_addr { host: "127.3.173.69" port: 32899 } } peers { permanent_uuid: "446b26061b9148dba78b4411bc7c6c0f" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 33387 } } peers { permanent_uuid: "64832ff33e914a3789ba59159c816e2a" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 38495 } }
W20250627 13:59:15.167605 9046 tablet.cc:2378] T 385eb017c5734053b4d4b703b69de506 P 07285654123141ec9b8b9bcc1ae30716: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250627 13:59:15.172147 9065 ts_tablet_manager.cc:1428] T 385eb017c5734053b4d4b703b69de506 P 446b26061b9148dba78b4411bc7c6c0f: Time spent starting tablet: real 0.037s user 0.031s sys 0.002s
W20250627 13:59:15.224362 8512 tablet.cc:2378] T 385eb017c5734053b4d4b703b69de506 P 446b26061b9148dba78b4411bc7c6c0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250627 13:59:15.311054 8778 tablet.cc:2378] T 385eb017c5734053b4d4b703b69de506 P 64832ff33e914a3789ba59159c816e2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250627 13:59:15.448526 9071 raft_consensus.cc:491] T 385eb017c5734053b4d4b703b69de506 P 64832ff33e914a3789ba59159c816e2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:59:15.448982 9071 raft_consensus.cc:513] T 385eb017c5734053b4d4b703b69de506 P 64832ff33e914a3789ba59159c816e2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07285654123141ec9b8b9bcc1ae30716" member_type: VOTER last_known_addr { host: "127.3.173.69" port: 32899 } } peers { permanent_uuid: "446b26061b9148dba78b4411bc7c6c0f" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 33387 } } peers { permanent_uuid: "64832ff33e914a3789ba59159c816e2a" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 38495 } }
I20250627 13:59:15.451478 9071 leader_election.cc:290] T 385eb017c5734053b4d4b703b69de506 P 64832ff33e914a3789ba59159c816e2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07285654123141ec9b8b9bcc1ae30716 (127.3.173.69:32899), 446b26061b9148dba78b4411bc7c6c0f (127.3.173.65:33387)
I20250627 13:59:15.464044 8999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "385eb017c5734053b4d4b703b69de506" candidate_uuid: "64832ff33e914a3789ba59159c816e2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07285654123141ec9b8b9bcc1ae30716" is_pre_election: true
I20250627 13:59:15.464486 8466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "385eb017c5734053b4d4b703b69de506" candidate_uuid: "64832ff33e914a3789ba59159c816e2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446b26061b9148dba78b4411bc7c6c0f" is_pre_election: true
I20250627 13:59:15.464942 8999 raft_consensus.cc:2466] T 385eb017c5734053b4d4b703b69de506 P 07285654123141ec9b8b9bcc1ae30716 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64832ff33e914a3789ba59159c816e2a in term 0.
I20250627 13:59:15.465340 8466 raft_consensus.cc:2466] T 385eb017c5734053b4d4b703b69de506 P 446b26061b9148dba78b4411bc7c6c0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64832ff33e914a3789ba59159c816e2a in term 0.
I20250627 13:59:15.466348 8668 leader_election.cc:304] T 385eb017c5734053b4d4b703b69de506 P 64832ff33e914a3789ba59159c816e2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07285654123141ec9b8b9bcc1ae30716, 64832ff33e914a3789ba59159c816e2a; no voters:
I20250627 13:59:15.467240 9071 raft_consensus.cc:2802] T 385eb017c5734053b4d4b703b69de506 P 64832ff33e914a3789ba59159c816e2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:59:15.467679 9071 raft_consensus.cc:491] T 385eb017c5734053b4d4b703b69de506 P 64832ff33e914a3789ba59159c816e2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:59:15.468062 9071 raft_consensus.cc:3058] T 385eb017c5734053b4d4b703b69de506 P 64832ff33e914a3789ba59159c816e2a [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:59:15.475155 9071 raft_consensus.cc:513] T 385eb017c5734053b4d4b703b69de506 P 64832ff33e914a3789ba59159c816e2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07285654123141ec9b8b9bcc1ae30716" member_type: VOTER last_known_addr { host: "127.3.173.69" port: 32899 } } peers { permanent_uuid: "446b26061b9148dba78b4411bc7c6c0f" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 33387 } } peers { permanent_uuid: "64832ff33e914a3789ba59159c816e2a" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 38495 } }
I20250627 13:59:15.476682 9071 leader_election.cc:290] T 385eb017c5734053b4d4b703b69de506 P 64832ff33e914a3789ba59159c816e2a [CANDIDATE]: Term 1 election: Requested vote from peers 07285654123141ec9b8b9bcc1ae30716 (127.3.173.69:32899), 446b26061b9148dba78b4411bc7c6c0f (127.3.173.65:33387)
I20250627 13:59:15.477614 8999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "385eb017c5734053b4d4b703b69de506" candidate_uuid: "64832ff33e914a3789ba59159c816e2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07285654123141ec9b8b9bcc1ae30716"
I20250627 13:59:15.478191 8999 raft_consensus.cc:3058] T 385eb017c5734053b4d4b703b69de506 P 07285654123141ec9b8b9bcc1ae30716 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:59:15.478185 8466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "385eb017c5734053b4d4b703b69de506" candidate_uuid: "64832ff33e914a3789ba59159c816e2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446b26061b9148dba78b4411bc7c6c0f"
I20250627 13:59:15.478691 8466 raft_consensus.cc:3058] T 385eb017c5734053b4d4b703b69de506 P 446b26061b9148dba78b4411bc7c6c0f [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:59:15.483554 8999 raft_consensus.cc:2466] T 385eb017c5734053b4d4b703b69de506 P 07285654123141ec9b8b9bcc1ae30716 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64832ff33e914a3789ba59159c816e2a in term 1.
I20250627 13:59:15.483618 8466 raft_consensus.cc:2466] T 385eb017c5734053b4d4b703b69de506 P 446b26061b9148dba78b4411bc7c6c0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64832ff33e914a3789ba59159c816e2a in term 1.
I20250627 13:59:15.484434 8668 leader_election.cc:304] T 385eb017c5734053b4d4b703b69de506 P 64832ff33e914a3789ba59159c816e2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07285654123141ec9b8b9bcc1ae30716, 64832ff33e914a3789ba59159c816e2a; no voters:
I20250627 13:59:15.484995 9071 raft_consensus.cc:2802] T 385eb017c5734053b4d4b703b69de506 P 64832ff33e914a3789ba59159c816e2a [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:59:15.486912 9071 raft_consensus.cc:695] T 385eb017c5734053b4d4b703b69de506 P 64832ff33e914a3789ba59159c816e2a [term 1 LEADER]: Becoming Leader. State: Replica: 64832ff33e914a3789ba59159c816e2a, State: Running, Role: LEADER
I20250627 13:59:15.487747 9071 consensus_queue.cc:237] T 385eb017c5734053b4d4b703b69de506 P 64832ff33e914a3789ba59159c816e2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07285654123141ec9b8b9bcc1ae30716" member_type: VOTER last_known_addr { host: "127.3.173.69" port: 32899 } } peers { permanent_uuid: "446b26061b9148dba78b4411bc7c6c0f" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 33387 } } peers { permanent_uuid: "64832ff33e914a3789ba59159c816e2a" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 38495 } }
I20250627 13:59:15.498759 8323 catalog_manager.cc:5582] T 385eb017c5734053b4d4b703b69de506 P 64832ff33e914a3789ba59159c816e2a reported cstate change: term changed from 0 to 1, leader changed from <none> to 64832ff33e914a3789ba59159c816e2a (127.3.173.67). New cstate: current_term: 1 leader_uuid: "64832ff33e914a3789ba59159c816e2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07285654123141ec9b8b9bcc1ae30716" member_type: VOTER last_known_addr { host: "127.3.173.69" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "446b26061b9148dba78b4411bc7c6c0f" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 33387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64832ff33e914a3789ba59159c816e2a" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 38495 } health_report { overall_health: HEALTHY } } }
I20250627 13:59:15.536096 3765 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250627 13:59:15.539445 3765 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 446b26061b9148dba78b4411bc7c6c0f to finish bootstrapping
I20250627 13:59:15.551494 3765 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 64832ff33e914a3789ba59159c816e2a to finish bootstrapping
I20250627 13:59:15.562467 3765 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 07285654123141ec9b8b9bcc1ae30716 to finish bootstrapping
I20250627 13:59:15.573873 3765 test_util.cc:276] Using random seed: 358798428
I20250627 13:59:15.598709 3765 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20250627 13:59:15.599572 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 8648
W20250627 13:59:15.633677 9083 negotiation.cc:337] Failed RPC negotiation. Trace:
0627 13:59:15.610167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.173.67:38495 (local address 127.0.0.1:44504)
0627 13:59:15.611067 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0627 13:59:15.611124 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0627 13:59:15.611407 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0627 13:59:15.624712 (+ 13305us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.173.67:38495: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":78}
W20250627 13:59:15.635344 9078 meta_cache.cc:302] tablet 385eb017c5734053b4d4b703b69de506: replica 64832ff33e914a3789ba59159c816e2a (127.3.173.67:38495) has failed: Network error: Client connection negotiation failed: client connection to 127.3.173.67:38495: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250627 13:59:15.652045 8959 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47218: Illegal state: replica 07285654123141ec9b8b9bcc1ae30716 is not leader of this config: current role FOLLOWER
W20250627 13:59:15.669194 8426 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33950: Illegal state: replica 446b26061b9148dba78b4411bc7c6c0f is not leader of this config: current role FOLLOWER
W20250627 13:59:15.688927 8959 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47218: Illegal state: replica 07285654123141ec9b8b9bcc1ae30716 is not leader of this config: current role FOLLOWER
W20250627 13:59:15.701555 8426 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33950: Illegal state: replica 446b26061b9148dba78b4411bc7c6c0f is not leader of this config: current role FOLLOWER
W20250627 13:59:15.729712 8959 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47218: Illegal state: replica 07285654123141ec9b8b9bcc1ae30716 is not leader of this config: current role FOLLOWER
W20250627 13:59:15.743505 8426 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33950: Illegal state: replica 446b26061b9148dba78b4411bc7c6c0f is not leader of this config: current role FOLLOWER
W20250627 13:59:15.777376 8959 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47218: Illegal state: replica 07285654123141ec9b8b9bcc1ae30716 is not leader of this config: current role FOLLOWER
W20250627 13:59:15.794309 8426 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33950: Illegal state: replica 446b26061b9148dba78b4411bc7c6c0f is not leader of this config: current role FOLLOWER
W20250627 13:59:15.833220 8959 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47218: Illegal state: replica 07285654123141ec9b8b9bcc1ae30716 is not leader of this config: current role FOLLOWER
W20250627 13:59:15.854769 8426 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33950: Illegal state: replica 446b26061b9148dba78b4411bc7c6c0f is not leader of this config: current role FOLLOWER
W20250627 13:59:15.901582 8959 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47218: Illegal state: replica 07285654123141ec9b8b9bcc1ae30716 is not leader of this config: current role FOLLOWER
W20250627 13:59:15.922534 8426 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33950: Illegal state: replica 446b26061b9148dba78b4411bc7c6c0f is not leader of this config: current role FOLLOWER
W20250627 13:59:15.974596 8959 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47218: Illegal state: replica 07285654123141ec9b8b9bcc1ae30716 is not leader of this config: current role FOLLOWER
W20250627 13:59:16.002375 8426 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33950: Illegal state: replica 446b26061b9148dba78b4411bc7c6c0f is not leader of this config: current role FOLLOWER
W20250627 13:59:16.060322 8959 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47218: Illegal state: replica 07285654123141ec9b8b9bcc1ae30716 is not leader of this config: current role FOLLOWER
W20250627 13:59:16.091392 8426 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33950: Illegal state: replica 446b26061b9148dba78b4411bc7c6c0f is not leader of this config: current role FOLLOWER
W20250627 13:59:16.161703 8959 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47218: Illegal state: replica 07285654123141ec9b8b9bcc1ae30716 is not leader of this config: current role FOLLOWER
W20250627 13:59:16.194582 8426 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33950: Illegal state: replica 446b26061b9148dba78b4411bc7c6c0f is not leader of this config: current role FOLLOWER
W20250627 13:59:16.266630 8959 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47218: Illegal state: replica 07285654123141ec9b8b9bcc1ae30716 is not leader of this config: current role FOLLOWER
W20250627 13:59:16.305629 8426 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33950: Illegal state: replica 446b26061b9148dba78b4411bc7c6c0f is not leader of this config: current role FOLLOWER
W20250627 13:59:16.389901 8959 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47218: Illegal state: replica 07285654123141ec9b8b9bcc1ae30716 is not leader of this config: current role FOLLOWER
W20250627 13:59:16.433027 8426 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33950: Illegal state: replica 446b26061b9148dba78b4411bc7c6c0f is not leader of this config: current role FOLLOWER
W20250627 13:59:16.520570 8959 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47218: Illegal state: replica 07285654123141ec9b8b9bcc1ae30716 is not leader of this config: current role FOLLOWER
W20250627 13:59:16.564421 8426 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33950: Illegal state: replica 446b26061b9148dba78b4411bc7c6c0f is not leader of this config: current role FOLLOWER
W20250627 13:59:16.659469 8959 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47218: Illegal state: replica 07285654123141ec9b8b9bcc1ae30716 is not leader of this config: current role FOLLOWER
W20250627 13:59:16.708101 8426 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33950: Illegal state: replica 446b26061b9148dba78b4411bc7c6c0f is not leader of this config: current role FOLLOWER
W20250627 13:59:16.760061 9078 meta_cache.cc:302] tablet 385eb017c5734053b4d4b703b69de506: replica 64832ff33e914a3789ba59159c816e2a (127.3.173.67:38495) has failed: Network error: Client connection negotiation failed: client connection to 127.3.173.67:38495: connect: Connection refused (error 111)
W20250627 13:59:16.805020 8959 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47218: Illegal state: replica 07285654123141ec9b8b9bcc1ae30716 is not leader of this config: current role FOLLOWER
W20250627 13:59:16.855324 8426 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33950: Illegal state: replica 446b26061b9148dba78b4411bc7c6c0f is not leader of this config: current role FOLLOWER
W20250627 13:59:16.962996 8959 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47218: Illegal state: replica 07285654123141ec9b8b9bcc1ae30716 is not leader of this config: current role FOLLOWER
W20250627 13:59:17.017524 8426 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33950: Illegal state: replica 446b26061b9148dba78b4411bc7c6c0f is not leader of this config: current role FOLLOWER
W20250627 13:59:17.128917 8959 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47218: Illegal state: replica 07285654123141ec9b8b9bcc1ae30716 is not leader of this config: current role FOLLOWER
W20250627 13:59:17.184653 8426 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33950: Illegal state: replica 446b26061b9148dba78b4411bc7c6c0f is not leader of this config: current role FOLLOWER
W20250627 13:59:17.299805 8959 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47218: Illegal state: replica 07285654123141ec9b8b9bcc1ae30716 is not leader of this config: current role FOLLOWER
W20250627 13:59:17.355288 8426 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33950: Illegal state: replica 446b26061b9148dba78b4411bc7c6c0f is not leader of this config: current role FOLLOWER
I20250627 13:59:17.431957 9101 raft_consensus.cc:491] T 385eb017c5734053b4d4b703b69de506 P 446b26061b9148dba78b4411bc7c6c0f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:59:17.432425 9101 raft_consensus.cc:513] T 385eb017c5734053b4d4b703b69de506 P 446b26061b9148dba78b4411bc7c6c0f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07285654123141ec9b8b9bcc1ae30716" member_type: VOTER last_known_addr { host: "127.3.173.69" port: 32899 } } peers { permanent_uuid: "446b26061b9148dba78b4411bc7c6c0f" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 33387 } } peers { permanent_uuid: "64832ff33e914a3789ba59159c816e2a" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 38495 } }
I20250627 13:59:17.435062 9101 leader_election.cc:290] T 385eb017c5734053b4d4b703b69de506 P 446b26061b9148dba78b4411bc7c6c0f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 07285654123141ec9b8b9bcc1ae30716 (127.3.173.69:32899), 64832ff33e914a3789ba59159c816e2a (127.3.173.67:38495)
W20250627 13:59:17.452684 8399 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.173.67:38495: connect: Connection refused (error 111)
W20250627 13:59:17.457297 8399 leader_election.cc:336] T 385eb017c5734053b4d4b703b69de506 P 446b26061b9148dba78b4411bc7c6c0f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 64832ff33e914a3789ba59159c816e2a (127.3.173.67:38495): Network error: Client connection negotiation failed: client connection to 127.3.173.67:38495: connect: Connection refused (error 111)
I20250627 13:59:17.459853 8999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "385eb017c5734053b4d4b703b69de506" candidate_uuid: "446b26061b9148dba78b4411bc7c6c0f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07285654123141ec9b8b9bcc1ae30716" is_pre_election: true
I20250627 13:59:17.460448 8999 raft_consensus.cc:2466] T 385eb017c5734053b4d4b703b69de506 P 07285654123141ec9b8b9bcc1ae30716 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 446b26061b9148dba78b4411bc7c6c0f in term 1.
I20250627 13:59:17.461373 8402 leader_election.cc:304] T 385eb017c5734053b4d4b703b69de506 P 446b26061b9148dba78b4411bc7c6c0f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07285654123141ec9b8b9bcc1ae30716, 446b26061b9148dba78b4411bc7c6c0f; no voters: 64832ff33e914a3789ba59159c816e2a
I20250627 13:59:17.462030 9101 raft_consensus.cc:2802] T 385eb017c5734053b4d4b703b69de506 P 446b26061b9148dba78b4411bc7c6c0f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250627 13:59:17.462386 9101 raft_consensus.cc:491] T 385eb017c5734053b4d4b703b69de506 P 446b26061b9148dba78b4411bc7c6c0f [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:59:17.462715 9101 raft_consensus.cc:3058] T 385eb017c5734053b4d4b703b69de506 P 446b26061b9148dba78b4411bc7c6c0f [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:59:17.468358 9101 raft_consensus.cc:513] T 385eb017c5734053b4d4b703b69de506 P 446b26061b9148dba78b4411bc7c6c0f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07285654123141ec9b8b9bcc1ae30716" member_type: VOTER last_known_addr { host: "127.3.173.69" port: 32899 } } peers { permanent_uuid: "446b26061b9148dba78b4411bc7c6c0f" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 33387 } } peers { permanent_uuid: "64832ff33e914a3789ba59159c816e2a" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 38495 } }
I20250627 13:59:17.470353 9101 leader_election.cc:290] T 385eb017c5734053b4d4b703b69de506 P 446b26061b9148dba78b4411bc7c6c0f [CANDIDATE]: Term 2 election: Requested vote from peers 07285654123141ec9b8b9bcc1ae30716 (127.3.173.69:32899), 64832ff33e914a3789ba59159c816e2a (127.3.173.67:38495)
I20250627 13:59:17.471459 8999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "385eb017c5734053b4d4b703b69de506" candidate_uuid: "446b26061b9148dba78b4411bc7c6c0f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07285654123141ec9b8b9bcc1ae30716"
I20250627 13:59:17.472033 8999 raft_consensus.cc:3058] T 385eb017c5734053b4d4b703b69de506 P 07285654123141ec9b8b9bcc1ae30716 [term 1 FOLLOWER]: Advancing to term 2
W20250627 13:59:17.475154 8959 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47218: Illegal state: replica 07285654123141ec9b8b9bcc1ae30716 is not leader of this config: current role FOLLOWER
W20250627 13:59:17.477425 8399 leader_election.cc:336] T 385eb017c5734053b4d4b703b69de506 P 446b26061b9148dba78b4411bc7c6c0f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 64832ff33e914a3789ba59159c816e2a (127.3.173.67:38495): Network error: Client connection negotiation failed: client connection to 127.3.173.67:38495: connect: Connection refused (error 111)
I20250627 13:59:17.478382 8999 raft_consensus.cc:2466] T 385eb017c5734053b4d4b703b69de506 P 07285654123141ec9b8b9bcc1ae30716 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 446b26061b9148dba78b4411bc7c6c0f in term 2.
I20250627 13:59:17.479136 8402 leader_election.cc:304] T 385eb017c5734053b4d4b703b69de506 P 446b26061b9148dba78b4411bc7c6c0f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07285654123141ec9b8b9bcc1ae30716, 446b26061b9148dba78b4411bc7c6c0f; no voters: 64832ff33e914a3789ba59159c816e2a
I20250627 13:59:17.479734 9101 raft_consensus.cc:2802] T 385eb017c5734053b4d4b703b69de506 P 446b26061b9148dba78b4411bc7c6c0f [term 2 FOLLOWER]: Leader election won for term 2
I20250627 13:59:17.481416 9101 raft_consensus.cc:695] T 385eb017c5734053b4d4b703b69de506 P 446b26061b9148dba78b4411bc7c6c0f [term 2 LEADER]: Becoming Leader. State: Replica: 446b26061b9148dba78b4411bc7c6c0f, State: Running, Role: LEADER
I20250627 13:59:17.482504 9101 consensus_queue.cc:237] T 385eb017c5734053b4d4b703b69de506 P 446b26061b9148dba78b4411bc7c6c0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07285654123141ec9b8b9bcc1ae30716" member_type: VOTER last_known_addr { host: "127.3.173.69" port: 32899 } } peers { permanent_uuid: "446b26061b9148dba78b4411bc7c6c0f" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 33387 } } peers { permanent_uuid: "64832ff33e914a3789ba59159c816e2a" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 38495 } }
I20250627 13:59:17.496001 8323 catalog_manager.cc:5582] T 385eb017c5734053b4d4b703b69de506 P 446b26061b9148dba78b4411bc7c6c0f reported cstate change: term changed from 1 to 2, leader changed from 64832ff33e914a3789ba59159c816e2a (127.3.173.67) to 446b26061b9148dba78b4411bc7c6c0f (127.3.173.65). New cstate: current_term: 2 leader_uuid: "446b26061b9148dba78b4411bc7c6c0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07285654123141ec9b8b9bcc1ae30716" member_type: VOTER last_known_addr { host: "127.3.173.69" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "446b26061b9148dba78b4411bc7c6c0f" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 33387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64832ff33e914a3789ba59159c816e2a" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 38495 } health_report { overall_health: UNKNOWN } } }
I20250627 13:59:17.541342 8999 raft_consensus.cc:1273] T 385eb017c5734053b4d4b703b69de506 P 07285654123141ec9b8b9bcc1ae30716 [term 2 FOLLOWER]: Refusing update from remote peer 446b26061b9148dba78b4411bc7c6c0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250627 13:59:17.542975 9106 consensus_queue.cc:1035] T 385eb017c5734053b4d4b703b69de506 P 446b26061b9148dba78b4411bc7c6c0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "07285654123141ec9b8b9bcc1ae30716" member_type: VOTER last_known_addr { host: "127.3.173.69" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250627 13:59:17.543767 8399 consensus_peers.cc:487] T 385eb017c5734053b4d4b703b69de506 P 446b26061b9148dba78b4411bc7c6c0f -> Peer 64832ff33e914a3789ba59159c816e2a (127.3.173.67:38495): Couldn't send request to peer 64832ff33e914a3789ba59159c816e2a. Status: Network error: Client connection negotiation failed: client connection to 127.3.173.67:38495: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250627 13:59:17.562379 9121 mvcc.cc:204] Tried to move back new op lower bound from 7172230174872727552 to 7172230174655647744. Current Snapshot: MvccSnapshot[applied={T|T < 7172230174872727552}]
I20250627 13:59:17.624044 8979 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:59:17.645023 8845 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:59:17.677188 8446 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:59:17.686947 8579 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:59:19.428364 8979 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:59:19.446206 8845 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:59:19.485080 8579 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:59:19.490617 8446 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
W20250627 13:59:20.070509 8399 consensus_peers.cc:487] T 385eb017c5734053b4d4b703b69de506 P 446b26061b9148dba78b4411bc7c6c0f -> Peer 64832ff33e914a3789ba59159c816e2a (127.3.173.67:38495): Couldn't send request to peer 64832ff33e914a3789ba59159c816e2a. Status: Network error: Client connection negotiation failed: client connection to 127.3.173.67:38495: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250627 13:59:21.187646 8845 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:59:21.193534 8979 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:59:21.220232 8579 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:59:21.237025 8446 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
W20250627 13:59:22.499780 8399 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.173.67:38495: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20250627 13:59:22.507174 8399 consensus_peers.cc:487] T 385eb017c5734053b4d4b703b69de506 P 446b26061b9148dba78b4411bc7c6c0f -> Peer 64832ff33e914a3789ba59159c816e2a (127.3.173.67:38495): Couldn't send request to peer 64832ff33e914a3789ba59159c816e2a. Status: Network error: Client connection negotiation failed: client connection to 127.3.173.67:38495: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250627 13:59:23.633713 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 8382
I20250627 13:59:23.673274 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 8515
I20250627 13:59:23.697834 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 8781
I20250627 13:59:23.721164 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 8914
I20250627 13:59:23.754734 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 8289
2025-06-27T13:59:23Z chronyd exiting
[ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4 (20261 ms)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest (20261 ms total)
[----------] 1 test from ListTableCliSimpleParamTest
[ RUN ] ListTableCliSimpleParamTest.TestListTables/2
I20250627 13:59:23.811300 3765 test_util.cc:276] Using random seed: 367035852
I20250627 13:59:23.815282 3765 ts_itest-base.cc:115] Starting cluster with:
I20250627 13:59:23.815464 3765 ts_itest-base.cc:116] --------------
I20250627 13:59:23.815625 3765 ts_itest-base.cc:117] 1 tablet servers
I20250627 13:59:23.815763 3765 ts_itest-base.cc:118] 1 replicas per TS
I20250627 13:59:23.815907 3765 ts_itest-base.cc:119] --------------
2025-06-27T13:59:23Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-27T13:59:23Z Disabled control of system clock
I20250627 13:59:23.859702 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.173.126:44893
--webserver_interface=127.3.173.126
--webserver_port=0
--builtin_ntp_servers=127.3.173.84:43647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.173.126:44893 with env {}
W20250627 13:59:24.160679 9228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:59:24.161221 9228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:59:24.161624 9228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:59:24.192200 9228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250627 13:59:24.192489 9228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:59:24.192685 9228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250627 13:59:24.192870 9228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250627 13:59:24.227253 9228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:43647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.173.126:44893
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.173.126:44893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.3.173.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:59:24.228467 9228 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:59:24.230015 9228 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:59:24.244212 9234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:59:24.244303 9237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:59:24.244427 9235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:59:25.383934 9236 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250627 13:59:25.384066 9228 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250627 13:59:25.387763 9228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:59:25.390183 9228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:59:25.391501 9228 hybrid_clock.cc:648] HybridClock initialized: now 1751032765391461 us; error 59 us; skew 500 ppm
I20250627 13:59:25.392280 9228 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:59:25.398456 9228 webserver.cc:469] Webserver started at http://127.3.173.126:45297/ using document root <none> and password file <none>
I20250627 13:59:25.399338 9228 fs_manager.cc:362] Metadata directory not provided
I20250627 13:59:25.399531 9228 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:59:25.399997 9228 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:59:25.404237 9228 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "958cfbdc65cc44bcbb29807e1ff9ea25"
format_stamp: "Formatted at 2025-06-27 13:59:25 on dist-test-slave-rn80"
I20250627 13:59:25.405278 9228 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "958cfbdc65cc44bcbb29807e1ff9ea25"
format_stamp: "Formatted at 2025-06-27 13:59:25 on dist-test-slave-rn80"
I20250627 13:59:25.411916 9228 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20250627 13:59:25.417112 9245 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:59:25.418081 9228 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250627 13:59:25.418407 9228 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
uuid: "958cfbdc65cc44bcbb29807e1ff9ea25"
format_stamp: "Formatted at 2025-06-27 13:59:25 on dist-test-slave-rn80"
I20250627 13:59:25.418682 9228 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:59:25.485430 9228 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:59:25.486927 9228 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:59:25.487389 9228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:59:25.554145 9228 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.126:44893
I20250627 13:59:25.554260 9296 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.126:44893 every 8 connection(s)
I20250627 13:59:25.556643 9228 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250627 13:59:25.559759 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 9228
I20250627 13:59:25.560431 3765 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751032624583798-3765-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250627 13:59:25.562330 9297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:59:25.584969 9297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 958cfbdc65cc44bcbb29807e1ff9ea25: Bootstrap starting.
I20250627 13:59:25.590174 9297 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 958cfbdc65cc44bcbb29807e1ff9ea25: Neither blocks nor log segments found. Creating new log.
I20250627 13:59:25.591689 9297 log.cc:826] T 00000000000000000000000000000000 P 958cfbdc65cc44bcbb29807e1ff9ea25: Log is configured to *not* fsync() on all Append() calls
I20250627 13:59:25.595629 9297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 958cfbdc65cc44bcbb29807e1ff9ea25: No bootstrap required, opened a new log
I20250627 13:59:25.611492 9297 raft_consensus.cc:357] T 00000000000000000000000000000000 P 958cfbdc65cc44bcbb29807e1ff9ea25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "958cfbdc65cc44bcbb29807e1ff9ea25" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 44893 } }
I20250627 13:59:25.612048 9297 raft_consensus.cc:383] T 00000000000000000000000000000000 P 958cfbdc65cc44bcbb29807e1ff9ea25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:59:25.612289 9297 raft_consensus.cc:738] T 00000000000000000000000000000000 P 958cfbdc65cc44bcbb29807e1ff9ea25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 958cfbdc65cc44bcbb29807e1ff9ea25, State: Initialized, Role: FOLLOWER
I20250627 13:59:25.612879 9297 consensus_queue.cc:260] T 00000000000000000000000000000000 P 958cfbdc65cc44bcbb29807e1ff9ea25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "958cfbdc65cc44bcbb29807e1ff9ea25" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 44893 } }
I20250627 13:59:25.613380 9297 raft_consensus.cc:397] T 00000000000000000000000000000000 P 958cfbdc65cc44bcbb29807e1ff9ea25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250627 13:59:25.613600 9297 raft_consensus.cc:491] T 00000000000000000000000000000000 P 958cfbdc65cc44bcbb29807e1ff9ea25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250627 13:59:25.613855 9297 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 958cfbdc65cc44bcbb29807e1ff9ea25 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:59:25.617949 9297 raft_consensus.cc:513] T 00000000000000000000000000000000 P 958cfbdc65cc44bcbb29807e1ff9ea25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "958cfbdc65cc44bcbb29807e1ff9ea25" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 44893 } }
I20250627 13:59:25.618611 9297 leader_election.cc:304] T 00000000000000000000000000000000 P 958cfbdc65cc44bcbb29807e1ff9ea25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 958cfbdc65cc44bcbb29807e1ff9ea25; no voters:
I20250627 13:59:25.620134 9297 leader_election.cc:290] T 00000000000000000000000000000000 P 958cfbdc65cc44bcbb29807e1ff9ea25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250627 13:59:25.620754 9302 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 958cfbdc65cc44bcbb29807e1ff9ea25 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:59:25.622728 9302 raft_consensus.cc:695] T 00000000000000000000000000000000 P 958cfbdc65cc44bcbb29807e1ff9ea25 [term 1 LEADER]: Becoming Leader. State: Replica: 958cfbdc65cc44bcbb29807e1ff9ea25, State: Running, Role: LEADER
I20250627 13:59:25.623293 9302 consensus_queue.cc:237] T 00000000000000000000000000000000 P 958cfbdc65cc44bcbb29807e1ff9ea25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "958cfbdc65cc44bcbb29807e1ff9ea25" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 44893 } }
I20250627 13:59:25.623729 9297 sys_catalog.cc:564] T 00000000000000000000000000000000 P 958cfbdc65cc44bcbb29807e1ff9ea25 [sys.catalog]: configured and running, proceeding with master startup.
I20250627 13:59:25.633872 9303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 958cfbdc65cc44bcbb29807e1ff9ea25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "958cfbdc65cc44bcbb29807e1ff9ea25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "958cfbdc65cc44bcbb29807e1ff9ea25" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 44893 } } }
I20250627 13:59:25.633634 9304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 958cfbdc65cc44bcbb29807e1ff9ea25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 958cfbdc65cc44bcbb29807e1ff9ea25. Latest consensus state: current_term: 1 leader_uuid: "958cfbdc65cc44bcbb29807e1ff9ea25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "958cfbdc65cc44bcbb29807e1ff9ea25" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 44893 } } }
I20250627 13:59:25.634495 9303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 958cfbdc65cc44bcbb29807e1ff9ea25 [sys.catalog]: This master's current role is: LEADER
I20250627 13:59:25.634737 9304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 958cfbdc65cc44bcbb29807e1ff9ea25 [sys.catalog]: This master's current role is: LEADER
I20250627 13:59:25.637320 9310 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250627 13:59:25.648579 9310 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250627 13:59:25.664023 9310 catalog_manager.cc:1349] Generated new cluster ID: b7edd1582b2a474a8481689ebd803b6f
I20250627 13:59:25.664255 9310 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250627 13:59:25.687738 9310 catalog_manager.cc:1372] Generated new certificate authority record
I20250627 13:59:25.689191 9310 catalog_manager.cc:1506] Loading token signing keys...
I20250627 13:59:25.705518 9310 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 958cfbdc65cc44bcbb29807e1ff9ea25: Generated new TSK 0
I20250627 13:59:25.706609 9310 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250627 13:59:25.727986 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.173.65:0
--local_ip_for_outbound_sockets=127.3.173.65
--webserver_interface=127.3.173.65
--webserver_port=0
--tserver_master_addrs=127.3.173.126:44893
--builtin_ntp_servers=127.3.173.84:43647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250627 13:59:26.020277 9321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:59:26.020761 9321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:59:26.021247 9321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:59:26.052093 9321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:59:26.052889 9321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.173.65
I20250627 13:59:26.088208 9321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:43647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.173.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.3.173.65
--webserver_port=0
--tserver_master_addrs=127.3.173.126:44893
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.173.65
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:59:26.089459 9321 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:59:26.091054 9321 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:59:26.108232 9328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:59:27.575122 9321 thread.cc:640] OpenStack (cloud detector) Time spent creating pthread: real 1.471s user 0.508s sys 0.962s
W20250627 13:59:27.506100 9326 debug-util.cc:398] Leaking SignalData structure 0x7b08000184e0 after lost signal to thread 9321
W20250627 13:59:27.575551 9321 thread.cc:606] OpenStack (cloud detector) Time spent starting thread: real 1.472s user 0.508s sys 0.963s
I20250627 13:59:27.575812 9321 server_base.cc:1048] running on GCE node
W20250627 13:59:26.108464 9327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:59:27.580736 9330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:59:27.581905 9321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:59:27.583887 9321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:59:27.585206 9321 hybrid_clock.cc:648] HybridClock initialized: now 1751032767585164 us; error 50 us; skew 500 ppm
I20250627 13:59:27.585886 9321 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:59:27.597303 9321 webserver.cc:469] Webserver started at http://127.3.173.65:43375/ using document root <none> and password file <none>
I20250627 13:59:27.598276 9321 fs_manager.cc:362] Metadata directory not provided
I20250627 13:59:27.598485 9321 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:59:27.598910 9321 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:59:27.603078 9321 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "c68b2409366042dc817c0dfe0c3f8b16"
format_stamp: "Formatted at 2025-06-27 13:59:27 on dist-test-slave-rn80"
I20250627 13:59:27.604027 9321 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "c68b2409366042dc817c0dfe0c3f8b16"
format_stamp: "Formatted at 2025-06-27 13:59:27 on dist-test-slave-rn80"
I20250627 13:59:27.610931 9321 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20250627 13:59:27.616909 9337 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:59:27.617928 9321 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20250627 13:59:27.618252 9321 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "c68b2409366042dc817c0dfe0c3f8b16"
format_stamp: "Formatted at 2025-06-27 13:59:27 on dist-test-slave-rn80"
I20250627 13:59:27.618556 9321 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:59:27.671175 9321 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:59:27.672601 9321 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:59:27.673045 9321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:59:27.675971 9321 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:59:27.680115 9321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:59:27.680311 9321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:59:27.680552 9321 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:59:27.680706 9321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:59:27.839605 9321 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.65:44733
I20250627 13:59:27.839748 9449 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.65:44733 every 8 connection(s)
I20250627 13:59:27.842089 9321 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250627 13:59:27.843674 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 9321
I20250627 13:59:27.844108 3765 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751032624583798-3765-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250627 13:59:27.868582 9450 heartbeater.cc:344] Connected to a master server at 127.3.173.126:44893
I20250627 13:59:27.869061 9450 heartbeater.cc:461] Registering TS with master...
I20250627 13:59:27.870234 9450 heartbeater.cc:507] Master 127.3.173.126:44893 requested a full tablet report, sending...
I20250627 13:59:27.872619 9262 ts_manager.cc:194] Registered new tserver with Master: c68b2409366042dc817c0dfe0c3f8b16 (127.3.173.65:44733)
I20250627 13:59:27.874406 9262 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.173.65:38515
I20250627 13:59:27.878777 3765 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250627 13:59:27.915923 9262 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53576:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250627 13:59:27.980574 9385 tablet_service.cc:1468] Processing CreateTablet for tablet 542f8754fa164c4bbadf14c745ab3108 (DEFAULT_TABLE table=TestTable [id=f8ae6c76aa834e3fb40ddc1fcbab7f9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250627 13:59:27.982016 9385 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 542f8754fa164c4bbadf14c745ab3108. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:59:28.002482 9465 tablet_bootstrap.cc:492] T 542f8754fa164c4bbadf14c745ab3108 P c68b2409366042dc817c0dfe0c3f8b16: Bootstrap starting.
I20250627 13:59:28.009595 9465 tablet_bootstrap.cc:654] T 542f8754fa164c4bbadf14c745ab3108 P c68b2409366042dc817c0dfe0c3f8b16: Neither blocks nor log segments found. Creating new log.
I20250627 13:59:28.011916 9465 log.cc:826] T 542f8754fa164c4bbadf14c745ab3108 P c68b2409366042dc817c0dfe0c3f8b16: Log is configured to *not* fsync() on all Append() calls
I20250627 13:59:28.016966 9465 tablet_bootstrap.cc:492] T 542f8754fa164c4bbadf14c745ab3108 P c68b2409366042dc817c0dfe0c3f8b16: No bootstrap required, opened a new log
I20250627 13:59:28.017432 9465 ts_tablet_manager.cc:1397] T 542f8754fa164c4bbadf14c745ab3108 P c68b2409366042dc817c0dfe0c3f8b16: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20250627 13:59:28.042793 9465 raft_consensus.cc:357] T 542f8754fa164c4bbadf14c745ab3108 P c68b2409366042dc817c0dfe0c3f8b16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c68b2409366042dc817c0dfe0c3f8b16" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 44733 } }
I20250627 13:59:28.043542 9465 raft_consensus.cc:383] T 542f8754fa164c4bbadf14c745ab3108 P c68b2409366042dc817c0dfe0c3f8b16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:59:28.043821 9465 raft_consensus.cc:738] T 542f8754fa164c4bbadf14c745ab3108 P c68b2409366042dc817c0dfe0c3f8b16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c68b2409366042dc817c0dfe0c3f8b16, State: Initialized, Role: FOLLOWER
I20250627 13:59:28.044633 9465 consensus_queue.cc:260] T 542f8754fa164c4bbadf14c745ab3108 P c68b2409366042dc817c0dfe0c3f8b16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c68b2409366042dc817c0dfe0c3f8b16" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 44733 } }
I20250627 13:59:28.045295 9465 raft_consensus.cc:397] T 542f8754fa164c4bbadf14c745ab3108 P c68b2409366042dc817c0dfe0c3f8b16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250627 13:59:28.045614 9465 raft_consensus.cc:491] T 542f8754fa164c4bbadf14c745ab3108 P c68b2409366042dc817c0dfe0c3f8b16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250627 13:59:28.046017 9465 raft_consensus.cc:3058] T 542f8754fa164c4bbadf14c745ab3108 P c68b2409366042dc817c0dfe0c3f8b16 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:59:28.052392 9465 raft_consensus.cc:513] T 542f8754fa164c4bbadf14c745ab3108 P c68b2409366042dc817c0dfe0c3f8b16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c68b2409366042dc817c0dfe0c3f8b16" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 44733 } }
I20250627 13:59:28.053277 9465 leader_election.cc:304] T 542f8754fa164c4bbadf14c745ab3108 P c68b2409366042dc817c0dfe0c3f8b16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c68b2409366042dc817c0dfe0c3f8b16; no voters:
I20250627 13:59:28.053618 9467 raft_consensus.cc:491] T 542f8754fa164c4bbadf14c745ab3108 P c68b2409366042dc817c0dfe0c3f8b16 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:59:28.054010 9467 raft_consensus.cc:513] T 542f8754fa164c4bbadf14c745ab3108 P c68b2409366042dc817c0dfe0c3f8b16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c68b2409366042dc817c0dfe0c3f8b16" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 44733 } }
I20250627 13:59:28.056213 9465 leader_election.cc:290] T 542f8754fa164c4bbadf14c745ab3108 P c68b2409366042dc817c0dfe0c3f8b16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250627 13:59:28.056721 9468 raft_consensus.cc:2802] T 542f8754fa164c4bbadf14c745ab3108 P c68b2409366042dc817c0dfe0c3f8b16 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:59:28.059940 9467 leader_election.cc:304] T 542f8754fa164c4bbadf14c745ab3108 P c68b2409366042dc817c0dfe0c3f8b16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c68b2409366042dc817c0dfe0c3f8b16; no voters:
I20250627 13:59:28.064400 9450 heartbeater.cc:499] Master 127.3.173.126:44893 was elected leader, sending a full tablet report...
I20250627 13:59:28.066000 9465 ts_tablet_manager.cc:1428] T 542f8754fa164c4bbadf14c745ab3108 P c68b2409366042dc817c0dfe0c3f8b16: Time spent starting tablet: real 0.048s user 0.030s sys 0.020s
I20250627 13:59:28.066699 9467 leader_election.cc:290] T 542f8754fa164c4bbadf14c745ab3108 P c68b2409366042dc817c0dfe0c3f8b16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers
I20250627 13:59:28.067220 9468 raft_consensus.cc:695] T 542f8754fa164c4bbadf14c745ab3108 P c68b2409366042dc817c0dfe0c3f8b16 [term 1 LEADER]: Becoming Leader. State: Replica: c68b2409366042dc817c0dfe0c3f8b16, State: Running, Role: LEADER
I20250627 13:59:28.067898 9468 consensus_queue.cc:237] T 542f8754fa164c4bbadf14c745ab3108 P c68b2409366042dc817c0dfe0c3f8b16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c68b2409366042dc817c0dfe0c3f8b16" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 44733 } }
I20250627 13:59:28.079211 9262 catalog_manager.cc:5582] T 542f8754fa164c4bbadf14c745ab3108 P c68b2409366042dc817c0dfe0c3f8b16 reported cstate change: term changed from 0 to 1, leader changed from <none> to c68b2409366042dc817c0dfe0c3f8b16 (127.3.173.65). New cstate: current_term: 1 leader_uuid: "c68b2409366042dc817c0dfe0c3f8b16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c68b2409366042dc817c0dfe0c3f8b16" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 44733 } health_report { overall_health: HEALTHY } } }
I20250627 13:59:28.107080 3765 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250627 13:59:28.109866 3765 ts_itest-base.cc:246] Waiting for 1 tablets on tserver c68b2409366042dc817c0dfe0c3f8b16 to finish bootstrapping
I20250627 13:59:30.784564 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 9321
I20250627 13:59:30.809551 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 9228
2025-06-27T13:59:30Z chronyd exiting
[ OK ] ListTableCliSimpleParamTest.TestListTables/2 (7052 ms)
[----------] 1 test from ListTableCliSimpleParamTest (7052 ms total)
[----------] 1 test from ListTableCliParamTest
[ RUN ] ListTableCliParamTest.ListTabletWithPartitionInfo/4
I20250627 13:59:30.863760 3765 test_util.cc:276] Using random seed: 374088312
[ OK ] ListTableCliParamTest.ListTabletWithPartitionInfo/4 (11 ms)
[----------] 1 test from ListTableCliParamTest (11 ms total)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest
[ RUN ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0
2025-06-27T13:59:30Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-27T13:59:30Z Disabled control of system clock
I20250627 13:59:30.914882 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.173.126:44191
--webserver_interface=127.3.173.126
--webserver_port=0
--builtin_ntp_servers=127.3.173.84:37861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.173.126:44191 with env {}
W20250627 13:59:31.207444 9495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:59:31.207989 9495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:59:31.208406 9495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:59:31.237823 9495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250627 13:59:31.238123 9495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:59:31.238389 9495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250627 13:59:31.238582 9495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250627 13:59:31.271561 9495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:37861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.173.126:44191
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.173.126:44191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.173.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:59:31.272771 9495 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:59:31.274318 9495 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:59:31.287545 9501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:59:31.287700 9502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:59:31.289149 9504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:59:32.453729 9503 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1164 milliseconds
I20250627 13:59:32.453835 9495 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250627 13:59:32.455070 9495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:59:32.458117 9495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:59:32.459537 9495 hybrid_clock.cc:648] HybridClock initialized: now 1751032772459488 us; error 72 us; skew 500 ppm
I20250627 13:59:32.460274 9495 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:59:32.466521 9495 webserver.cc:469] Webserver started at http://127.3.173.126:43377/ using document root <none> and password file <none>
I20250627 13:59:32.467422 9495 fs_manager.cc:362] Metadata directory not provided
I20250627 13:59:32.467612 9495 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:59:32.468008 9495 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:59:32.472481 9495 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/master-0/data/instance:
uuid: "64b3e79f4000465296417bc05784b56b"
format_stamp: "Formatted at 2025-06-27 13:59:32 on dist-test-slave-rn80"
I20250627 13:59:32.473556 9495 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/master-0/wal/instance:
uuid: "64b3e79f4000465296417bc05784b56b"
format_stamp: "Formatted at 2025-06-27 13:59:32 on dist-test-slave-rn80"
I20250627 13:59:32.480980 9495 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250627 13:59:32.486490 9511 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:59:32.487556 9495 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250627 13:59:32.487860 9495 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/master-0/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/master-0/wal
uuid: "64b3e79f4000465296417bc05784b56b"
format_stamp: "Formatted at 2025-06-27 13:59:32 on dist-test-slave-rn80"
I20250627 13:59:32.488201 9495 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:59:32.535625 9495 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:59:32.537012 9495 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:59:32.537446 9495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:59:32.603421 9495 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.126:44191
I20250627 13:59:32.603506 9562 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.126:44191 every 8 connection(s)
I20250627 13:59:32.605964 9495 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/master-0/data/info.pb
I20250627 13:59:32.610961 9563 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:59:32.612480 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 9495
I20250627 13:59:32.612866 3765 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/master-0/wal/instance
I20250627 13:59:32.634929 9563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64b3e79f4000465296417bc05784b56b: Bootstrap starting.
I20250627 13:59:32.641664 9563 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64b3e79f4000465296417bc05784b56b: Neither blocks nor log segments found. Creating new log.
I20250627 13:59:32.643384 9563 log.cc:826] T 00000000000000000000000000000000 P 64b3e79f4000465296417bc05784b56b: Log is configured to *not* fsync() on all Append() calls
I20250627 13:59:32.647976 9563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64b3e79f4000465296417bc05784b56b: No bootstrap required, opened a new log
I20250627 13:59:32.665560 9563 raft_consensus.cc:357] T 00000000000000000000000000000000 P 64b3e79f4000465296417bc05784b56b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64b3e79f4000465296417bc05784b56b" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 44191 } }
I20250627 13:59:32.666303 9563 raft_consensus.cc:383] T 00000000000000000000000000000000 P 64b3e79f4000465296417bc05784b56b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:59:32.666529 9563 raft_consensus.cc:738] T 00000000000000000000000000000000 P 64b3e79f4000465296417bc05784b56b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64b3e79f4000465296417bc05784b56b, State: Initialized, Role: FOLLOWER
I20250627 13:59:32.667196 9563 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64b3e79f4000465296417bc05784b56b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64b3e79f4000465296417bc05784b56b" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 44191 } }
I20250627 13:59:32.667702 9563 raft_consensus.cc:397] T 00000000000000000000000000000000 P 64b3e79f4000465296417bc05784b56b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250627 13:59:32.667968 9563 raft_consensus.cc:491] T 00000000000000000000000000000000 P 64b3e79f4000465296417bc05784b56b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250627 13:59:32.668255 9563 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 64b3e79f4000465296417bc05784b56b [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:59:32.672909 9563 raft_consensus.cc:513] T 00000000000000000000000000000000 P 64b3e79f4000465296417bc05784b56b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64b3e79f4000465296417bc05784b56b" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 44191 } }
I20250627 13:59:32.673600 9563 leader_election.cc:304] T 00000000000000000000000000000000 P 64b3e79f4000465296417bc05784b56b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64b3e79f4000465296417bc05784b56b; no voters:
I20250627 13:59:32.675287 9563 leader_election.cc:290] T 00000000000000000000000000000000 P 64b3e79f4000465296417bc05784b56b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250627 13:59:32.676173 9568 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 64b3e79f4000465296417bc05784b56b [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:59:32.678336 9568 raft_consensus.cc:695] T 00000000000000000000000000000000 P 64b3e79f4000465296417bc05784b56b [term 1 LEADER]: Becoming Leader. State: Replica: 64b3e79f4000465296417bc05784b56b, State: Running, Role: LEADER
I20250627 13:59:32.679025 9568 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64b3e79f4000465296417bc05784b56b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64b3e79f4000465296417bc05784b56b" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 44191 } }
I20250627 13:59:32.679352 9563 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64b3e79f4000465296417bc05784b56b [sys.catalog]: configured and running, proceeding with master startup.
I20250627 13:59:32.689793 9570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64b3e79f4000465296417bc05784b56b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64b3e79f4000465296417bc05784b56b. Latest consensus state: current_term: 1 leader_uuid: "64b3e79f4000465296417bc05784b56b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64b3e79f4000465296417bc05784b56b" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 44191 } } }
I20250627 13:59:32.689849 9569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64b3e79f4000465296417bc05784b56b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64b3e79f4000465296417bc05784b56b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64b3e79f4000465296417bc05784b56b" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 44191 } } }
I20250627 13:59:32.690723 9569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64b3e79f4000465296417bc05784b56b [sys.catalog]: This master's current role is: LEADER
I20250627 13:59:32.690723 9570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64b3e79f4000465296417bc05784b56b [sys.catalog]: This master's current role is: LEADER
I20250627 13:59:32.696575 9577 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250627 13:59:32.706147 9577 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250627 13:59:32.722610 9577 catalog_manager.cc:1349] Generated new cluster ID: e83de1af403649799eb9f2477d9323b5
I20250627 13:59:32.722903 9577 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250627 13:59:32.742010 9577 catalog_manager.cc:1372] Generated new certificate authority record
I20250627 13:59:32.743448 9577 catalog_manager.cc:1506] Loading token signing keys...
I20250627 13:59:32.761238 9577 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 64b3e79f4000465296417bc05784b56b: Generated new TSK 0
I20250627 13:59:32.762378 9577 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250627 13:59:32.782261 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.173.65:0
--local_ip_for_outbound_sockets=127.3.173.65
--webserver_interface=127.3.173.65
--webserver_port=0
--tserver_master_addrs=127.3.173.126:44191
--builtin_ntp_servers=127.3.173.84:37861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20250627 13:59:33.074731 9587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:59:33.075215 9587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:59:33.075748 9587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:59:33.108373 9587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:59:33.109174 9587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.173.65
I20250627 13:59:33.144445 9587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:37861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.173.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.173.65
--webserver_port=0
--tserver_master_addrs=127.3.173.126:44191
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.173.65
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:59:33.145733 9587 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:59:33.147478 9587 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:59:33.163542 9594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:59:33.166293 9593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:59:33.166464 9596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:59:33.166976 9587 server_base.cc:1048] running on GCE node
I20250627 13:59:34.330372 9587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:59:34.333005 9587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:59:34.334429 9587 hybrid_clock.cc:648] HybridClock initialized: now 1751032774334374 us; error 73 us; skew 500 ppm
I20250627 13:59:34.335196 9587 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:59:34.342299 9587 webserver.cc:469] Webserver started at http://127.3.173.65:34845/ using document root <none> and password file <none>
I20250627 13:59:34.343178 9587 fs_manager.cc:362] Metadata directory not provided
I20250627 13:59:34.343397 9587 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:59:34.343842 9587 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:59:34.348778 9587 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-0/data/instance:
uuid: "59d296ac81f349499f4b306a382bc0cd"
format_stamp: "Formatted at 2025-06-27 13:59:34 on dist-test-slave-rn80"
I20250627 13:59:34.349792 9587 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-0/wal/instance:
uuid: "59d296ac81f349499f4b306a382bc0cd"
format_stamp: "Formatted at 2025-06-27 13:59:34 on dist-test-slave-rn80"
I20250627 13:59:34.356791 9587 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250627 13:59:34.362944 9603 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:59:34.364145 9587 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250627 13:59:34.364544 9587 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-0/wal
uuid: "59d296ac81f349499f4b306a382bc0cd"
format_stamp: "Formatted at 2025-06-27 13:59:34 on dist-test-slave-rn80"
I20250627 13:59:34.364913 9587 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:59:34.419222 9587 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:59:34.420704 9587 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:59:34.421190 9587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:59:34.423835 9587 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:59:34.427752 9587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:59:34.427975 9587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:59:34.428220 9587 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:59:34.428390 9587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:59:34.561729 9587 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.65:43065
I20250627 13:59:34.561829 9715 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.65:43065 every 8 connection(s)
I20250627 13:59:34.564270 9587 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-0/data/info.pb
I20250627 13:59:34.573383 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 9587
I20250627 13:59:34.573770 3765 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-0/wal/instance
I20250627 13:59:34.580144 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.173.66:0
--local_ip_for_outbound_sockets=127.3.173.66
--webserver_interface=127.3.173.66
--webserver_port=0
--tserver_master_addrs=127.3.173.126:44191
--builtin_ntp_servers=127.3.173.84:37861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250627 13:59:34.585073 9716 heartbeater.cc:344] Connected to a master server at 127.3.173.126:44191
I20250627 13:59:34.585577 9716 heartbeater.cc:461] Registering TS with master...
I20250627 13:59:34.586858 9716 heartbeater.cc:507] Master 127.3.173.126:44191 requested a full tablet report, sending...
I20250627 13:59:34.589505 9528 ts_manager.cc:194] Registered new tserver with Master: 59d296ac81f349499f4b306a382bc0cd (127.3.173.65:43065)
I20250627 13:59:34.591485 9528 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.173.65:60381
W20250627 13:59:34.872030 9720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:59:34.872555 9720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:59:34.873052 9720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:59:34.904284 9720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:59:34.905119 9720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.173.66
I20250627 13:59:34.939594 9720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:37861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.173.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.173.66
--webserver_port=0
--tserver_master_addrs=127.3.173.126:44191
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.173.66
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:59:34.940883 9720 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:59:34.942487 9720 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:59:34.956460 9728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:59:35.594616 9716 heartbeater.cc:499] Master 127.3.173.126:44191 was elected leader, sending a full tablet report...
W20250627 13:59:34.956521 9727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:59:34.958528 9730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:59:36.078039 9729 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250627 13:59:36.078088 9720 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250627 13:59:36.082412 9720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:59:36.084993 9720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:59:36.086367 9720 hybrid_clock.cc:648] HybridClock initialized: now 1751032776086332 us; error 54 us; skew 500 ppm
I20250627 13:59:36.087132 9720 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:59:36.093416 9720 webserver.cc:469] Webserver started at http://127.3.173.66:46235/ using document root <none> and password file <none>
I20250627 13:59:36.094406 9720 fs_manager.cc:362] Metadata directory not provided
I20250627 13:59:36.094623 9720 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:59:36.095057 9720 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:59:36.099359 9720 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-1/data/instance:
uuid: "55235744bbb244bbb4cce13e237f951d"
format_stamp: "Formatted at 2025-06-27 13:59:36 on dist-test-slave-rn80"
I20250627 13:59:36.100423 9720 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-1/wal/instance:
uuid: "55235744bbb244bbb4cce13e237f951d"
format_stamp: "Formatted at 2025-06-27 13:59:36 on dist-test-slave-rn80"
I20250627 13:59:36.107055 9720 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250627 13:59:36.112931 9737 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:59:36.114250 9720 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.004s
I20250627 13:59:36.114578 9720 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-1/wal
uuid: "55235744bbb244bbb4cce13e237f951d"
format_stamp: "Formatted at 2025-06-27 13:59:36 on dist-test-slave-rn80"
I20250627 13:59:36.114903 9720 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:59:36.189497 9720 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:59:36.190860 9720 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:59:36.191247 9720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:59:36.193567 9720 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:59:36.197494 9720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:59:36.197677 9720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:59:36.197943 9720 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:59:36.198094 9720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:59:36.324579 9720 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.66:44059
I20250627 13:59:36.324676 9849 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.66:44059 every 8 connection(s)
I20250627 13:59:36.326949 9720 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-1/data/info.pb
I20250627 13:59:36.333875 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 9720
I20250627 13:59:36.334501 3765 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-1/wal/instance
I20250627 13:59:36.340905 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.173.67:0
--local_ip_for_outbound_sockets=127.3.173.67
--webserver_interface=127.3.173.67
--webserver_port=0
--tserver_master_addrs=127.3.173.126:44191
--builtin_ntp_servers=127.3.173.84:37861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250627 13:59:36.347124 9850 heartbeater.cc:344] Connected to a master server at 127.3.173.126:44191
I20250627 13:59:36.347525 9850 heartbeater.cc:461] Registering TS with master...
I20250627 13:59:36.348603 9850 heartbeater.cc:507] Master 127.3.173.126:44191 requested a full tablet report, sending...
I20250627 13:59:36.350633 9528 ts_manager.cc:194] Registered new tserver with Master: 55235744bbb244bbb4cce13e237f951d (127.3.173.66:44059)
I20250627 13:59:36.351768 9528 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.173.66:59297
W20250627 13:59:36.632368 9854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:59:36.632880 9854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:59:36.633383 9854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:59:36.663514 9854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:59:36.664364 9854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.173.67
I20250627 13:59:36.698340 9854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:37861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.173.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.173.67
--webserver_port=0
--tserver_master_addrs=127.3.173.126:44191
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.173.67
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:59:36.699723 9854 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:59:36.701282 9854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:59:36.715791 9860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:59:37.355073 9850 heartbeater.cc:499] Master 127.3.173.126:44191 was elected leader, sending a full tablet report...
W20250627 13:59:36.716982 9861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:59:36.719112 9854 server_base.cc:1048] running on GCE node
W20250627 13:59:37.858374 9863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:59:37.860463 9854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:59:37.863075 9854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:59:37.864476 9854 hybrid_clock.cc:648] HybridClock initialized: now 1751032777864439 us; error 48 us; skew 500 ppm
I20250627 13:59:37.865200 9854 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:59:37.875311 9854 webserver.cc:469] Webserver started at http://127.3.173.67:37163/ using document root <none> and password file <none>
I20250627 13:59:37.876160 9854 fs_manager.cc:362] Metadata directory not provided
I20250627 13:59:37.876338 9854 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:59:37.876726 9854 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:59:37.880919 9854 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-2/data/instance:
uuid: "e5c3905e0806414faebde321b160a377"
format_stamp: "Formatted at 2025-06-27 13:59:37 on dist-test-slave-rn80"
I20250627 13:59:37.881997 9854 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-2/wal/instance:
uuid: "e5c3905e0806414faebde321b160a377"
format_stamp: "Formatted at 2025-06-27 13:59:37 on dist-test-slave-rn80"
I20250627 13:59:37.888581 9854 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250627 13:59:37.893844 9870 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:59:37.894814 9854 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250627 13:59:37.895072 9854 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-2/wal
uuid: "e5c3905e0806414faebde321b160a377"
format_stamp: "Formatted at 2025-06-27 13:59:37 on dist-test-slave-rn80"
I20250627 13:59:37.895359 9854 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:59:37.958864 9854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:59:37.960242 9854 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:59:37.960625 9854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:59:37.962972 9854 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:59:37.967020 9854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:59:37.967206 9854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:59:37.967396 9854 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:59:37.967532 9854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:59:38.094758 9854 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.67:40835
I20250627 13:59:38.094851 9982 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.67:40835 every 8 connection(s)
I20250627 13:59:38.097123 9854 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-2/data/info.pb
I20250627 13:59:38.106542 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 9854
I20250627 13:59:38.107036 3765 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-2/wal/instance
I20250627 13:59:38.117640 9983 heartbeater.cc:344] Connected to a master server at 127.3.173.126:44191
I20250627 13:59:38.118050 9983 heartbeater.cc:461] Registering TS with master...
I20250627 13:59:38.119235 9983 heartbeater.cc:507] Master 127.3.173.126:44191 requested a full tablet report, sending...
I20250627 13:59:38.121618 9528 ts_manager.cc:194] Registered new tserver with Master: e5c3905e0806414faebde321b160a377 (127.3.173.67:40835)
I20250627 13:59:38.122802 9528 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.173.67:35981
I20250627 13:59:38.126638 3765 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250627 13:59:38.155273 3765 test_util.cc:276] Using random seed: 381379828
I20250627 13:59:38.194516 9528 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59866:
name: "pre_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250627 13:59:38.196839 9528 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table pre_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250627 13:59:38.244712 9785 tablet_service.cc:1468] Processing CreateTablet for tablet 8fc980318ce7411db461a2ef0d3dc4cb (DEFAULT_TABLE table=pre_rebuild [id=f8c27828794343bcafe854c9389e9583]), partition=RANGE (key) PARTITION UNBOUNDED
I20250627 13:59:38.246721 9785 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fc980318ce7411db461a2ef0d3dc4cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:59:38.250913 9651 tablet_service.cc:1468] Processing CreateTablet for tablet 8fc980318ce7411db461a2ef0d3dc4cb (DEFAULT_TABLE table=pre_rebuild [id=f8c27828794343bcafe854c9389e9583]), partition=RANGE (key) PARTITION UNBOUNDED
I20250627 13:59:38.251510 9918 tablet_service.cc:1468] Processing CreateTablet for tablet 8fc980318ce7411db461a2ef0d3dc4cb (DEFAULT_TABLE table=pre_rebuild [id=f8c27828794343bcafe854c9389e9583]), partition=RANGE (key) PARTITION UNBOUNDED
I20250627 13:59:38.252686 9651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fc980318ce7411db461a2ef0d3dc4cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:59:38.253243 9918 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fc980318ce7411db461a2ef0d3dc4cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:59:38.267845 10007 tablet_bootstrap.cc:492] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d: Bootstrap starting.
I20250627 13:59:38.273051 10007 tablet_bootstrap.cc:654] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d: Neither blocks nor log segments found. Creating new log.
I20250627 13:59:38.275077 10007 log.cc:826] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d: Log is configured to *not* fsync() on all Append() calls
I20250627 13:59:38.280339 10010 tablet_bootstrap.cc:492] T 8fc980318ce7411db461a2ef0d3dc4cb P e5c3905e0806414faebde321b160a377: Bootstrap starting.
I20250627 13:59:38.280359 10009 tablet_bootstrap.cc:492] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd: Bootstrap starting.
I20250627 13:59:38.281142 10007 tablet_bootstrap.cc:492] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d: No bootstrap required, opened a new log
I20250627 13:59:38.281658 10007 ts_tablet_manager.cc:1397] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.007s
I20250627 13:59:38.285547 10009 tablet_bootstrap.cc:654] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd: Neither blocks nor log segments found. Creating new log.
I20250627 13:59:38.287500 10010 tablet_bootstrap.cc:654] T 8fc980318ce7411db461a2ef0d3dc4cb P e5c3905e0806414faebde321b160a377: Neither blocks nor log segments found. Creating new log.
I20250627 13:59:38.287791 10009 log.cc:826] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd: Log is configured to *not* fsync() on all Append() calls
I20250627 13:59:38.289896 10010 log.cc:826] T 8fc980318ce7411db461a2ef0d3dc4cb P e5c3905e0806414faebde321b160a377: Log is configured to *not* fsync() on all Append() calls
I20250627 13:59:38.293339 10009 tablet_bootstrap.cc:492] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd: No bootstrap required, opened a new log
I20250627 13:59:38.293725 10009 ts_tablet_manager.cc:1397] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.002s
I20250627 13:59:38.295192 10010 tablet_bootstrap.cc:492] T 8fc980318ce7411db461a2ef0d3dc4cb P e5c3905e0806414faebde321b160a377: No bootstrap required, opened a new log
I20250627 13:59:38.295650 10010 ts_tablet_manager.cc:1397] T 8fc980318ce7411db461a2ef0d3dc4cb P e5c3905e0806414faebde321b160a377: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.004s
I20250627 13:59:38.307612 10007 raft_consensus.cc:357] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55235744bbb244bbb4cce13e237f951d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 44059 } } peers { permanent_uuid: "59d296ac81f349499f4b306a382bc0cd" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43065 } } peers { permanent_uuid: "e5c3905e0806414faebde321b160a377" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40835 } }
I20250627 13:59:38.308683 10007 raft_consensus.cc:383] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:59:38.309032 10007 raft_consensus.cc:738] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55235744bbb244bbb4cce13e237f951d, State: Initialized, Role: FOLLOWER
I20250627 13:59:38.309937 10007 consensus_queue.cc:260] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55235744bbb244bbb4cce13e237f951d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 44059 } } peers { permanent_uuid: "59d296ac81f349499f4b306a382bc0cd" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43065 } } peers { permanent_uuid: "e5c3905e0806414faebde321b160a377" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40835 } }
I20250627 13:59:38.313967 10007 ts_tablet_manager.cc:1428] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d: Time spent starting tablet: real 0.032s user 0.024s sys 0.006s
I20250627 13:59:38.314651 10009 raft_consensus.cc:357] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55235744bbb244bbb4cce13e237f951d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 44059 } } peers { permanent_uuid: "59d296ac81f349499f4b306a382bc0cd" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43065 } } peers { permanent_uuid: "e5c3905e0806414faebde321b160a377" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40835 } }
I20250627 13:59:38.315358 10009 raft_consensus.cc:383] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:59:38.315666 10009 raft_consensus.cc:738] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59d296ac81f349499f4b306a382bc0cd, State: Initialized, Role: FOLLOWER
I20250627 13:59:38.316527 10009 consensus_queue.cc:260] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55235744bbb244bbb4cce13e237f951d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 44059 } } peers { permanent_uuid: "59d296ac81f349499f4b306a382bc0cd" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43065 } } peers { permanent_uuid: "e5c3905e0806414faebde321b160a377" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40835 } }
I20250627 13:59:38.321519 10009 ts_tablet_manager.cc:1428] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd: Time spent starting tablet: real 0.028s user 0.026s sys 0.002s
I20250627 13:59:38.321300 10010 raft_consensus.cc:357] T 8fc980318ce7411db461a2ef0d3dc4cb P e5c3905e0806414faebde321b160a377 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55235744bbb244bbb4cce13e237f951d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 44059 } } peers { permanent_uuid: "59d296ac81f349499f4b306a382bc0cd" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43065 } } peers { permanent_uuid: "e5c3905e0806414faebde321b160a377" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40835 } }
I20250627 13:59:38.323138 10010 raft_consensus.cc:383] T 8fc980318ce7411db461a2ef0d3dc4cb P e5c3905e0806414faebde321b160a377 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250627 13:59:38.323158 9717 tablet.cc:2378] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250627 13:59:38.323436 10010 raft_consensus.cc:738] T 8fc980318ce7411db461a2ef0d3dc4cb P e5c3905e0806414faebde321b160a377 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c3905e0806414faebde321b160a377, State: Initialized, Role: FOLLOWER
I20250627 13:59:38.324383 10010 consensus_queue.cc:260] T 8fc980318ce7411db461a2ef0d3dc4cb P e5c3905e0806414faebde321b160a377 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55235744bbb244bbb4cce13e237f951d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 44059 } } peers { permanent_uuid: "59d296ac81f349499f4b306a382bc0cd" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43065 } } peers { permanent_uuid: "e5c3905e0806414faebde321b160a377" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40835 } }
I20250627 13:59:38.327947 9983 heartbeater.cc:499] Master 127.3.173.126:44191 was elected leader, sending a full tablet report...
I20250627 13:59:38.329231 10010 ts_tablet_manager.cc:1428] T 8fc980318ce7411db461a2ef0d3dc4cb P e5c3905e0806414faebde321b160a377: Time spent starting tablet: real 0.033s user 0.027s sys 0.003s
W20250627 13:59:38.333148 9851 tablet.cc:2378] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250627 13:59:38.351609 9984 tablet.cc:2378] T 8fc980318ce7411db461a2ef0d3dc4cb P e5c3905e0806414faebde321b160a377: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250627 13:59:38.384629 10013 raft_consensus.cc:491] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:59:38.385134 10013 raft_consensus.cc:513] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55235744bbb244bbb4cce13e237f951d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 44059 } } peers { permanent_uuid: "59d296ac81f349499f4b306a382bc0cd" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43065 } } peers { permanent_uuid: "e5c3905e0806414faebde321b160a377" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40835 } }
I20250627 13:59:38.387326 10013 leader_election.cc:290] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59d296ac81f349499f4b306a382bc0cd (127.3.173.65:43065), e5c3905e0806414faebde321b160a377 (127.3.173.67:40835)
I20250627 13:59:38.398447 9938 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8fc980318ce7411db461a2ef0d3dc4cb" candidate_uuid: "55235744bbb244bbb4cce13e237f951d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c3905e0806414faebde321b160a377" is_pre_election: true
I20250627 13:59:38.398469 9671 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8fc980318ce7411db461a2ef0d3dc4cb" candidate_uuid: "55235744bbb244bbb4cce13e237f951d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59d296ac81f349499f4b306a382bc0cd" is_pre_election: true
I20250627 13:59:38.399354 9671 raft_consensus.cc:2466] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55235744bbb244bbb4cce13e237f951d in term 0.
I20250627 13:59:38.399353 9938 raft_consensus.cc:2466] T 8fc980318ce7411db461a2ef0d3dc4cb P e5c3905e0806414faebde321b160a377 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55235744bbb244bbb4cce13e237f951d in term 0.
I20250627 13:59:38.400712 9740 leader_election.cc:304] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 55235744bbb244bbb4cce13e237f951d, 59d296ac81f349499f4b306a382bc0cd, e5c3905e0806414faebde321b160a377; no voters:
I20250627 13:59:38.401412 10013 raft_consensus.cc:2802] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:59:38.401727 10013 raft_consensus.cc:491] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:59:38.401995 10013 raft_consensus.cc:3058] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:59:38.406214 10013 raft_consensus.cc:513] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55235744bbb244bbb4cce13e237f951d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 44059 } } peers { permanent_uuid: "59d296ac81f349499f4b306a382bc0cd" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43065 } } peers { permanent_uuid: "e5c3905e0806414faebde321b160a377" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40835 } }
I20250627 13:59:38.407533 10013 leader_election.cc:290] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d [CANDIDATE]: Term 1 election: Requested vote from peers 59d296ac81f349499f4b306a382bc0cd (127.3.173.65:43065), e5c3905e0806414faebde321b160a377 (127.3.173.67:40835)
I20250627 13:59:38.408354 9671 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8fc980318ce7411db461a2ef0d3dc4cb" candidate_uuid: "55235744bbb244bbb4cce13e237f951d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59d296ac81f349499f4b306a382bc0cd"
I20250627 13:59:38.408468 9938 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8fc980318ce7411db461a2ef0d3dc4cb" candidate_uuid: "55235744bbb244bbb4cce13e237f951d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c3905e0806414faebde321b160a377"
I20250627 13:59:38.408764 9671 raft_consensus.cc:3058] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:59:38.408898 9938 raft_consensus.cc:3058] T 8fc980318ce7411db461a2ef0d3dc4cb P e5c3905e0806414faebde321b160a377 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:59:38.413092 9671 raft_consensus.cc:2466] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55235744bbb244bbb4cce13e237f951d in term 1.
I20250627 13:59:38.413196 9938 raft_consensus.cc:2466] T 8fc980318ce7411db461a2ef0d3dc4cb P e5c3905e0806414faebde321b160a377 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55235744bbb244bbb4cce13e237f951d in term 1.
I20250627 13:59:38.414073 9739 leader_election.cc:304] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55235744bbb244bbb4cce13e237f951d, 59d296ac81f349499f4b306a382bc0cd; no voters:
I20250627 13:59:38.414737 10013 raft_consensus.cc:2802] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:59:38.416234 10013 raft_consensus.cc:695] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d [term 1 LEADER]: Becoming Leader. State: Replica: 55235744bbb244bbb4cce13e237f951d, State: Running, Role: LEADER
I20250627 13:59:38.416951 10013 consensus_queue.cc:237] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55235744bbb244bbb4cce13e237f951d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 44059 } } peers { permanent_uuid: "59d296ac81f349499f4b306a382bc0cd" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43065 } } peers { permanent_uuid: "e5c3905e0806414faebde321b160a377" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40835 } }
I20250627 13:59:38.428155 9526 catalog_manager.cc:5582] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d reported cstate change: term changed from 0 to 1, leader changed from <none> to 55235744bbb244bbb4cce13e237f951d (127.3.173.66). New cstate: current_term: 1 leader_uuid: "55235744bbb244bbb4cce13e237f951d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55235744bbb244bbb4cce13e237f951d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 44059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59d296ac81f349499f4b306a382bc0cd" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5c3905e0806414faebde321b160a377" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40835 } health_report { overall_health: UNKNOWN } } }
I20250627 13:59:38.560850 9938 raft_consensus.cc:1273] T 8fc980318ce7411db461a2ef0d3dc4cb P e5c3905e0806414faebde321b160a377 [term 1 FOLLOWER]: Refusing update from remote peer 55235744bbb244bbb4cce13e237f951d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250627 13:59:38.561182 9671 raft_consensus.cc:1273] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd [term 1 FOLLOWER]: Refusing update from remote peer 55235744bbb244bbb4cce13e237f951d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250627 13:59:38.562656 10013 consensus_queue.cc:1035] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c3905e0806414faebde321b160a377" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:59:38.563529 10018 consensus_queue.cc:1035] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d [LEADER]: Connected to new peer: Peer: permanent_uuid: "59d296ac81f349499f4b306a382bc0cd" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:59:38.596276 10025 mvcc.cc:204] Tried to move back new op lower bound from 7172230260968853504 to 7172230260402978816. Current Snapshot: MvccSnapshot[applied={T|T < 7172230260968853504}]
W20250627 13:59:42.631733 10022 meta_cache.cc:1261] Time spent looking up entry by key: real 0.051s user 0.002s sys 0.000s
I20250627 13:59:43.530274 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 9495
W20250627 13:59:43.684880 9983 heartbeater.cc:646] Failed to heartbeat to 127.3.173.126:44191 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.173.126:44191: connect: Connection refused (error 111)
W20250627 13:59:43.686146 9850 heartbeater.cc:646] Failed to heartbeat to 127.3.173.126:44191 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.173.126:44191: connect: Connection refused (error 111)
W20250627 13:59:43.726681 9716 heartbeater.cc:646] Failed to heartbeat to 127.3.173.126:44191 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.173.126:44191: connect: Connection refused (error 111)
W20250627 13:59:43.892316 10053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:59:43.892886 10053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:59:43.922858 10053 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250627 13:59:45.280345 10053 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.311s user 0.454s sys 0.841s
W20250627 13:59:45.280697 10053 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.312s user 0.454s sys 0.841s
I20250627 13:59:45.411506 10053 minidump.cc:252] Setting minidump size limit to 20M
I20250627 13:59:45.413867 10053 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:59:45.414922 10053 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:59:45.425820 10087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:59:45.425830 10088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:59:45.555240 10090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:59:45.559779 10053 server_base.cc:1048] running on GCE node
I20250627 13:59:45.560871 10053 hybrid_clock.cc:584] initializing the hybrid clock with 'system' time source
I20250627 13:59:45.561316 10053 hybrid_clock.cc:648] HybridClock initialized: now 1751032785561290 us; error 250016 us; skew 500 ppm
I20250627 13:59:45.561956 10053 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:59:45.566704 10053 webserver.cc:469] Webserver started at http://0.0.0.0:37771/ using document root <none> and password file <none>
I20250627 13:59:45.567540 10053 fs_manager.cc:362] Metadata directory not provided
I20250627 13:59:45.567775 10053 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:59:45.568189 10053 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:59:45.572413 10053 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/master-0/data/instance:
uuid: "ef9ad88cfad848aca4aeaaa2d343faf9"
format_stamp: "Formatted at 2025-06-27 13:59:45 on dist-test-slave-rn80"
I20250627 13:59:45.573450 10053 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/master-0/wal/instance:
uuid: "ef9ad88cfad848aca4aeaaa2d343faf9"
format_stamp: "Formatted at 2025-06-27 13:59:45 on dist-test-slave-rn80"
I20250627 13:59:45.579272 10053 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20250627 13:59:45.583922 10095 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:59:45.584796 10053 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250627 13:59:45.585095 10053 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/master-0/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/master-0/wal
uuid: "ef9ad88cfad848aca4aeaaa2d343faf9"
format_stamp: "Formatted at 2025-06-27 13:59:45 on dist-test-slave-rn80"
I20250627 13:59:45.585391 10053 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:59:45.754194 10053 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:59:45.755506 10053 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:59:45.755929 10053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:59:45.760463 10053 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:59:45.774628 10053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9: Bootstrap starting.
I20250627 13:59:45.779453 10053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9: Neither blocks nor log segments found. Creating new log.
I20250627 13:59:45.781121 10053 log.cc:826] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9: Log is configured to *not* fsync() on all Append() calls
I20250627 13:59:45.784897 10053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9: No bootstrap required, opened a new log
I20250627 13:59:45.800550 10053 raft_consensus.cc:357] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9ad88cfad848aca4aeaaa2d343faf9" member_type: VOTER }
I20250627 13:59:45.801110 10053 raft_consensus.cc:383] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:59:45.801393 10053 raft_consensus.cc:738] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef9ad88cfad848aca4aeaaa2d343faf9, State: Initialized, Role: FOLLOWER
I20250627 13:59:45.802204 10053 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9ad88cfad848aca4aeaaa2d343faf9" member_type: VOTER }
I20250627 13:59:45.802654 10053 raft_consensus.cc:397] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250627 13:59:45.802865 10053 raft_consensus.cc:491] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250627 13:59:45.803115 10053 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:59:45.806944 10053 raft_consensus.cc:513] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9ad88cfad848aca4aeaaa2d343faf9" member_type: VOTER }
I20250627 13:59:45.807545 10053 leader_election.cc:304] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef9ad88cfad848aca4aeaaa2d343faf9; no voters:
I20250627 13:59:45.809253 10053 leader_election.cc:290] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250627 13:59:45.809584 10105 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:59:45.811632 10105 raft_consensus.cc:695] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9 [term 1 LEADER]: Becoming Leader. State: Replica: ef9ad88cfad848aca4aeaaa2d343faf9, State: Running, Role: LEADER
I20250627 13:59:45.812501 10105 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9ad88cfad848aca4aeaaa2d343faf9" member_type: VOTER }
I20250627 13:59:45.819893 10106 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef9ad88cfad848aca4aeaaa2d343faf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9ad88cfad848aca4aeaaa2d343faf9" member_type: VOTER } }
I20250627 13:59:45.820128 10107 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef9ad88cfad848aca4aeaaa2d343faf9. Latest consensus state: current_term: 1 leader_uuid: "ef9ad88cfad848aca4aeaaa2d343faf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9ad88cfad848aca4aeaaa2d343faf9" member_type: VOTER } }
I20250627 13:59:45.820827 10107 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9 [sys.catalog]: This master's current role is: LEADER
I20250627 13:59:45.821262 10106 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9 [sys.catalog]: This master's current role is: LEADER
I20250627 13:59:45.833079 10053 tablet_replica.cc:331] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9: stopping tablet replica
I20250627 13:59:45.833575 10053 raft_consensus.cc:2241] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9 [term 1 LEADER]: Raft consensus shutting down.
I20250627 13:59:45.833950 10053 raft_consensus.cc:2270] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:59:45.835907 10053 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250627 13:59:45.836318 10053 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250627 13:59:45.861274 10053 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
I20250627 13:59:46.891083 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 9587
I20250627 13:59:46.925905 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 9720
I20250627 13:59:46.968318 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 9854
I20250627 13:59:46.999302 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.173.126:44191
--webserver_interface=127.3.173.126
--webserver_port=43377
--builtin_ntp_servers=127.3.173.84:37861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.173.126:44191 with env {}
W20250627 13:59:47.290596 10115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:59:47.291172 10115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:59:47.291613 10115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:59:47.322024 10115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250627 13:59:47.322387 10115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:59:47.322624 10115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250627 13:59:47.322880 10115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250627 13:59:47.357127 10115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:37861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.173.126:44191
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.173.126:44191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.173.126
--webserver_port=43377
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:59:47.358525 10115 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:59:47.360138 10115 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:59:47.375766 10121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:59:47.375833 10122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:59:47.376642 10124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:59:47.378201 10115 server_base.cc:1048] running on GCE node
I20250627 13:59:48.540921 10115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:59:48.543349 10115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:59:48.544679 10115 hybrid_clock.cc:648] HybridClock initialized: now 1751032788544654 us; error 39 us; skew 500 ppm
I20250627 13:59:48.545441 10115 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:59:48.551414 10115 webserver.cc:469] Webserver started at http://127.3.173.126:43377/ using document root <none> and password file <none>
I20250627 13:59:48.552248 10115 fs_manager.cc:362] Metadata directory not provided
I20250627 13:59:48.552456 10115 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:59:48.559861 10115 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20250627 13:59:48.564076 10131 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:59:48.565085 10115 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250627 13:59:48.565461 10115 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/master-0/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/master-0/wal
uuid: "ef9ad88cfad848aca4aeaaa2d343faf9"
format_stamp: "Formatted at 2025-06-27 13:59:45 on dist-test-slave-rn80"
I20250627 13:59:48.567339 10115 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:59:48.617758 10115 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:59:48.619225 10115 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:59:48.619696 10115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:59:48.686369 10115 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.126:44191
I20250627 13:59:48.686460 10182 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.126:44191 every 8 connection(s)
I20250627 13:59:48.688938 10115 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/master-0/data/info.pb
I20250627 13:59:48.692020 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 10115
I20250627 13:59:48.693972 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.173.65:43065
--local_ip_for_outbound_sockets=127.3.173.65
--tserver_master_addrs=127.3.173.126:44191
--webserver_port=34845
--webserver_interface=127.3.173.65
--builtin_ntp_servers=127.3.173.84:37861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250627 13:59:48.703341 10183 sys_catalog.cc:263] Verifying existing consensus state
I20250627 13:59:48.714861 10183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9: Bootstrap starting.
I20250627 13:59:48.723927 10183 log.cc:826] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9: Log is configured to *not* fsync() on all Append() calls
I20250627 13:59:48.734892 10183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:59:48.735577 10183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9: Bootstrap complete.
I20250627 13:59:48.754945 10183 raft_consensus.cc:357] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9ad88cfad848aca4aeaaa2d343faf9" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 44191 } }
I20250627 13:59:48.755614 10183 raft_consensus.cc:738] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef9ad88cfad848aca4aeaaa2d343faf9, State: Initialized, Role: FOLLOWER
I20250627 13:59:48.756301 10183 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9ad88cfad848aca4aeaaa2d343faf9" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 44191 } }
I20250627 13:59:48.756799 10183 raft_consensus.cc:397] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250627 13:59:48.757040 10183 raft_consensus.cc:491] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250627 13:59:48.757311 10183 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:59:48.761178 10183 raft_consensus.cc:513] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9ad88cfad848aca4aeaaa2d343faf9" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 44191 } }
I20250627 13:59:48.761773 10183 leader_election.cc:304] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef9ad88cfad848aca4aeaaa2d343faf9; no voters:
I20250627 13:59:48.763489 10183 leader_election.cc:290] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250627 13:59:48.763852 10187 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9 [term 2 FOLLOWER]: Leader election won for term 2
I20250627 13:59:48.766995 10187 raft_consensus.cc:695] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9 [term 2 LEADER]: Becoming Leader. State: Replica: ef9ad88cfad848aca4aeaaa2d343faf9, State: Running, Role: LEADER
I20250627 13:59:48.767980 10183 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9 [sys.catalog]: configured and running, proceeding with master startup.
I20250627 13:59:48.767796 10187 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9ad88cfad848aca4aeaaa2d343faf9" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 44191 } }
I20250627 13:59:48.778709 10188 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ef9ad88cfad848aca4aeaaa2d343faf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9ad88cfad848aca4aeaaa2d343faf9" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 44191 } } }
I20250627 13:59:48.779325 10188 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9 [sys.catalog]: This master's current role is: LEADER
I20250627 13:59:48.777915 10189 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef9ad88cfad848aca4aeaaa2d343faf9. Latest consensus state: current_term: 2 leader_uuid: "ef9ad88cfad848aca4aeaaa2d343faf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9ad88cfad848aca4aeaaa2d343faf9" member_type: VOTER last_known_addr { host: "127.3.173.126" port: 44191 } } }
I20250627 13:59:48.782419 10189 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9 [sys.catalog]: This master's current role is: LEADER
I20250627 13:59:48.790416 10194 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250627 13:59:48.803468 10194 catalog_manager.cc:671] Loaded metadata for table pre_rebuild [id=64bdffa0d01c4ba1ae3cd8b5812b187f]
I20250627 13:59:48.810945 10194 tablet_loader.cc:96] loaded metadata for tablet 8fc980318ce7411db461a2ef0d3dc4cb (table pre_rebuild [id=64bdffa0d01c4ba1ae3cd8b5812b187f])
I20250627 13:59:48.812592 10194 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250627 13:59:48.844547 10194 catalog_manager.cc:1349] Generated new cluster ID: 1a4993d9069e496aa3366160de693d25
I20250627 13:59:48.844834 10194 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250627 13:59:48.860282 10205 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250627 13:59:48.877969 10194 catalog_manager.cc:1372] Generated new certificate authority record
I20250627 13:59:48.879982 10194 catalog_manager.cc:1506] Loading token signing keys...
I20250627 13:59:48.898159 10194 catalog_manager.cc:5955] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9: Generated new TSK 0
I20250627 13:59:48.899145 10194 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250627 13:59:49.031244 10185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:59:49.031739 10185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:59:49.032246 10185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:59:49.062366 10185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:59:49.063192 10185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.173.65
I20250627 13:59:49.095801 10185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:37861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.173.65:43065
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.173.65
--webserver_port=34845
--tserver_master_addrs=127.3.173.126:44191
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.173.65
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:59:49.097095 10185 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:59:49.098688 10185 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:59:49.115438 10212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:59:49.118088 10185 server_base.cc:1048] running on GCE node
W20250627 13:59:49.118031 10214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:59:49.117290 10211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:59:50.248143 10185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:59:50.250808 10185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:59:50.252198 10185 hybrid_clock.cc:648] HybridClock initialized: now 1751032790252160 us; error 46 us; skew 500 ppm
I20250627 13:59:50.253168 10185 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:59:50.265452 10185 webserver.cc:469] Webserver started at http://127.3.173.65:34845/ using document root <none> and password file <none>
I20250627 13:59:50.266620 10185 fs_manager.cc:362] Metadata directory not provided
I20250627 13:59:50.266866 10185 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:59:50.276656 10185 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20250627 13:59:50.282182 10221 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:59:50.283282 10185 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20250627 13:59:50.283645 10185 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-0/wal
uuid: "59d296ac81f349499f4b306a382bc0cd"
format_stamp: "Formatted at 2025-06-27 13:59:34 on dist-test-slave-rn80"
I20250627 13:59:50.286213 10185 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:59:50.356325 10185 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:59:50.357771 10185 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:59:50.358230 10185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:59:50.361199 10185 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:59:50.367142 10228 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250627 13:59:50.374413 10185 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250627 13:59:50.374642 10185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.002s
I20250627 13:59:50.374917 10185 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250627 13:59:50.379395 10185 ts_tablet_manager.cc:610] Registered 1 tablets
I20250627 13:59:50.379642 10185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.003s sys 0.001s
I20250627 13:59:50.380733 10228 tablet_bootstrap.cc:492] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd: Bootstrap starting.
I20250627 13:59:50.553473 10185 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.65:43065
I20250627 13:59:50.553704 10334 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.65:43065 every 8 connection(s)
I20250627 13:59:50.556849 10185 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-0/data/info.pb
I20250627 13:59:50.559697 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 10185
I20250627 13:59:50.561549 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.173.66:44059
--local_ip_for_outbound_sockets=127.3.173.66
--tserver_master_addrs=127.3.173.126:44191
--webserver_port=46235
--webserver_interface=127.3.173.66
--builtin_ntp_servers=127.3.173.84:37861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250627 13:59:50.610175 10335 heartbeater.cc:344] Connected to a master server at 127.3.173.126:44191
I20250627 13:59:50.610615 10335 heartbeater.cc:461] Registering TS with master...
I20250627 13:59:50.611539 10335 heartbeater.cc:507] Master 127.3.173.126:44191 requested a full tablet report, sending...
I20250627 13:59:50.614849 10148 ts_manager.cc:194] Registered new tserver with Master: 59d296ac81f349499f4b306a382bc0cd (127.3.173.65:43065)
I20250627 13:59:50.619583 10148 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.173.65:38345
I20250627 13:59:50.709839 10228 log.cc:826] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd: Log is configured to *not* fsync() on all Append() calls
W20250627 13:59:50.890462 10339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:59:50.890959 10339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:59:50.891426 10339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:59:50.922463 10339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:59:50.923303 10339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.173.66
I20250627 13:59:50.956980 10339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:37861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.173.66:44059
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.173.66
--webserver_port=46235
--tserver_master_addrs=127.3.173.126:44191
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.173.66
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:59:50.958312 10339 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:59:50.959887 10339 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:59:50.977164 10347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:59:51.623268 10335 heartbeater.cc:499] Master 127.3.173.126:44191 was elected leader, sending a full tablet report...
W20250627 13:59:50.977233 10349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:59:50.978350 10346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:59:52.160840 10348 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250627 13:59:52.160893 10339 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250627 13:59:52.165251 10339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:59:52.167924 10339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:59:52.169338 10339 hybrid_clock.cc:648] HybridClock initialized: now 1751032792169283 us; error 73 us; skew 500 ppm
I20250627 13:59:52.170231 10339 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:59:52.177155 10339 webserver.cc:469] Webserver started at http://127.3.173.66:46235/ using document root <none> and password file <none>
I20250627 13:59:52.178314 10339 fs_manager.cc:362] Metadata directory not provided
I20250627 13:59:52.178610 10339 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:59:52.187260 10339 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20250627 13:59:52.191978 10356 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:59:52.193007 10339 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250627 13:59:52.193293 10339 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-1/wal
uuid: "55235744bbb244bbb4cce13e237f951d"
format_stamp: "Formatted at 2025-06-27 13:59:36 on dist-test-slave-rn80"
I20250627 13:59:52.195183 10339 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:59:52.246887 10339 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:59:52.248276 10339 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:59:52.248893 10339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:59:52.251966 10339 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:59:52.258025 10363 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250627 13:59:52.265200 10339 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250627 13:59:52.265415 10339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.002s sys 0.000s
I20250627 13:59:52.265690 10339 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250627 13:59:52.270299 10339 ts_tablet_manager.cc:610] Registered 1 tablets
I20250627 13:59:52.270545 10339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.005s sys 0.000s
I20250627 13:59:52.270913 10363 tablet_bootstrap.cc:492] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d: Bootstrap starting.
I20250627 13:59:52.442392 10339 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.66:44059
I20250627 13:59:52.442576 10469 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.66:44059 every 8 connection(s)
I20250627 13:59:52.446022 10339 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-1/data/info.pb
I20250627 13:59:52.449751 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 10339
I20250627 13:59:52.451732 3765 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
/tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.173.67:40835
--local_ip_for_outbound_sockets=127.3.173.67
--tserver_master_addrs=127.3.173.126:44191
--webserver_port=37163
--webserver_interface=127.3.173.67
--builtin_ntp_servers=127.3.173.84:37861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250627 13:59:52.493906 10470 heartbeater.cc:344] Connected to a master server at 127.3.173.126:44191
I20250627 13:59:52.494503 10470 heartbeater.cc:461] Registering TS with master...
I20250627 13:59:52.495911 10470 heartbeater.cc:507] Master 127.3.173.126:44191 requested a full tablet report, sending...
I20250627 13:59:52.500469 10148 ts_manager.cc:194] Registered new tserver with Master: 55235744bbb244bbb4cce13e237f951d (127.3.173.66:44059)
I20250627 13:59:52.503439 10148 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.173.66:57889
I20250627 13:59:52.610903 10363 log.cc:826] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d: Log is configured to *not* fsync() on all Append() calls
I20250627 13:59:52.795235 10228 tablet_bootstrap.cc:492] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:59:52.796339 10228 tablet_bootstrap.cc:492] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd: Bootstrap complete.
I20250627 13:59:52.798065 10228 ts_tablet_manager.cc:1397] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd: Time spent bootstrapping tablet: real 2.418s user 2.332s sys 0.065s
W20250627 13:59:52.815498 10474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:59:52.816015 10474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:59:52.816542 10474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250627 13:59:52.814786 10228 raft_consensus.cc:357] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55235744bbb244bbb4cce13e237f951d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 44059 } } peers { permanent_uuid: "59d296ac81f349499f4b306a382bc0cd" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43065 } } peers { permanent_uuid: "e5c3905e0806414faebde321b160a377" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40835 } }
I20250627 13:59:52.817927 10228 raft_consensus.cc:738] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59d296ac81f349499f4b306a382bc0cd, State: Initialized, Role: FOLLOWER
I20250627 13:59:52.818852 10228 consensus_queue.cc:260] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55235744bbb244bbb4cce13e237f951d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 44059 } } peers { permanent_uuid: "59d296ac81f349499f4b306a382bc0cd" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43065 } } peers { permanent_uuid: "e5c3905e0806414faebde321b160a377" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40835 } }
I20250627 13:59:52.822331 10228 ts_tablet_manager.cc:1428] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd: Time spent starting tablet: real 0.024s user 0.018s sys 0.006s
W20250627 13:59:52.852080 10474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:59:52.852967 10474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.173.67
I20250627 13:59:52.887703 10474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.173.84:37861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.173.67:40835
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.173.67
--webserver_port=37163
--tserver_master_addrs=127.3.173.126:44191
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.173.67
--log_dir=/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6797
TSAN enabled
I20250627 13:59:52.888996 10474 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:59:52.890651 10474 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:59:52.908257 10483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:59:53.507246 10470 heartbeater.cc:499] Master 127.3.173.126:44191 was elected leader, sending a full tablet report...
I20250627 13:59:54.064595 10488 raft_consensus.cc:491] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:59:54.065168 10488 raft_consensus.cc:513] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55235744bbb244bbb4cce13e237f951d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 44059 } } peers { permanent_uuid: "59d296ac81f349499f4b306a382bc0cd" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43065 } } peers { permanent_uuid: "e5c3905e0806414faebde321b160a377" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40835 } }
I20250627 13:59:54.083272 10488 leader_election.cc:290] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 55235744bbb244bbb4cce13e237f951d (127.3.173.66:44059), e5c3905e0806414faebde321b160a377 (127.3.173.67:40835)
W20250627 13:59:54.084337 10224 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.173.67:40835: connect: Connection refused (error 111)
W20250627 13:59:54.096041 10224 leader_election.cc:336] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e5c3905e0806414faebde321b160a377 (127.3.173.67:40835): Network error: Client connection negotiation failed: client connection to 127.3.173.67:40835: connect: Connection refused (error 111)
I20250627 13:59:54.097728 10425 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8fc980318ce7411db461a2ef0d3dc4cb" candidate_uuid: "59d296ac81f349499f4b306a382bc0cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "55235744bbb244bbb4cce13e237f951d" is_pre_election: true
W20250627 13:59:54.110683 10222 leader_election.cc:343] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 55235744bbb244bbb4cce13e237f951d (127.3.173.66:44059): Illegal state: must be running to vote when last-logged opid is not known
I20250627 13:59:54.111157 10222 leader_election.cc:304] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59d296ac81f349499f4b306a382bc0cd; no voters: 55235744bbb244bbb4cce13e237f951d, e5c3905e0806414faebde321b160a377
I20250627 13:59:54.112026 10488 raft_consensus.cc:2747] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250627 13:59:52.911087 10482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:59:52.911353 10485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:59:54.118252 10484 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250627 13:59:54.118275 10474 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250627 13:59:54.122963 10474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:59:54.125756 10474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:59:54.127249 10474 hybrid_clock.cc:648] HybridClock initialized: now 1751032794127197 us; error 53 us; skew 500 ppm
I20250627 13:59:54.128340 10474 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:59:54.135998 10474 webserver.cc:469] Webserver started at http://127.3.173.67:37163/ using document root <none> and password file <none>
I20250627 13:59:54.137216 10474 fs_manager.cc:362] Metadata directory not provided
I20250627 13:59:54.137519 10474 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:59:54.147917 10474 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.003s
I20250627 13:59:54.153627 10497 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:59:54.154882 10474 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250627 13:59:54.155263 10474 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-2/wal
uuid: "e5c3905e0806414faebde321b160a377"
format_stamp: "Formatted at 2025-06-27 13:59:37 on dist-test-slave-rn80"
I20250627 13:59:54.158001 10474 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:59:54.235956 10474 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:59:54.238323 10474 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:59:54.238991 10474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:59:54.242837 10474 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:59:54.249747 10504 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250627 13:59:54.260788 10474 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250627 13:59:54.261060 10474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.001s sys 0.001s
I20250627 13:59:54.261337 10474 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250627 13:59:54.266012 10474 ts_tablet_manager.cc:610] Registered 1 tablets
I20250627 13:59:54.266263 10474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.005s sys 0.000s
I20250627 13:59:54.266724 10504 tablet_bootstrap.cc:492] T 8fc980318ce7411db461a2ef0d3dc4cb P e5c3905e0806414faebde321b160a377: Bootstrap starting.
I20250627 13:59:54.512617 10474 rpc_server.cc:307] RPC server started. Bound to: 127.3.173.67:40835
I20250627 13:59:54.512790 10610 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.173.67:40835 every 8 connection(s)
I20250627 13:59:54.515733 10474 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-2/data/info.pb
I20250627 13:59:54.516944 3765 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu as pid 10474
I20250627 13:59:54.585630 10611 heartbeater.cc:344] Connected to a master server at 127.3.173.126:44191
I20250627 13:59:54.586059 10611 heartbeater.cc:461] Registering TS with master...
I20250627 13:59:54.587251 10611 heartbeater.cc:507] Master 127.3.173.126:44191 requested a full tablet report, sending...
I20250627 13:59:54.591593 10148 ts_manager.cc:194] Registered new tserver with Master: e5c3905e0806414faebde321b160a377 (127.3.173.67:40835)
I20250627 13:59:54.593832 10148 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.173.67:42289
I20250627 13:59:54.606392 3765 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250627 13:59:54.625757 10504 log.cc:826] T 8fc980318ce7411db461a2ef0d3dc4cb P e5c3905e0806414faebde321b160a377: Log is configured to *not* fsync() on all Append() calls
I20250627 13:59:54.898756 10363 tablet_bootstrap.cc:492] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:59:54.899546 10363 tablet_bootstrap.cc:492] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d: Bootstrap complete.
I20250627 13:59:54.900964 10363 ts_tablet_manager.cc:1397] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d: Time spent bootstrapping tablet: real 2.630s user 2.535s sys 0.088s
I20250627 13:59:54.905997 10363 raft_consensus.cc:357] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55235744bbb244bbb4cce13e237f951d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 44059 } } peers { permanent_uuid: "59d296ac81f349499f4b306a382bc0cd" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43065 } } peers { permanent_uuid: "e5c3905e0806414faebde321b160a377" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40835 } }
I20250627 13:59:54.907939 10363 raft_consensus.cc:738] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55235744bbb244bbb4cce13e237f951d, State: Initialized, Role: FOLLOWER
I20250627 13:59:54.908699 10363 consensus_queue.cc:260] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55235744bbb244bbb4cce13e237f951d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 44059 } } peers { permanent_uuid: "59d296ac81f349499f4b306a382bc0cd" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43065 } } peers { permanent_uuid: "e5c3905e0806414faebde321b160a377" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40835 } }
I20250627 13:59:54.911976 10363 ts_tablet_manager.cc:1428] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20250627 13:59:55.598309 10611 heartbeater.cc:499] Master 127.3.173.126:44191 was elected leader, sending a full tablet report...
I20250627 13:59:55.818518 10625 raft_consensus.cc:491] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:59:55.818902 10625 raft_consensus.cc:513] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55235744bbb244bbb4cce13e237f951d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 44059 } } peers { permanent_uuid: "59d296ac81f349499f4b306a382bc0cd" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43065 } } peers { permanent_uuid: "e5c3905e0806414faebde321b160a377" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40835 } }
I20250627 13:59:55.820276 10625 leader_election.cc:290] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 55235744bbb244bbb4cce13e237f951d (127.3.173.66:44059), e5c3905e0806414faebde321b160a377 (127.3.173.67:40835)
I20250627 13:59:55.821267 10425 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8fc980318ce7411db461a2ef0d3dc4cb" candidate_uuid: "59d296ac81f349499f4b306a382bc0cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "55235744bbb244bbb4cce13e237f951d" is_pre_election: true
I20250627 13:59:55.822017 10425 raft_consensus.cc:2466] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59d296ac81f349499f4b306a382bc0cd in term 1.
I20250627 13:59:55.823366 10222 leader_election.cc:304] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55235744bbb244bbb4cce13e237f951d, 59d296ac81f349499f4b306a382bc0cd; no voters:
I20250627 13:59:55.824739 10625 raft_consensus.cc:2802] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250627 13:59:55.825212 10625 raft_consensus.cc:491] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:59:55.825637 10625 raft_consensus.cc:3058] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:59:55.833268 10625 raft_consensus.cc:513] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55235744bbb244bbb4cce13e237f951d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 44059 } } peers { permanent_uuid: "59d296ac81f349499f4b306a382bc0cd" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43065 } } peers { permanent_uuid: "e5c3905e0806414faebde321b160a377" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40835 } }
I20250627 13:59:55.835136 10625 leader_election.cc:290] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd [CANDIDATE]: Term 2 election: Requested vote from peers 55235744bbb244bbb4cce13e237f951d (127.3.173.66:44059), e5c3905e0806414faebde321b160a377 (127.3.173.67:40835)
I20250627 13:59:55.835932 10425 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8fc980318ce7411db461a2ef0d3dc4cb" candidate_uuid: "59d296ac81f349499f4b306a382bc0cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "55235744bbb244bbb4cce13e237f951d"
I20250627 13:59:55.836491 10425 raft_consensus.cc:3058] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:59:55.845239 10425 raft_consensus.cc:2466] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59d296ac81f349499f4b306a382bc0cd in term 2.
I20250627 13:59:55.837388 10566 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8fc980318ce7411db461a2ef0d3dc4cb" candidate_uuid: "59d296ac81f349499f4b306a382bc0cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "e5c3905e0806414faebde321b160a377" is_pre_election: true
I20250627 13:59:55.837574 10565 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8fc980318ce7411db461a2ef0d3dc4cb" candidate_uuid: "59d296ac81f349499f4b306a382bc0cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "e5c3905e0806414faebde321b160a377"
I20250627 13:59:55.846447 10222 leader_election.cc:304] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55235744bbb244bbb4cce13e237f951d, 59d296ac81f349499f4b306a382bc0cd; no voters:
I20250627 13:59:55.847218 10625 raft_consensus.cc:2802] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd [term 2 FOLLOWER]: Leader election won for term 2
W20250627 13:59:55.848021 10224 leader_election.cc:343] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e5c3905e0806414faebde321b160a377 (127.3.173.67:40835): Illegal state: must be running to vote when last-logged opid is not known
I20250627 13:59:55.849310 10625 raft_consensus.cc:695] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd [term 2 LEADER]: Becoming Leader. State: Replica: 59d296ac81f349499f4b306a382bc0cd, State: Running, Role: LEADER
I20250627 13:59:55.850250 10625 consensus_queue.cc:237] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55235744bbb244bbb4cce13e237f951d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 44059 } } peers { permanent_uuid: "59d296ac81f349499f4b306a382bc0cd" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43065 } } peers { permanent_uuid: "e5c3905e0806414faebde321b160a377" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40835 } }
I20250627 13:59:55.860945 10148 catalog_manager.cc:5582] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd reported cstate change: term changed from 0 to 2, leader changed from <none> to 59d296ac81f349499f4b306a382bc0cd (127.3.173.65), VOTER 55235744bbb244bbb4cce13e237f951d (127.3.173.66) added, VOTER 59d296ac81f349499f4b306a382bc0cd (127.3.173.65) added, VOTER e5c3905e0806414faebde321b160a377 (127.3.173.67) added. New cstate: current_term: 2 leader_uuid: "59d296ac81f349499f4b306a382bc0cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55235744bbb244bbb4cce13e237f951d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 44059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59d296ac81f349499f4b306a382bc0cd" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43065 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5c3905e0806414faebde321b160a377" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40835 } health_report { overall_health: UNKNOWN } } }
W20250627 13:59:56.326386 10224 consensus_peers.cc:487] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd -> Peer e5c3905e0806414faebde321b160a377 (127.3.173.67:40835): Couldn't send request to peer e5c3905e0806414faebde321b160a377. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:59:56.448112 10425 raft_consensus.cc:1273] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d [term 2 FOLLOWER]: Refusing update from remote peer 59d296ac81f349499f4b306a382bc0cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250627 13:59:56.449855 10625 consensus_queue.cc:1035] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "55235744bbb244bbb4cce13e237f951d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 44059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
W20250627 13:59:56.507509 3765 scanner-internal.cc:458] Time spent opening tablet: real 1.863s user 0.007s sys 0.001s
I20250627 13:59:56.589010 10290 consensus_queue.cc:237] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 2.206, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "55235744bbb244bbb4cce13e237f951d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 44059 } } peers { permanent_uuid: "59d296ac81f349499f4b306a382bc0cd" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43065 } }
I20250627 13:59:56.594197 10425 raft_consensus.cc:1273] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d [term 2 FOLLOWER]: Refusing update from remote peer 59d296ac81f349499f4b306a382bc0cd: Log matching property violated. Preceding OpId in replica: term: 2 index: 206. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
I20250627 13:59:56.596131 10640 consensus_queue.cc:1035] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "55235744bbb244bbb4cce13e237f951d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 44059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.001s
I20250627 13:59:56.607757 10640 raft_consensus.cc:2953] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd [term 2 LEADER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER e5c3905e0806414faebde321b160a377 (127.3.173.67) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "55235744bbb244bbb4cce13e237f951d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 44059 } } peers { permanent_uuid: "59d296ac81f349499f4b306a382bc0cd" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43065 } } }
I20250627 13:59:56.610755 10425 raft_consensus.cc:2953] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d [term 2 FOLLOWER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER e5c3905e0806414faebde321b160a377 (127.3.173.67) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "55235744bbb244bbb4cce13e237f951d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 44059 } } peers { permanent_uuid: "59d296ac81f349499f4b306a382bc0cd" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43065 } } }
I20250627 13:59:56.632488 10133 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 8fc980318ce7411db461a2ef0d3dc4cb with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:59:56.637230 10148 catalog_manager.cc:5582] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd reported cstate change: config changed from index -1 to 207, VOTER e5c3905e0806414faebde321b160a377 (127.3.173.67) evicted. New cstate: current_term: 2 leader_uuid: "59d296ac81f349499f4b306a382bc0cd" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "55235744bbb244bbb4cce13e237f951d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 44059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59d296ac81f349499f4b306a382bc0cd" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43065 } health_report { overall_health: HEALTHY } } }
I20250627 13:59:56.679255 10504 tablet_bootstrap.cc:492] T 8fc980318ce7411db461a2ef0d3dc4cb P e5c3905e0806414faebde321b160a377: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:59:56.680372 10504 tablet_bootstrap.cc:492] T 8fc980318ce7411db461a2ef0d3dc4cb P e5c3905e0806414faebde321b160a377: Bootstrap complete.
I20250627 13:59:56.682229 10504 ts_tablet_manager.cc:1397] T 8fc980318ce7411db461a2ef0d3dc4cb P e5c3905e0806414faebde321b160a377: Time spent bootstrapping tablet: real 2.416s user 2.302s sys 0.064s
I20250627 13:59:56.689805 10290 consensus_queue.cc:237] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 2.207, Last appended by leader: 205, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "59d296ac81f349499f4b306a382bc0cd" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43065 } }
I20250627 13:59:56.692178 10640 raft_consensus.cc:2953] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd [term 2 LEADER]: Committing config change with OpId 2.208: config changed from index 207 to 208, VOTER 55235744bbb244bbb4cce13e237f951d (127.3.173.66) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "59d296ac81f349499f4b306a382bc0cd" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43065 } } }
I20250627 13:59:56.709167 10546 tablet_service.cc:1515] Processing DeleteTablet for tablet 8fc980318ce7411db461a2ef0d3dc4cb with delete_type TABLET_DATA_TOMBSTONED (TS e5c3905e0806414faebde321b160a377 not found in new config with opid_index 207) from {username='slave'} at 127.0.0.1:60476
I20250627 13:59:56.709882 10133 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 8fc980318ce7411db461a2ef0d3dc4cb with cas_config_opid_index 207: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:59:56.715909 10147 catalog_manager.cc:5582] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd reported cstate change: config changed from index 207 to 208, VOTER 55235744bbb244bbb4cce13e237f951d (127.3.173.66) evicted. New cstate: current_term: 2 leader_uuid: "59d296ac81f349499f4b306a382bc0cd" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "59d296ac81f349499f4b306a382bc0cd" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43065 } health_report { overall_health: HEALTHY } } }
W20250627 13:59:56.726022 10134 catalog_manager.cc:4908] TS e5c3905e0806414faebde321b160a377 (127.3.173.67:40835): delete failed for tablet 8fc980318ce7411db461a2ef0d3dc4cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8fc980318ce7411db461a2ef0d3dc4cb already in progress: opening tablet
I20250627 13:59:56.713701 10504 raft_consensus.cc:357] T 8fc980318ce7411db461a2ef0d3dc4cb P e5c3905e0806414faebde321b160a377 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55235744bbb244bbb4cce13e237f951d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 44059 } } peers { permanent_uuid: "59d296ac81f349499f4b306a382bc0cd" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43065 } } peers { permanent_uuid: "e5c3905e0806414faebde321b160a377" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40835 } }
I20250627 13:59:56.729090 10504 raft_consensus.cc:738] T 8fc980318ce7411db461a2ef0d3dc4cb P e5c3905e0806414faebde321b160a377 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c3905e0806414faebde321b160a377, State: Initialized, Role: FOLLOWER
I20250627 13:59:56.729987 10504 consensus_queue.cc:260] T 8fc980318ce7411db461a2ef0d3dc4cb P e5c3905e0806414faebde321b160a377 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55235744bbb244bbb4cce13e237f951d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 44059 } } peers { permanent_uuid: "59d296ac81f349499f4b306a382bc0cd" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43065 } } peers { permanent_uuid: "e5c3905e0806414faebde321b160a377" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40835 } }
I20250627 13:59:56.743614 10504 ts_tablet_manager.cc:1428] T 8fc980318ce7411db461a2ef0d3dc4cb P e5c3905e0806414faebde321b160a377: Time spent starting tablet: real 0.061s user 0.015s sys 0.000s
I20250627 13:59:56.750595 10546 tablet_service.cc:1515] Processing DeleteTablet for tablet 8fc980318ce7411db461a2ef0d3dc4cb with delete_type TABLET_DATA_TOMBSTONED (Replica has no consensus available (current committed config index is 208)) from {username='slave'} at 127.0.0.1:60476
I20250627 13:59:56.753109 10648 tablet_replica.cc:331] stopping tablet replica
I20250627 13:59:56.753845 10648 raft_consensus.cc:2241] T 8fc980318ce7411db461a2ef0d3dc4cb P e5c3905e0806414faebde321b160a377 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:59:56.754403 10648 raft_consensus.cc:2270] T 8fc980318ce7411db461a2ef0d3dc4cb P e5c3905e0806414faebde321b160a377 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:59:56.768607 10405 tablet_service.cc:1515] Processing DeleteTablet for tablet 8fc980318ce7411db461a2ef0d3dc4cb with delete_type TABLET_DATA_TOMBSTONED (TS 55235744bbb244bbb4cce13e237f951d not found in new config with opid_index 208) from {username='slave'} at 127.0.0.1:36042
I20250627 13:59:56.780207 10651 tablet_replica.cc:331] stopping tablet replica
I20250627 13:59:56.781100 10651 raft_consensus.cc:2241] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d [term 2 FOLLOWER]: Raft consensus shutting down.
I20250627 13:59:56.781800 10651 raft_consensus.cc:2270] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d [term 2 FOLLOWER]: Raft consensus is shut down!
I20250627 13:59:56.802958 10648 ts_tablet_manager.cc:1905] T 8fc980318ce7411db461a2ef0d3dc4cb P e5c3905e0806414faebde321b160a377: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:59:56.813850 10651 ts_tablet_manager.cc:1905] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:59:56.822775 10648 ts_tablet_manager.cc:1918] T 8fc980318ce7411db461a2ef0d3dc4cb P e5c3905e0806414faebde321b160a377: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.205
I20250627 13:59:56.823320 10648 log.cc:1199] T 8fc980318ce7411db461a2ef0d3dc4cb P e5c3905e0806414faebde321b160a377: Deleting WAL directory at /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-2/wal/wals/8fc980318ce7411db461a2ef0d3dc4cb
I20250627 13:59:56.825120 10134 catalog_manager.cc:4928] TS e5c3905e0806414faebde321b160a377 (127.3.173.67:40835): tablet 8fc980318ce7411db461a2ef0d3dc4cb (table pre_rebuild [id=64bdffa0d01c4ba1ae3cd8b5812b187f]) successfully deleted
I20250627 13:59:56.828940 10651 ts_tablet_manager.cc:1918] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.207
I20250627 13:59:56.829370 10651 log.cc:1199] T 8fc980318ce7411db461a2ef0d3dc4cb P 55235744bbb244bbb4cce13e237f951d: Deleting WAL directory at /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-1/wal/wals/8fc980318ce7411db461a2ef0d3dc4cb
I20250627 13:59:56.831113 10132 catalog_manager.cc:4928] TS 55235744bbb244bbb4cce13e237f951d (127.3.173.66:44059): tablet 8fc980318ce7411db461a2ef0d3dc4cb (table pre_rebuild [id=64bdffa0d01c4ba1ae3cd8b5812b187f]) successfully deleted
I20250627 13:59:57.206983 10405 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:59:57.220085 10546 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:59:57.259752 10270 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef9ad88cfad848aca4aeaaa2d343faf9 | 127.3.173.126:44191 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.3.173.84:37861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
55235744bbb244bbb4cce13e237f951d | 127.3.173.66:44059 | HEALTHY | <none> | 0 | 0
59d296ac81f349499f4b306a382bc0cd | 127.3.173.65:43065 | HEALTHY | <none> | 1 | 0
e5c3905e0806414faebde321b160a377 | 127.3.173.67:40835 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.3.173.65 | experimental | 127.3.173.65:43065
local_ip_for_outbound_sockets | 127.3.173.66 | experimental | 127.3.173.66:44059
local_ip_for_outbound_sockets | 127.3.173.67 | experimental | 127.3.173.67:40835
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-0/data/info.pb | hidden | 127.3.173.65:43065
server_dump_info_path | /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.173.66:44059
server_dump_info_path | /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.173.67:40835
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.3.173.84:37861 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------+----+---------+---------------+---------+------------+------------------+-------------
pre_rebuild | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 0
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 1
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250627 13:59:57.545478 3765 log_verifier.cc:126] Checking tablet 8fc980318ce7411db461a2ef0d3dc4cb
I20250627 13:59:57.833196 3765 log_verifier.cc:177] Verified matching terms for 208 ops in tablet 8fc980318ce7411db461a2ef0d3dc4cb
I20250627 13:59:57.835568 10147 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55982:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250627 13:59:57.836230 10147 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55982:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250627 13:59:57.849393 10147 catalog_manager.cc:5869] T 00000000000000000000000000000000 P ef9ad88cfad848aca4aeaaa2d343faf9: Sending DeleteTablet for 1 replicas of tablet 8fc980318ce7411db461a2ef0d3dc4cb
I20250627 13:59:57.851367 3765 test_util.cc:276] Using random seed: 401075920
I20250627 13:59:57.851274 10270 tablet_service.cc:1515] Processing DeleteTablet for tablet 8fc980318ce7411db461a2ef0d3dc4cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-27 13:59:57 UTC) from {username='slave'} at 127.0.0.1:58444
I20250627 13:59:57.853101 10681 tablet_replica.cc:331] stopping tablet replica
I20250627 13:59:57.854359 10681 raft_consensus.cc:2241] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd [term 2 LEADER]: Raft consensus shutting down.
I20250627 13:59:57.854947 10681 raft_consensus.cc:2270] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd [term 2 FOLLOWER]: Raft consensus is shut down!
I20250627 13:59:57.889565 10147 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56022:
name: "post_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250627 13:59:57.891376 10681 ts_tablet_manager.cc:1905] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250627 13:59:57.892199 10147 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table post_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250627 13:59:57.907092 10681 ts_tablet_manager.cc:1918] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.208
I20250627 13:59:57.907613 10681 log.cc:1199] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd: Deleting WAL directory at /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-0/wal/wals/8fc980318ce7411db461a2ef0d3dc4cb
I20250627 13:59:57.908833 10681 ts_tablet_manager.cc:1939] T 8fc980318ce7411db461a2ef0d3dc4cb P 59d296ac81f349499f4b306a382bc0cd: Deleting consensus metadata
I20250627 13:59:57.912101 10133 catalog_manager.cc:4928] TS 59d296ac81f349499f4b306a382bc0cd (127.3.173.65:43065): tablet 8fc980318ce7411db461a2ef0d3dc4cb (table pre_rebuild [id=64bdffa0d01c4ba1ae3cd8b5812b187f]) successfully deleted
I20250627 13:59:57.915520 10405 tablet_service.cc:1468] Processing CreateTablet for tablet 6bd78d5932d7466b895396e7cfa4e30a (DEFAULT_TABLE table=post_rebuild [id=e9326a5c24324f0889e83ba88caab5ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250627 13:59:57.916131 10546 tablet_service.cc:1468] Processing CreateTablet for tablet 6bd78d5932d7466b895396e7cfa4e30a (DEFAULT_TABLE table=post_rebuild [id=e9326a5c24324f0889e83ba88caab5ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250627 13:59:57.916790 10405 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bd78d5932d7466b895396e7cfa4e30a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:59:57.917726 10546 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bd78d5932d7466b895396e7cfa4e30a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:59:57.924257 10270 tablet_service.cc:1468] Processing CreateTablet for tablet 6bd78d5932d7466b895396e7cfa4e30a (DEFAULT_TABLE table=post_rebuild [id=e9326a5c24324f0889e83ba88caab5ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250627 13:59:57.925666 10270 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bd78d5932d7466b895396e7cfa4e30a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:59:57.942605 10688 tablet_bootstrap.cc:492] T 6bd78d5932d7466b895396e7cfa4e30a P e5c3905e0806414faebde321b160a377: Bootstrap starting.
I20250627 13:59:57.948468 10688 tablet_bootstrap.cc:654] T 6bd78d5932d7466b895396e7cfa4e30a P e5c3905e0806414faebde321b160a377: Neither blocks nor log segments found. Creating new log.
I20250627 13:59:57.948426 10690 tablet_bootstrap.cc:492] T 6bd78d5932d7466b895396e7cfa4e30a P 55235744bbb244bbb4cce13e237f951d: Bootstrap starting.
I20250627 13:59:57.949236 10689 tablet_bootstrap.cc:492] T 6bd78d5932d7466b895396e7cfa4e30a P 59d296ac81f349499f4b306a382bc0cd: Bootstrap starting.
I20250627 13:59:57.954684 10690 tablet_bootstrap.cc:654] T 6bd78d5932d7466b895396e7cfa4e30a P 55235744bbb244bbb4cce13e237f951d: Neither blocks nor log segments found. Creating new log.
I20250627 13:59:57.960610 10688 tablet_bootstrap.cc:492] T 6bd78d5932d7466b895396e7cfa4e30a P e5c3905e0806414faebde321b160a377: No bootstrap required, opened a new log
I20250627 13:59:57.961123 10688 ts_tablet_manager.cc:1397] T 6bd78d5932d7466b895396e7cfa4e30a P e5c3905e0806414faebde321b160a377: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.010s
I20250627 13:59:57.963635 10689 tablet_bootstrap.cc:654] T 6bd78d5932d7466b895396e7cfa4e30a P 59d296ac81f349499f4b306a382bc0cd: Neither blocks nor log segments found. Creating new log.
I20250627 13:59:57.963726 10688 raft_consensus.cc:357] T 6bd78d5932d7466b895396e7cfa4e30a P e5c3905e0806414faebde321b160a377 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55235744bbb244bbb4cce13e237f951d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 44059 } } peers { permanent_uuid: "e5c3905e0806414faebde321b160a377" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40835 } } peers { permanent_uuid: "59d296ac81f349499f4b306a382bc0cd" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43065 } }
I20250627 13:59:57.964608 10688 raft_consensus.cc:383] T 6bd78d5932d7466b895396e7cfa4e30a P e5c3905e0806414faebde321b160a377 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:59:57.964934 10688 raft_consensus.cc:738] T 6bd78d5932d7466b895396e7cfa4e30a P e5c3905e0806414faebde321b160a377 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c3905e0806414faebde321b160a377, State: Initialized, Role: FOLLOWER
I20250627 13:59:57.965799 10690 tablet_bootstrap.cc:492] T 6bd78d5932d7466b895396e7cfa4e30a P 55235744bbb244bbb4cce13e237f951d: No bootstrap required, opened a new log
I20250627 13:59:57.965862 10688 consensus_queue.cc:260] T 6bd78d5932d7466b895396e7cfa4e30a P e5c3905e0806414faebde321b160a377 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55235744bbb244bbb4cce13e237f951d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 44059 } } peers { permanent_uuid: "e5c3905e0806414faebde321b160a377" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40835 } } peers { permanent_uuid: "59d296ac81f349499f4b306a382bc0cd" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43065 } }
I20250627 13:59:57.966323 10690 ts_tablet_manager.cc:1397] T 6bd78d5932d7466b895396e7cfa4e30a P 55235744bbb244bbb4cce13e237f951d: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.004s
I20250627 13:59:57.969722 10690 raft_consensus.cc:357] T 6bd78d5932d7466b895396e7cfa4e30a P 55235744bbb244bbb4cce13e237f951d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55235744bbb244bbb4cce13e237f951d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 44059 } } peers { permanent_uuid: "e5c3905e0806414faebde321b160a377" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40835 } } peers { permanent_uuid: "59d296ac81f349499f4b306a382bc0cd" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43065 } }
I20250627 13:59:57.970618 10690 raft_consensus.cc:383] T 6bd78d5932d7466b895396e7cfa4e30a P 55235744bbb244bbb4cce13e237f951d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:59:57.970947 10690 raft_consensus.cc:738] T 6bd78d5932d7466b895396e7cfa4e30a P 55235744bbb244bbb4cce13e237f951d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55235744bbb244bbb4cce13e237f951d, State: Initialized, Role: FOLLOWER
I20250627 13:59:57.971715 10690 consensus_queue.cc:260] T 6bd78d5932d7466b895396e7cfa4e30a P 55235744bbb244bbb4cce13e237f951d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55235744bbb244bbb4cce13e237f951d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 44059 } } peers { permanent_uuid: "e5c3905e0806414faebde321b160a377" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40835 } } peers { permanent_uuid: "59d296ac81f349499f4b306a382bc0cd" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43065 } }
I20250627 13:59:57.974920 10688 ts_tablet_manager.cc:1428] T 6bd78d5932d7466b895396e7cfa4e30a P e5c3905e0806414faebde321b160a377: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20250627 13:59:57.977427 10689 tablet_bootstrap.cc:492] T 6bd78d5932d7466b895396e7cfa4e30a P 59d296ac81f349499f4b306a382bc0cd: No bootstrap required, opened a new log
I20250627 13:59:57.977919 10689 ts_tablet_manager.cc:1397] T 6bd78d5932d7466b895396e7cfa4e30a P 59d296ac81f349499f4b306a382bc0cd: Time spent bootstrapping tablet: real 0.029s user 0.010s sys 0.007s
I20250627 13:59:57.978324 10690 ts_tablet_manager.cc:1428] T 6bd78d5932d7466b895396e7cfa4e30a P 55235744bbb244bbb4cce13e237f951d: Time spent starting tablet: real 0.012s user 0.007s sys 0.003s
I20250627 13:59:57.981101 10689 raft_consensus.cc:357] T 6bd78d5932d7466b895396e7cfa4e30a P 59d296ac81f349499f4b306a382bc0cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55235744bbb244bbb4cce13e237f951d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 44059 } } peers { permanent_uuid: "e5c3905e0806414faebde321b160a377" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40835 } } peers { permanent_uuid: "59d296ac81f349499f4b306a382bc0cd" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43065 } }
I20250627 13:59:57.981884 10689 raft_consensus.cc:383] T 6bd78d5932d7466b895396e7cfa4e30a P 59d296ac81f349499f4b306a382bc0cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:59:57.982246 10689 raft_consensus.cc:738] T 6bd78d5932d7466b895396e7cfa4e30a P 59d296ac81f349499f4b306a382bc0cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59d296ac81f349499f4b306a382bc0cd, State: Initialized, Role: FOLLOWER
I20250627 13:59:57.983035 10689 consensus_queue.cc:260] T 6bd78d5932d7466b895396e7cfa4e30a P 59d296ac81f349499f4b306a382bc0cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55235744bbb244bbb4cce13e237f951d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 44059 } } peers { permanent_uuid: "e5c3905e0806414faebde321b160a377" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40835 } } peers { permanent_uuid: "59d296ac81f349499f4b306a382bc0cd" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43065 } }
I20250627 13:59:57.988976 10689 ts_tablet_manager.cc:1428] T 6bd78d5932d7466b895396e7cfa4e30a P 59d296ac81f349499f4b306a382bc0cd: Time spent starting tablet: real 0.011s user 0.005s sys 0.003s
I20250627 13:59:58.014789 10695 raft_consensus.cc:491] T 6bd78d5932d7466b895396e7cfa4e30a P 55235744bbb244bbb4cce13e237f951d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:59:58.015301 10695 raft_consensus.cc:513] T 6bd78d5932d7466b895396e7cfa4e30a P 55235744bbb244bbb4cce13e237f951d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55235744bbb244bbb4cce13e237f951d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 44059 } } peers { permanent_uuid: "e5c3905e0806414faebde321b160a377" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40835 } } peers { permanent_uuid: "59d296ac81f349499f4b306a382bc0cd" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43065 } }
I20250627 13:59:58.019927 10695 leader_election.cc:290] T 6bd78d5932d7466b895396e7cfa4e30a P 55235744bbb244bbb4cce13e237f951d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5c3905e0806414faebde321b160a377 (127.3.173.67:40835), 59d296ac81f349499f4b306a382bc0cd (127.3.173.65:43065)
I20250627 13:59:58.040880 10290 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6bd78d5932d7466b895396e7cfa4e30a" candidate_uuid: "55235744bbb244bbb4cce13e237f951d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59d296ac81f349499f4b306a382bc0cd" is_pre_election: true
I20250627 13:59:58.041666 10290 raft_consensus.cc:2466] T 6bd78d5932d7466b895396e7cfa4e30a P 59d296ac81f349499f4b306a382bc0cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55235744bbb244bbb4cce13e237f951d in term 0.
I20250627 13:59:58.041832 10565 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6bd78d5932d7466b895396e7cfa4e30a" candidate_uuid: "55235744bbb244bbb4cce13e237f951d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c3905e0806414faebde321b160a377" is_pre_election: true
I20250627 13:59:58.042520 10565 raft_consensus.cc:2466] T 6bd78d5932d7466b895396e7cfa4e30a P e5c3905e0806414faebde321b160a377 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55235744bbb244bbb4cce13e237f951d in term 0.
I20250627 13:59:58.042935 10358 leader_election.cc:304] T 6bd78d5932d7466b895396e7cfa4e30a P 55235744bbb244bbb4cce13e237f951d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55235744bbb244bbb4cce13e237f951d, 59d296ac81f349499f4b306a382bc0cd; no voters:
I20250627 13:59:58.043815 10695 raft_consensus.cc:2802] T 6bd78d5932d7466b895396e7cfa4e30a P 55235744bbb244bbb4cce13e237f951d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:59:58.044142 10695 raft_consensus.cc:491] T 6bd78d5932d7466b895396e7cfa4e30a P 55235744bbb244bbb4cce13e237f951d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:59:58.044437 10695 raft_consensus.cc:3058] T 6bd78d5932d7466b895396e7cfa4e30a P 55235744bbb244bbb4cce13e237f951d [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:59:58.051029 10695 raft_consensus.cc:513] T 6bd78d5932d7466b895396e7cfa4e30a P 55235744bbb244bbb4cce13e237f951d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55235744bbb244bbb4cce13e237f951d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 44059 } } peers { permanent_uuid: "e5c3905e0806414faebde321b160a377" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40835 } } peers { permanent_uuid: "59d296ac81f349499f4b306a382bc0cd" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43065 } }
I20250627 13:59:58.052873 10695 leader_election.cc:290] T 6bd78d5932d7466b895396e7cfa4e30a P 55235744bbb244bbb4cce13e237f951d [CANDIDATE]: Term 1 election: Requested vote from peers e5c3905e0806414faebde321b160a377 (127.3.173.67:40835), 59d296ac81f349499f4b306a382bc0cd (127.3.173.65:43065)
I20250627 13:59:58.053750 10565 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6bd78d5932d7466b895396e7cfa4e30a" candidate_uuid: "55235744bbb244bbb4cce13e237f951d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c3905e0806414faebde321b160a377"
I20250627 13:59:58.053980 10290 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6bd78d5932d7466b895396e7cfa4e30a" candidate_uuid: "55235744bbb244bbb4cce13e237f951d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59d296ac81f349499f4b306a382bc0cd"
I20250627 13:59:58.054303 10565 raft_consensus.cc:3058] T 6bd78d5932d7466b895396e7cfa4e30a P e5c3905e0806414faebde321b160a377 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:59:58.054404 10290 raft_consensus.cc:3058] T 6bd78d5932d7466b895396e7cfa4e30a P 59d296ac81f349499f4b306a382bc0cd [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:59:58.058759 10290 raft_consensus.cc:2466] T 6bd78d5932d7466b895396e7cfa4e30a P 59d296ac81f349499f4b306a382bc0cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55235744bbb244bbb4cce13e237f951d in term 1.
I20250627 13:59:58.059641 10358 leader_election.cc:304] T 6bd78d5932d7466b895396e7cfa4e30a P 55235744bbb244bbb4cce13e237f951d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55235744bbb244bbb4cce13e237f951d, 59d296ac81f349499f4b306a382bc0cd; no voters:
I20250627 13:59:58.060227 10695 raft_consensus.cc:2802] T 6bd78d5932d7466b895396e7cfa4e30a P 55235744bbb244bbb4cce13e237f951d [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:59:58.060925 10565 raft_consensus.cc:2466] T 6bd78d5932d7466b895396e7cfa4e30a P e5c3905e0806414faebde321b160a377 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55235744bbb244bbb4cce13e237f951d in term 1.
I20250627 13:59:58.064330 10695 raft_consensus.cc:695] T 6bd78d5932d7466b895396e7cfa4e30a P 55235744bbb244bbb4cce13e237f951d [term 1 LEADER]: Becoming Leader. State: Replica: 55235744bbb244bbb4cce13e237f951d, State: Running, Role: LEADER
I20250627 13:59:58.065135 10695 consensus_queue.cc:237] T 6bd78d5932d7466b895396e7cfa4e30a P 55235744bbb244bbb4cce13e237f951d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55235744bbb244bbb4cce13e237f951d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 44059 } } peers { permanent_uuid: "e5c3905e0806414faebde321b160a377" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40835 } } peers { permanent_uuid: "59d296ac81f349499f4b306a382bc0cd" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43065 } }
W20250627 13:59:58.068293 10614 tablet.cc:2378] T 6bd78d5932d7466b895396e7cfa4e30a P e5c3905e0806414faebde321b160a377: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250627 13:59:58.073426 10147 catalog_manager.cc:5582] T 6bd78d5932d7466b895396e7cfa4e30a P 55235744bbb244bbb4cce13e237f951d reported cstate change: term changed from 0 to 1, leader changed from <none> to 55235744bbb244bbb4cce13e237f951d (127.3.173.66). New cstate: current_term: 1 leader_uuid: "55235744bbb244bbb4cce13e237f951d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55235744bbb244bbb4cce13e237f951d" member_type: VOTER last_known_addr { host: "127.3.173.66" port: 44059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5c3905e0806414faebde321b160a377" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59d296ac81f349499f4b306a382bc0cd" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43065 } health_report { overall_health: UNKNOWN } } }
W20250627 13:59:58.104827 10336 tablet.cc:2378] T 6bd78d5932d7466b895396e7cfa4e30a P 59d296ac81f349499f4b306a382bc0cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250627 13:59:58.215653 10471 tablet.cc:2378] T 6bd78d5932d7466b895396e7cfa4e30a P 55235744bbb244bbb4cce13e237f951d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250627 13:59:58.230513 10565 raft_consensus.cc:1273] T 6bd78d5932d7466b895396e7cfa4e30a P e5c3905e0806414faebde321b160a377 [term 1 FOLLOWER]: Refusing update from remote peer 55235744bbb244bbb4cce13e237f951d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250627 13:59:58.230533 10290 raft_consensus.cc:1273] T 6bd78d5932d7466b895396e7cfa4e30a P 59d296ac81f349499f4b306a382bc0cd [term 1 FOLLOWER]: Refusing update from remote peer 55235744bbb244bbb4cce13e237f951d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250627 13:59:58.232240 10695 consensus_queue.cc:1035] T 6bd78d5932d7466b895396e7cfa4e30a P 55235744bbb244bbb4cce13e237f951d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c3905e0806414faebde321b160a377" member_type: VOTER last_known_addr { host: "127.3.173.67" port: 40835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:59:58.233624 10695 consensus_queue.cc:1035] T 6bd78d5932d7466b895396e7cfa4e30a P 55235744bbb244bbb4cce13e237f951d [LEADER]: Connected to new peer: Peer: permanent_uuid: "59d296ac81f349499f4b306a382bc0cd" member_type: VOTER last_known_addr { host: "127.3.173.65" port: 43065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:59:58.269168 10709 mvcc.cc:204] Tried to move back new op lower bound from 7172230341536145408 to 7172230340882608128. Current Snapshot: MvccSnapshot[applied={T|T < 7172230341536145408}]
I20250627 13:59:58.277020 10711 mvcc.cc:204] Tried to move back new op lower bound from 7172230341536145408 to 7172230340882608128. Current Snapshot: MvccSnapshot[applied={T|T < 7172230341536145408}]
I20250627 13:59:58.277925 10710 mvcc.cc:204] Tried to move back new op lower bound from 7172230341536145408 to 7172230340882608128. Current Snapshot: MvccSnapshot[applied={T|T < 7172230341536145408}]
W20250627 14:00:00.175854 10179 debug-util.cc:398] Leaking SignalData structure 0x7b08000a8100 after lost signal to thread 10116
W20250627 14:00:00.176541 10179 debug-util.cc:398] Leaking SignalData structure 0x7b080009e1e0 after lost signal to thread 10182
I20250627 14:00:03.931504 10270 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 14:00:03.931535 10405 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 14:00:03.944130 10546 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef9ad88cfad848aca4aeaaa2d343faf9 | 127.3.173.126:44191 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.3.173.84:37861 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
55235744bbb244bbb4cce13e237f951d | 127.3.173.66:44059 | HEALTHY | <none> | 1 | 0
59d296ac81f349499f4b306a382bc0cd | 127.3.173.65:43065 | HEALTHY | <none> | 0 | 0
e5c3905e0806414faebde321b160a377 | 127.3.173.67:40835 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.3.173.65 | experimental | 127.3.173.65:43065
local_ip_for_outbound_sockets | 127.3.173.66 | experimental | 127.3.173.66:44059
local_ip_for_outbound_sockets | 127.3.173.67 | experimental | 127.3.173.67:40835
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-0/data/info.pb | hidden | 127.3.173.65:43065
server_dump_info_path | /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.173.66:44059
server_dump_info_path | /tmp/dist-test-taskc8WqWj/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751032624583798-3765-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.173.67:40835
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.3.173.84:37861 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
--------------+----+---------+---------------+---------+------------+------------------+-------------
post_rebuild | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250627 14:00:04.156386 3765 log_verifier.cc:126] Checking tablet 6bd78d5932d7466b895396e7cfa4e30a
I20250627 14:00:04.937439 3765 log_verifier.cc:177] Verified matching terms for 205 ops in tablet 6bd78d5932d7466b895396e7cfa4e30a
I20250627 14:00:04.938441 3765 log_verifier.cc:126] Checking tablet 8fc980318ce7411db461a2ef0d3dc4cb
I20250627 14:00:04.938732 3765 log_verifier.cc:177] Verified matching terms for 0 ops in tablet 8fc980318ce7411db461a2ef0d3dc4cb
I20250627 14:00:04.950137 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 10185
I20250627 14:00:04.992725 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 10339
I20250627 14:00:05.040673 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 10474
I20250627 14:00:05.079325 3765 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskc8WqWj/build/tsan/bin/kudu with pid 10115
2025-06-27T14:00:05Z chronyd exiting
[ OK ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0 (34267 ms)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest (34267 ms total)
[----------] Global test environment tear-down
[==========] 9 tests from 5 test suites ran. (180489 ms total)
[ PASSED ] 8 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AdminCliTest.TestRebuildTables
1 FAILED TEST
I20250627 14:00:05.148116 3765 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/meta_cache.cc:302: suppressed but not reported on 4 messages since previous log ~48 seconds ago