Note: This is test shard 6 of 8.
[==========] Running 9 tests from 5 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from AdminCliTest
[ RUN ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250622 13:57:41.796110 15461 test_util.cc:276] Using random seed: 2056717469
W20250622 13:57:42.945071 15461 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.110s user 0.383s sys 0.724s
W20250622 13:57:42.945608 15461 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.110s user 0.383s sys 0.724s
I20250622 13:57:42.947902 15461 ts_itest-base.cc:115] Starting cluster with:
I20250622 13:57:42.948143 15461 ts_itest-base.cc:116] --------------
I20250622 13:57:42.948341 15461 ts_itest-base.cc:117] 4 tablet servers
I20250622 13:57:42.948556 15461 ts_itest-base.cc:118] 3 replicas per TS
I20250622 13:57:42.948758 15461 ts_itest-base.cc:119] --------------
2025-06-22T13:57:42Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-22T13:57:42Z Disabled control of system clock
I20250622 13:57:42.986879 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.25.126:43047
--webserver_interface=127.15.25.126
--webserver_port=0
--builtin_ntp_servers=127.15.25.84:34015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.25.126:43047 with env {}
W20250622 13:57:43.267042 15475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:57:43.267599 15475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 13:57:43.268108 15475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:57:43.297119 15475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250622 13:57:43.297389 15475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 13:57:43.297662 15475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250622 13:57:43.297911 15475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250622 13:57:43.331463 15475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:34015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.25.126:43047
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.25.126:43047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.25.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 13:57:43.332759 15475 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 13:57:43.334383 15475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:57:43.348093 15482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:57:43.348111 15481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:57:43.350005 15484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:57:43.350545 15475 server_base.cc:1048] running on GCE node
I20250622 13:57:44.440950 15475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 13:57:44.443500 15475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 13:57:44.444804 15475 hybrid_clock.cc:648] HybridClock initialized: now 1750600664444777 us; error 54 us; skew 500 ppm
I20250622 13:57:44.445508 15475 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:57:44.455943 15475 webserver.cc:469] Webserver started at http://127.15.25.126:44089/ using document root <none> and password file <none>
I20250622 13:57:44.456882 15475 fs_manager.cc:362] Metadata directory not provided
I20250622 13:57:44.457079 15475 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:57:44.457541 15475 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:57:44.461889 15475 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "03985fa59a0545208b4dbdc9eb16232d"
format_stamp: "Formatted at 2025-06-22 13:57:44 on dist-test-slave-6tph"
I20250622 13:57:44.462932 15475 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "03985fa59a0545208b4dbdc9eb16232d"
format_stamp: "Formatted at 2025-06-22 13:57:44 on dist-test-slave-6tph"
I20250622 13:57:44.469800 15475 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250622 13:57:44.474999 15491 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:57:44.475924 15475 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250622 13:57:44.476220 15475 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
uuid: "03985fa59a0545208b4dbdc9eb16232d"
format_stamp: "Formatted at 2025-06-22 13:57:44 on dist-test-slave-6tph"
I20250622 13:57:44.476544 15475 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:57:44.522449 15475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:57:44.523826 15475 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 13:57:44.524266 15475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:57:44.592550 15475 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.126:43047
I20250622 13:57:44.592633 15542 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.126:43047 every 8 connection(s)
I20250622 13:57:44.595067 15475 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250622 13:57:44.599647 15543 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:57:44.600689 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 15475
I20250622 13:57:44.601061 15461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250622 13:57:44.617617 15543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d: Bootstrap starting.
I20250622 13:57:44.622614 15543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d: Neither blocks nor log segments found. Creating new log.
I20250622 13:57:44.624259 15543 log.cc:826] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d: Log is configured to *not* fsync() on all Append() calls
I20250622 13:57:44.628535 15543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d: No bootstrap required, opened a new log
I20250622 13:57:44.646248 15543 raft_consensus.cc:357] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03985fa59a0545208b4dbdc9eb16232d" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 43047 } }
I20250622 13:57:44.646996 15543 raft_consensus.cc:383] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:57:44.647258 15543 raft_consensus.cc:738] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03985fa59a0545208b4dbdc9eb16232d, State: Initialized, Role: FOLLOWER
I20250622 13:57:44.647884 15543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03985fa59a0545208b4dbdc9eb16232d" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 43047 } }
I20250622 13:57:44.648367 15543 raft_consensus.cc:397] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 13:57:44.648635 15543 raft_consensus.cc:491] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 13:57:44.648928 15543 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:57:44.652392 15543 raft_consensus.cc:513] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03985fa59a0545208b4dbdc9eb16232d" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 43047 } }
I20250622 13:57:44.653072 15543 leader_election.cc:304] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03985fa59a0545208b4dbdc9eb16232d; no voters:
I20250622 13:57:44.655223 15543 leader_election.cc:290] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 13:57:44.655900 15548 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:57:44.657940 15548 raft_consensus.cc:695] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d [term 1 LEADER]: Becoming Leader. State: Replica: 03985fa59a0545208b4dbdc9eb16232d, State: Running, Role: LEADER
I20250622 13:57:44.658514 15548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03985fa59a0545208b4dbdc9eb16232d" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 43047 } }
I20250622 13:57:44.658937 15543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d [sys.catalog]: configured and running, proceeding with master startup.
I20250622 13:57:44.669999 15549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03985fa59a0545208b4dbdc9eb16232d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03985fa59a0545208b4dbdc9eb16232d" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 43047 } } }
I20250622 13:57:44.669517 15550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03985fa59a0545208b4dbdc9eb16232d. Latest consensus state: current_term: 1 leader_uuid: "03985fa59a0545208b4dbdc9eb16232d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03985fa59a0545208b4dbdc9eb16232d" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 43047 } } }
I20250622 13:57:44.670547 15549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d [sys.catalog]: This master's current role is: LEADER
I20250622 13:57:44.670548 15550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d [sys.catalog]: This master's current role is: LEADER
I20250622 13:57:44.674422 15558 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 13:57:44.685492 15558 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 13:57:44.701620 15558 catalog_manager.cc:1349] Generated new cluster ID: bd494f3f44514c85b8ced9d73eeda8f3
I20250622 13:57:44.701929 15558 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 13:57:44.717973 15558 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 13:57:44.719388 15558 catalog_manager.cc:1506] Loading token signing keys...
I20250622 13:57:44.732033 15558 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d: Generated new TSK 0
I20250622 13:57:44.732831 15558 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 13:57:44.748844 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.25.65:0
--local_ip_for_outbound_sockets=127.15.25.65
--webserver_interface=127.15.25.65
--webserver_port=0
--tserver_master_addrs=127.15.25.126:43047
--builtin_ntp_servers=127.15.25.84:34015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250622 13:57:45.034152 15567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:57:45.034662 15567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 13:57:45.035136 15567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:57:45.065263 15567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 13:57:45.066090 15567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.25.65
I20250622 13:57:45.099869 15567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:34015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.25.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.15.25.65
--webserver_port=0
--tserver_master_addrs=127.15.25.126:43047
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.25.65
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 13:57:45.101158 15567 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 13:57:45.102752 15567 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:57:45.119542 15576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:57:45.119661 15574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:57:45.123322 15573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:57:46.208676 15575 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250622 13:57:46.209156 15567 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 13:57:46.213920 15567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 13:57:46.216452 15567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 13:57:46.217844 15567 hybrid_clock.cc:648] HybridClock initialized: now 1750600666217806 us; error 42 us; skew 500 ppm
I20250622 13:57:46.218761 15567 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:57:46.225697 15567 webserver.cc:469] Webserver started at http://127.15.25.65:43729/ using document root <none> and password file <none>
I20250622 13:57:46.226915 15567 fs_manager.cc:362] Metadata directory not provided
I20250622 13:57:46.227180 15567 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:57:46.227746 15567 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:57:46.234035 15567 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "bf2d062dd668430dbb1809685aef8190"
format_stamp: "Formatted at 2025-06-22 13:57:46 on dist-test-slave-6tph"
I20250622 13:57:46.235422 15567 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "bf2d062dd668430dbb1809685aef8190"
format_stamp: "Formatted at 2025-06-22 13:57:46 on dist-test-slave-6tph"
I20250622 13:57:46.244239 15567 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20250622 13:57:46.250921 15583 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:57:46.251972 15567 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250622 13:57:46.252283 15567 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "bf2d062dd668430dbb1809685aef8190"
format_stamp: "Formatted at 2025-06-22 13:57:46 on dist-test-slave-6tph"
I20250622 13:57:46.252588 15567 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:57:46.305617 15567 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:57:46.307109 15567 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 13:57:46.307528 15567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:57:46.310467 15567 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:57:46.314420 15567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:57:46.314613 15567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:57:46.314867 15567 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:57:46.315008 15567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:57:46.442152 15567 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.65:44059
I20250622 13:57:46.442246 15695 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.65:44059 every 8 connection(s)
I20250622 13:57:46.444614 15567 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250622 13:57:46.445302 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 15567
I20250622 13:57:46.445791 15461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250622 13:57:46.453990 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.25.66:0
--local_ip_for_outbound_sockets=127.15.25.66
--webserver_interface=127.15.25.66
--webserver_port=0
--tserver_master_addrs=127.15.25.126:43047
--builtin_ntp_servers=127.15.25.84:34015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 13:57:46.469318 15696 heartbeater.cc:344] Connected to a master server at 127.15.25.126:43047
I20250622 13:57:46.469756 15696 heartbeater.cc:461] Registering TS with master...
I20250622 13:57:46.470755 15696 heartbeater.cc:507] Master 127.15.25.126:43047 requested a full tablet report, sending...
I20250622 13:57:46.473259 15508 ts_manager.cc:194] Registered new tserver with Master: bf2d062dd668430dbb1809685aef8190 (127.15.25.65:44059)
I20250622 13:57:46.475279 15508 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.25.65:53459
W20250622 13:57:46.745594 15700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:57:46.746205 15700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 13:57:46.746771 15700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:57:46.780632 15700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 13:57:46.781504 15700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.25.66
I20250622 13:57:46.814199 15700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:34015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.25.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.15.25.66
--webserver_port=0
--tserver_master_addrs=127.15.25.126:43047
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.25.66
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 13:57:46.815547 15700 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 13:57:46.817097 15700 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:57:46.832160 15707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:57:47.478289 15696 heartbeater.cc:499] Master 127.15.25.126:43047 was elected leader, sending a full tablet report...
W20250622 13:57:46.832203 15706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:57:46.832654 15709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:57:47.919622 15708 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250622 13:57:47.919754 15700 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 13:57:47.923615 15700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 13:57:47.925705 15700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 13:57:47.927065 15700 hybrid_clock.cc:648] HybridClock initialized: now 1750600667927018 us; error 51 us; skew 500 ppm
I20250622 13:57:47.927860 15700 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:57:47.933380 15700 webserver.cc:469] Webserver started at http://127.15.25.66:43141/ using document root <none> and password file <none>
I20250622 13:57:47.934234 15700 fs_manager.cc:362] Metadata directory not provided
I20250622 13:57:47.934439 15700 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:57:47.934859 15700 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:57:47.938946 15700 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "e62fcd99af494fd68cdc66266dc46077"
format_stamp: "Formatted at 2025-06-22 13:57:47 on dist-test-slave-6tph"
I20250622 13:57:47.939913 15700 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "e62fcd99af494fd68cdc66266dc46077"
format_stamp: "Formatted at 2025-06-22 13:57:47 on dist-test-slave-6tph"
I20250622 13:57:47.946455 15700 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250622 13:57:47.951567 15716 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:57:47.952510 15700 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20250622 13:57:47.952833 15700 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "e62fcd99af494fd68cdc66266dc46077"
format_stamp: "Formatted at 2025-06-22 13:57:47 on dist-test-slave-6tph"
I20250622 13:57:47.953122 15700 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:57:48.007776 15700 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:57:48.009075 15700 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 13:57:48.009490 15700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:57:48.012233 15700 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:57:48.016274 15700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:57:48.016458 15700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:57:48.016726 15700 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:57:48.016857 15700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:57:48.140082 15700 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.66:43433
I20250622 13:57:48.140168 15828 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.66:43433 every 8 connection(s)
I20250622 13:57:48.142532 15700 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250622 13:57:48.152763 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 15700
I20250622 13:57:48.153121 15461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250622 13:57:48.158743 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.25.67:0
--local_ip_for_outbound_sockets=127.15.25.67
--webserver_interface=127.15.25.67
--webserver_port=0
--tserver_master_addrs=127.15.25.126:43047
--builtin_ntp_servers=127.15.25.84:34015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 13:57:48.161208 15829 heartbeater.cc:344] Connected to a master server at 127.15.25.126:43047
I20250622 13:57:48.161597 15829 heartbeater.cc:461] Registering TS with master...
I20250622 13:57:48.162539 15829 heartbeater.cc:507] Master 127.15.25.126:43047 requested a full tablet report, sending...
I20250622 13:57:48.164359 15508 ts_manager.cc:194] Registered new tserver with Master: e62fcd99af494fd68cdc66266dc46077 (127.15.25.66:43433)
I20250622 13:57:48.165987 15508 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.25.66:45269
W20250622 13:57:48.433960 15833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:57:48.434367 15833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 13:57:48.434758 15833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:57:48.466174 15833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 13:57:48.466959 15833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.25.67
I20250622 13:57:48.501286 15833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:34015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.25.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.15.25.67
--webserver_port=0
--tserver_master_addrs=127.15.25.126:43047
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.25.67
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 13:57:48.502605 15833 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 13:57:48.504086 15833 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:57:48.519486 15840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:57:49.169119 15829 heartbeater.cc:499] Master 127.15.25.126:43047 was elected leader, sending a full tablet report...
W20250622 13:57:48.520767 15842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:57:48.519513 15839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:57:49.628893 15841 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250622 13:57:49.629069 15833 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 13:57:49.633177 15833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 13:57:49.635391 15833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 13:57:49.636734 15833 hybrid_clock.cc:648] HybridClock initialized: now 1750600669636686 us; error 65 us; skew 500 ppm
I20250622 13:57:49.637494 15833 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:57:49.658842 15833 webserver.cc:469] Webserver started at http://127.15.25.67:46859/ using document root <none> and password file <none>
I20250622 13:57:49.659793 15833 fs_manager.cc:362] Metadata directory not provided
I20250622 13:57:49.660048 15833 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:57:49.660552 15833 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:57:49.664927 15833 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "d01741a82fc94c0598d6dc3f8a7ec34f"
format_stamp: "Formatted at 2025-06-22 13:57:49 on dist-test-slave-6tph"
I20250622 13:57:49.666121 15833 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "d01741a82fc94c0598d6dc3f8a7ec34f"
format_stamp: "Formatted at 2025-06-22 13:57:49 on dist-test-slave-6tph"
I20250622 13:57:49.673084 15833 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20250622 13:57:49.678787 15849 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:57:49.679776 15833 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250622 13:57:49.680092 15833 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "d01741a82fc94c0598d6dc3f8a7ec34f"
format_stamp: "Formatted at 2025-06-22 13:57:49 on dist-test-slave-6tph"
I20250622 13:57:49.680425 15833 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:57:49.736469 15833 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:57:49.737957 15833 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 13:57:49.738389 15833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:57:49.740751 15833 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:57:49.744738 15833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:57:49.744954 15833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:57:49.745203 15833 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:57:49.745357 15833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:57:49.868441 15833 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.67:36419
I20250622 13:57:49.868536 15961 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.67:36419 every 8 connection(s)
I20250622 13:57:49.870775 15833 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250622 13:57:49.876129 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 15833
I20250622 13:57:49.876622 15461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250622 13:57:49.881861 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.25.68:0
--local_ip_for_outbound_sockets=127.15.25.68
--webserver_interface=127.15.25.68
--webserver_port=0
--tserver_master_addrs=127.15.25.126:43047
--builtin_ntp_servers=127.15.25.84:34015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 13:57:49.890055 15962 heartbeater.cc:344] Connected to a master server at 127.15.25.126:43047
I20250622 13:57:49.890420 15962 heartbeater.cc:461] Registering TS with master...
I20250622 13:57:49.891315 15962 heartbeater.cc:507] Master 127.15.25.126:43047 requested a full tablet report, sending...
I20250622 13:57:49.893235 15508 ts_manager.cc:194] Registered new tserver with Master: d01741a82fc94c0598d6dc3f8a7ec34f (127.15.25.67:36419)
I20250622 13:57:49.894357 15508 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.25.67:56735
W20250622 13:57:50.154562 15966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:57:50.155061 15966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 13:57:50.155543 15966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:57:50.185781 15966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 13:57:50.186594 15966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.25.68
I20250622 13:57:50.219696 15966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:34015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.25.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.15.25.68
--webserver_port=0
--tserver_master_addrs=127.15.25.126:43047
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.25.68
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 13:57:50.220968 15966 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 13:57:50.222543 15966 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:57:50.236442 15973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:57:50.897065 15962 heartbeater.cc:499] Master 127.15.25.126:43047 was elected leader, sending a full tablet report...
W20250622 13:57:50.236809 15972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:57:50.239033 15975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:57:50.239126 15966 server_base.cc:1048] running on GCE node
I20250622 13:57:51.332911 15966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 13:57:51.335127 15966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 13:57:51.336472 15966 hybrid_clock.cc:648] HybridClock initialized: now 1750600671336428 us; error 61 us; skew 500 ppm
I20250622 13:57:51.337278 15966 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:57:51.343664 15966 webserver.cc:469] Webserver started at http://127.15.25.68:46283/ using document root <none> and password file <none>
I20250622 13:57:51.344565 15966 fs_manager.cc:362] Metadata directory not provided
I20250622 13:57:51.344769 15966 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:57:51.345240 15966 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:57:51.349622 15966 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "1afa76052465463bae6da1e8fc63fe63"
format_stamp: "Formatted at 2025-06-22 13:57:51 on dist-test-slave-6tph"
I20250622 13:57:51.350701 15966 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "1afa76052465463bae6da1e8fc63fe63"
format_stamp: "Formatted at 2025-06-22 13:57:51 on dist-test-slave-6tph"
I20250622 13:57:51.357713 15966 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250622 13:57:51.363288 15982 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:57:51.364256 15966 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250622 13:57:51.364584 15966 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "1afa76052465463bae6da1e8fc63fe63"
format_stamp: "Formatted at 2025-06-22 13:57:51 on dist-test-slave-6tph"
I20250622 13:57:51.364909 15966 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:57:51.421633 15966 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:57:51.423071 15966 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 13:57:51.423471 15966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:57:51.425854 15966 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:57:51.429478 15966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:57:51.429654 15966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:57:51.429903 15966 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:57:51.430039 15966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:57:51.559460 15966 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.68:43453
I20250622 13:57:51.559543 16094 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.68:43453 every 8 connection(s)
I20250622 13:57:51.561872 15966 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250622 13:57:51.566222 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 15966
I20250622 13:57:51.566779 15461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250622 13:57:51.580802 16095 heartbeater.cc:344] Connected to a master server at 127.15.25.126:43047
I20250622 13:57:51.581175 16095 heartbeater.cc:461] Registering TS with master...
I20250622 13:57:51.582089 16095 heartbeater.cc:507] Master 127.15.25.126:43047 requested a full tablet report, sending...
I20250622 13:57:51.584028 15508 ts_manager.cc:194] Registered new tserver with Master: 1afa76052465463bae6da1e8fc63fe63 (127.15.25.68:43453)
I20250622 13:57:51.585355 15508 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.25.68:56017
I20250622 13:57:51.587018 15461 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250622 13:57:51.622133 15507 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46524:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250622 13:57:51.693245 15764 tablet_service.cc:1468] Processing CreateTablet for tablet 429c18b60ef7410d8b6aa93578dc576f (DEFAULT_TABLE table=TestTable [id=32a0ea662d144a3fb8deb57c3d9f6c66]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 13:57:51.695036 15764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 429c18b60ef7410d8b6aa93578dc576f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:57:51.696861 15897 tablet_service.cc:1468] Processing CreateTablet for tablet 429c18b60ef7410d8b6aa93578dc576f (DEFAULT_TABLE table=TestTable [id=32a0ea662d144a3fb8deb57c3d9f6c66]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 13:57:51.698634 15897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 429c18b60ef7410d8b6aa93578dc576f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:57:51.702402 15631 tablet_service.cc:1468] Processing CreateTablet for tablet 429c18b60ef7410d8b6aa93578dc576f (DEFAULT_TABLE table=TestTable [id=32a0ea662d144a3fb8deb57c3d9f6c66]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 13:57:51.704099 15631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 429c18b60ef7410d8b6aa93578dc576f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:57:51.718268 16114 tablet_bootstrap.cc:492] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f: Bootstrap starting.
I20250622 13:57:51.722632 16115 tablet_bootstrap.cc:492] T 429c18b60ef7410d8b6aa93578dc576f P e62fcd99af494fd68cdc66266dc46077: Bootstrap starting.
I20250622 13:57:51.725359 16114 tablet_bootstrap.cc:654] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f: Neither blocks nor log segments found. Creating new log.
I20250622 13:57:51.727706 16114 log.cc:826] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f: Log is configured to *not* fsync() on all Append() calls
I20250622 13:57:51.728070 16116 tablet_bootstrap.cc:492] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190: Bootstrap starting.
I20250622 13:57:51.729746 16115 tablet_bootstrap.cc:654] T 429c18b60ef7410d8b6aa93578dc576f P e62fcd99af494fd68cdc66266dc46077: Neither blocks nor log segments found. Creating new log.
I20250622 13:57:51.732458 16115 log.cc:826] T 429c18b60ef7410d8b6aa93578dc576f P e62fcd99af494fd68cdc66266dc46077: Log is configured to *not* fsync() on all Append() calls
I20250622 13:57:51.734863 16116 tablet_bootstrap.cc:654] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190: Neither blocks nor log segments found. Creating new log.
I20250622 13:57:51.735402 16114 tablet_bootstrap.cc:492] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f: No bootstrap required, opened a new log
I20250622 13:57:51.735867 16114 ts_tablet_manager.cc:1397] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.004s
I20250622 13:57:51.736569 16116 log.cc:826] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190: Log is configured to *not* fsync() on all Append() calls
I20250622 13:57:51.737357 16115 tablet_bootstrap.cc:492] T 429c18b60ef7410d8b6aa93578dc576f P e62fcd99af494fd68cdc66266dc46077: No bootstrap required, opened a new log
I20250622 13:57:51.737879 16115 ts_tablet_manager.cc:1397] T 429c18b60ef7410d8b6aa93578dc576f P e62fcd99af494fd68cdc66266dc46077: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.007s
I20250622 13:57:51.740676 16116 tablet_bootstrap.cc:492] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190: No bootstrap required, opened a new log
I20250622 13:57:51.741055 16116 ts_tablet_manager.cc:1397] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.005s
I20250622 13:57:51.758782 16116 raft_consensus.cc:357] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62fcd99af494fd68cdc66266dc46077" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 43433 } } peers { permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 36419 } } peers { permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 } }
I20250622 13:57:51.759413 16116 raft_consensus.cc:383] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:57:51.759654 16116 raft_consensus.cc:738] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf2d062dd668430dbb1809685aef8190, State: Initialized, Role: FOLLOWER
I20250622 13:57:51.760416 16116 consensus_queue.cc:260] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62fcd99af494fd68cdc66266dc46077" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 43433 } } peers { permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 36419 } } peers { permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 } }
I20250622 13:57:51.761391 16114 raft_consensus.cc:357] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62fcd99af494fd68cdc66266dc46077" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 43433 } } peers { permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 36419 } } peers { permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 } }
I20250622 13:57:51.762334 16114 raft_consensus.cc:383] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:57:51.762634 16114 raft_consensus.cc:738] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d01741a82fc94c0598d6dc3f8a7ec34f, State: Initialized, Role: FOLLOWER
I20250622 13:57:51.763286 16115 raft_consensus.cc:357] T 429c18b60ef7410d8b6aa93578dc576f P e62fcd99af494fd68cdc66266dc46077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62fcd99af494fd68cdc66266dc46077" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 43433 } } peers { permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 36419 } } peers { permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 } }
I20250622 13:57:51.763569 16114 consensus_queue.cc:260] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62fcd99af494fd68cdc66266dc46077" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 43433 } } peers { permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 36419 } } peers { permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 } }
I20250622 13:57:51.764081 16115 raft_consensus.cc:383] T 429c18b60ef7410d8b6aa93578dc576f P e62fcd99af494fd68cdc66266dc46077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:57:51.764343 16115 raft_consensus.cc:738] T 429c18b60ef7410d8b6aa93578dc576f P e62fcd99af494fd68cdc66266dc46077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e62fcd99af494fd68cdc66266dc46077, State: Initialized, Role: FOLLOWER
I20250622 13:57:51.765442 16115 consensus_queue.cc:260] T 429c18b60ef7410d8b6aa93578dc576f P e62fcd99af494fd68cdc66266dc46077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62fcd99af494fd68cdc66266dc46077" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 43433 } } peers { permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 36419 } } peers { permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 } }
I20250622 13:57:51.766767 16116 ts_tablet_manager.cc:1428] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190: Time spent starting tablet: real 0.025s user 0.022s sys 0.004s
I20250622 13:57:51.769778 16114 ts_tablet_manager.cc:1428] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f: Time spent starting tablet: real 0.034s user 0.023s sys 0.011s
I20250622 13:57:51.772570 16115 ts_tablet_manager.cc:1428] T 429c18b60ef7410d8b6aa93578dc576f P e62fcd99af494fd68cdc66266dc46077: Time spent starting tablet: real 0.034s user 0.032s sys 0.001s
I20250622 13:57:51.840698 16122 raft_consensus.cc:491] T 429c18b60ef7410d8b6aa93578dc576f P e62fcd99af494fd68cdc66266dc46077 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:57:51.841148 16122 raft_consensus.cc:513] T 429c18b60ef7410d8b6aa93578dc576f P e62fcd99af494fd68cdc66266dc46077 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62fcd99af494fd68cdc66266dc46077" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 43433 } } peers { permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 36419 } } peers { permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 } }
I20250622 13:57:51.843415 16122 leader_election.cc:290] T 429c18b60ef7410d8b6aa93578dc576f P e62fcd99af494fd68cdc66266dc46077 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d01741a82fc94c0598d6dc3f8a7ec34f (127.15.25.67:36419), bf2d062dd668430dbb1809685aef8190 (127.15.25.65:44059)
I20250622 13:57:51.853844 15651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "429c18b60ef7410d8b6aa93578dc576f" candidate_uuid: "e62fcd99af494fd68cdc66266dc46077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf2d062dd668430dbb1809685aef8190" is_pre_election: true
I20250622 13:57:51.853881 15917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "429c18b60ef7410d8b6aa93578dc576f" candidate_uuid: "e62fcd99af494fd68cdc66266dc46077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" is_pre_election: true
I20250622 13:57:51.854490 15917 raft_consensus.cc:2466] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e62fcd99af494fd68cdc66266dc46077 in term 0.
I20250622 13:57:51.854490 15651 raft_consensus.cc:2466] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e62fcd99af494fd68cdc66266dc46077 in term 0.
I20250622 13:57:51.855612 15718 leader_election.cc:304] T 429c18b60ef7410d8b6aa93578dc576f P e62fcd99af494fd68cdc66266dc46077 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf2d062dd668430dbb1809685aef8190, e62fcd99af494fd68cdc66266dc46077; no voters:
I20250622 13:57:51.856254 16122 raft_consensus.cc:2802] T 429c18b60ef7410d8b6aa93578dc576f P e62fcd99af494fd68cdc66266dc46077 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:57:51.856492 16122 raft_consensus.cc:491] T 429c18b60ef7410d8b6aa93578dc576f P e62fcd99af494fd68cdc66266dc46077 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:57:51.856724 16122 raft_consensus.cc:3058] T 429c18b60ef7410d8b6aa93578dc576f P e62fcd99af494fd68cdc66266dc46077 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:57:51.860600 16122 raft_consensus.cc:513] T 429c18b60ef7410d8b6aa93578dc576f P e62fcd99af494fd68cdc66266dc46077 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62fcd99af494fd68cdc66266dc46077" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 43433 } } peers { permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 36419 } } peers { permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 } }
I20250622 13:57:51.861977 16122 leader_election.cc:290] T 429c18b60ef7410d8b6aa93578dc576f P e62fcd99af494fd68cdc66266dc46077 [CANDIDATE]: Term 1 election: Requested vote from peers d01741a82fc94c0598d6dc3f8a7ec34f (127.15.25.67:36419), bf2d062dd668430dbb1809685aef8190 (127.15.25.65:44059)
I20250622 13:57:51.862710 15917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "429c18b60ef7410d8b6aa93578dc576f" candidate_uuid: "e62fcd99af494fd68cdc66266dc46077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f"
I20250622 13:57:51.862802 15651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "429c18b60ef7410d8b6aa93578dc576f" candidate_uuid: "e62fcd99af494fd68cdc66266dc46077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf2d062dd668430dbb1809685aef8190"
I20250622 13:57:51.863072 15917 raft_consensus.cc:3058] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:57:51.863176 15651 raft_consensus.cc:3058] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:57:51.867549 15917 raft_consensus.cc:2466] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e62fcd99af494fd68cdc66266dc46077 in term 1.
I20250622 13:57:51.868383 15719 leader_election.cc:304] T 429c18b60ef7410d8b6aa93578dc576f P e62fcd99af494fd68cdc66266dc46077 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d01741a82fc94c0598d6dc3f8a7ec34f, e62fcd99af494fd68cdc66266dc46077; no voters:
I20250622 13:57:51.868754 15651 raft_consensus.cc:2466] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e62fcd99af494fd68cdc66266dc46077 in term 1.
I20250622 13:57:51.868963 16122 raft_consensus.cc:2802] T 429c18b60ef7410d8b6aa93578dc576f P e62fcd99af494fd68cdc66266dc46077 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:57:51.870568 16122 raft_consensus.cc:695] T 429c18b60ef7410d8b6aa93578dc576f P e62fcd99af494fd68cdc66266dc46077 [term 1 LEADER]: Becoming Leader. State: Replica: e62fcd99af494fd68cdc66266dc46077, State: Running, Role: LEADER
I20250622 13:57:51.871263 16122 consensus_queue.cc:237] T 429c18b60ef7410d8b6aa93578dc576f P e62fcd99af494fd68cdc66266dc46077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62fcd99af494fd68cdc66266dc46077" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 43433 } } peers { permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 36419 } } peers { permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 } }
W20250622 13:57:51.877068 15963 tablet.cc:2378] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 13:57:51.882409 15507 catalog_manager.cc:5582] T 429c18b60ef7410d8b6aa93578dc576f P e62fcd99af494fd68cdc66266dc46077 reported cstate change: term changed from 0 to 1, leader changed from <none> to e62fcd99af494fd68cdc66266dc46077 (127.15.25.66). New cstate: current_term: 1 leader_uuid: "e62fcd99af494fd68cdc66266dc46077" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62fcd99af494fd68cdc66266dc46077" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 43433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 36419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 } health_report { overall_health: UNKNOWN } } }
W20250622 13:57:51.900324 15830 tablet.cc:2378] T 429c18b60ef7410d8b6aa93578dc576f P e62fcd99af494fd68cdc66266dc46077: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 13:57:51.936070 15461 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250622 13:57:51.939738 15461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver bf2d062dd668430dbb1809685aef8190 to finish bootstrapping
I20250622 13:57:51.953240 15461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver e62fcd99af494fd68cdc66266dc46077 to finish bootstrapping
W20250622 13:57:51.955557 15697 tablet.cc:2378] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 13:57:51.962951 15461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver d01741a82fc94c0598d6dc3f8a7ec34f to finish bootstrapping
I20250622 13:57:51.972283 15461 kudu-admin-test.cc:709] Waiting for Master to see the current replicas...
I20250622 13:57:51.975209 15461 kudu-admin-test.cc:716] Tablet locations:
tablet_locations {
tablet_id: "429c18b60ef7410d8b6aa93578dc576f"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: LEADER
}
interned_replicas {
ts_info_idx: 1
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "e62fcd99af494fd68cdc66266dc46077"
rpc_addresses {
host: "127.15.25.66"
port: 43433
}
}
ts_infos {
permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f"
rpc_addresses {
host: "127.15.25.67"
port: 36419
}
}
ts_infos {
permanent_uuid: "bf2d062dd668430dbb1809685aef8190"
rpc_addresses {
host: "127.15.25.65"
port: 44059
}
}
I20250622 13:57:52.327769 16122 consensus_queue.cc:1035] T 429c18b60ef7410d8b6aa93578dc576f P e62fcd99af494fd68cdc66266dc46077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:57:52.343135 16122 consensus_queue.cc:1035] T 429c18b60ef7410d8b6aa93578dc576f P e62fcd99af494fd68cdc66266dc46077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 36419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:57:52.343789 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 15700
W20250622 13:57:52.367062 15493 connection.cc:537] server connection from 127.15.25.66:45269 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250622 13:57:52.368144 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 15475
I20250622 13:57:52.396940 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.25.126:43047
--webserver_interface=127.15.25.126
--webserver_port=44089
--builtin_ntp_servers=127.15.25.84:34015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.25.126:43047 with env {}
I20250622 13:57:52.588320 16095 heartbeater.cc:499] Master 127.15.25.126:43047 was elected leader, sending a full tablet report...
W20250622 13:57:52.591624 16095 heartbeater.cc:646] Failed to heartbeat to 127.15.25.126:43047 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.25.126:43047: connect: Connection refused (error 111)
W20250622 13:57:52.674747 16134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:57:52.675310 16134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 13:57:52.675745 16134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:57:52.706463 16134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250622 13:57:52.706815 16134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 13:57:52.707077 16134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250622 13:57:52.707365 16134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250622 13:57:52.741777 16134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:34015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.25.126:43047
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.25.126:43047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.25.126
--webserver_port=44089
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 13:57:52.743196 16134 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 13:57:52.744750 16134 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:57:52.759334 16141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:57:52.785688 15962 heartbeater.cc:646] Failed to heartbeat to 127.15.25.126:43047 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.25.126:43047: connect: Connection refused (error 111)
W20250622 13:57:53.352109 15696 heartbeater.cc:646] Failed to heartbeat to 127.15.25.126:43047 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.25.126:43047: connect: Connection refused (error 111)
I20250622 13:57:53.525595 16149 raft_consensus.cc:491] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:57:53.526036 16149 raft_consensus.cc:513] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62fcd99af494fd68cdc66266dc46077" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 43433 } } peers { permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 36419 } } peers { permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 } }
I20250622 13:57:53.527993 16149 leader_election.cc:290] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e62fcd99af494fd68cdc66266dc46077 (127.15.25.66:43433), bf2d062dd668430dbb1809685aef8190 (127.15.25.65:44059)
W20250622 13:57:53.533650 15851 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.25.66:43433: connect: Connection refused (error 111)
W20250622 13:57:53.538008 15851 leader_election.cc:336] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e62fcd99af494fd68cdc66266dc46077 (127.15.25.66:43433): Network error: Client connection negotiation failed: client connection to 127.15.25.66:43433: connect: Connection refused (error 111)
I20250622 13:57:53.542081 15651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "429c18b60ef7410d8b6aa93578dc576f" candidate_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf2d062dd668430dbb1809685aef8190" is_pre_election: true
I20250622 13:57:53.543128 15851 leader_election.cc:304] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d01741a82fc94c0598d6dc3f8a7ec34f; no voters: bf2d062dd668430dbb1809685aef8190, e62fcd99af494fd68cdc66266dc46077
I20250622 13:57:53.543666 16149 raft_consensus.cc:2747] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250622 13:57:53.839937 16155 raft_consensus.cc:491] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e62fcd99af494fd68cdc66266dc46077)
I20250622 13:57:53.840283 16155 raft_consensus.cc:513] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62fcd99af494fd68cdc66266dc46077" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 43433 } } peers { permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 36419 } } peers { permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 } }
I20250622 13:57:53.842494 16155 leader_election.cc:290] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e62fcd99af494fd68cdc66266dc46077 (127.15.25.66:43433), d01741a82fc94c0598d6dc3f8a7ec34f (127.15.25.67:36419)
W20250622 13:57:53.845645 15585 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.25.66:43433: connect: Connection refused (error 111)
W20250622 13:57:53.856866 15585 leader_election.cc:336] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e62fcd99af494fd68cdc66266dc46077 (127.15.25.66:43433): Network error: Client connection negotiation failed: client connection to 127.15.25.66:43433: connect: Connection refused (error 111)
I20250622 13:57:53.861989 15917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "429c18b60ef7410d8b6aa93578dc576f" candidate_uuid: "bf2d062dd668430dbb1809685aef8190" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" is_pre_election: true
I20250622 13:57:53.862411 15917 raft_consensus.cc:2466] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf2d062dd668430dbb1809685aef8190 in term 1.
I20250622 13:57:53.863535 15586 leader_election.cc:304] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf2d062dd668430dbb1809685aef8190, d01741a82fc94c0598d6dc3f8a7ec34f; no voters: e62fcd99af494fd68cdc66266dc46077
I20250622 13:57:53.864316 16155 raft_consensus.cc:2802] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250622 13:57:53.864557 16155 raft_consensus.cc:491] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e62fcd99af494fd68cdc66266dc46077)
I20250622 13:57:53.864912 16155 raft_consensus.cc:3058] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:57:53.869164 16155 raft_consensus.cc:513] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62fcd99af494fd68cdc66266dc46077" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 43433 } } peers { permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 36419 } } peers { permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 } }
I20250622 13:57:53.870663 16155 leader_election.cc:290] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190 [CANDIDATE]: Term 2 election: Requested vote from peers e62fcd99af494fd68cdc66266dc46077 (127.15.25.66:43433), d01741a82fc94c0598d6dc3f8a7ec34f (127.15.25.67:36419)
I20250622 13:57:53.872159 15917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "429c18b60ef7410d8b6aa93578dc576f" candidate_uuid: "bf2d062dd668430dbb1809685aef8190" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f"
I20250622 13:57:53.872687 15917 raft_consensus.cc:3058] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [term 1 FOLLOWER]: Advancing to term 2
W20250622 13:57:53.874373 15585 leader_election.cc:336] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e62fcd99af494fd68cdc66266dc46077 (127.15.25.66:43433): Network error: Client connection negotiation failed: client connection to 127.15.25.66:43433: connect: Connection refused (error 111)
W20250622 13:57:52.759334 16142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:57:53.874770 16143 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250622 13:57:53.874948 16134 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250622 13:57:52.759637 16144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:57:53.878465 15917 raft_consensus.cc:2466] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf2d062dd668430dbb1809685aef8190 in term 2.
I20250622 13:57:53.879568 15586 leader_election.cc:304] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bf2d062dd668430dbb1809685aef8190, d01741a82fc94c0598d6dc3f8a7ec34f; no voters: e62fcd99af494fd68cdc66266dc46077
I20250622 13:57:53.880338 16155 raft_consensus.cc:2802] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190 [term 2 FOLLOWER]: Leader election won for term 2
I20250622 13:57:53.880507 16134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 13:57:53.882310 16155 raft_consensus.cc:695] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190 [term 2 LEADER]: Becoming Leader. State: Replica: bf2d062dd668430dbb1809685aef8190, State: Running, Role: LEADER
I20250622 13:57:53.883822 16155 consensus_queue.cc:237] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62fcd99af494fd68cdc66266dc46077" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 43433 } } peers { permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 36419 } } peers { permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 } }
I20250622 13:57:53.894181 16134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 13:57:53.897807 16134 hybrid_clock.cc:648] HybridClock initialized: now 1750600673897744 us; error 72 us; skew 500 ppm
I20250622 13:57:53.898519 16134 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:57:53.904219 16134 webserver.cc:469] Webserver started at http://127.15.25.126:44089/ using document root <none> and password file <none>
I20250622 13:57:53.905164 16134 fs_manager.cc:362] Metadata directory not provided
I20250622 13:57:53.905392 16134 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:57:53.912401 16134 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20250622 13:57:53.916424 16165 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:57:53.917447 16134 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20250622 13:57:53.917781 16134 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
uuid: "03985fa59a0545208b4dbdc9eb16232d"
format_stamp: "Formatted at 2025-06-22 13:57:44 on dist-test-slave-6tph"
I20250622 13:57:53.919795 16134 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:57:53.980561 16134 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:57:53.982060 16134 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 13:57:53.982494 16134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:57:54.051613 16134 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.126:43047
I20250622 13:57:54.051824 16216 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.126:43047 every 8 connection(s)
I20250622 13:57:54.054404 16134 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250622 13:57:54.062350 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 16134
I20250622 13:57:54.062805 15461 kudu-admin-test.cc:735] Forcing unsafe config change on tserver d01741a82fc94c0598d6dc3f8a7ec34f
I20250622 13:57:54.063949 16217 sys_catalog.cc:263] Verifying existing consensus state
I20250622 13:57:54.068585 16217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d: Bootstrap starting.
I20250622 13:57:54.121371 16217 log.cc:826] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d: Log is configured to *not* fsync() on all Append() calls
I20250622 13:57:54.149261 16217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:57:54.150285 16217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d: Bootstrap complete.
I20250622 13:57:54.183526 16217 raft_consensus.cc:357] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03985fa59a0545208b4dbdc9eb16232d" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 43047 } }
I20250622 13:57:54.187022 16217 raft_consensus.cc:738] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03985fa59a0545208b4dbdc9eb16232d, State: Initialized, Role: FOLLOWER
I20250622 13:57:54.188097 16217 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03985fa59a0545208b4dbdc9eb16232d" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 43047 } }
I20250622 13:57:54.188870 16217 raft_consensus.cc:397] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 13:57:54.189252 16217 raft_consensus.cc:491] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 13:57:54.189709 16217 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:57:54.197070 16217 raft_consensus.cc:513] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03985fa59a0545208b4dbdc9eb16232d" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 43047 } }
I20250622 13:57:54.197762 16217 leader_election.cc:304] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03985fa59a0545208b4dbdc9eb16232d; no voters:
I20250622 13:57:54.199512 16217 leader_election.cc:290] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d [CANDIDATE]: Term 2 election: Requested vote from peers
I20250622 13:57:54.199923 16221 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d [term 2 FOLLOWER]: Leader election won for term 2
I20250622 13:57:54.202950 16221 raft_consensus.cc:695] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d [term 2 LEADER]: Becoming Leader. State: Replica: 03985fa59a0545208b4dbdc9eb16232d, State: Running, Role: LEADER
I20250622 13:57:54.203789 16221 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03985fa59a0545208b4dbdc9eb16232d" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 43047 } }
I20250622 13:57:54.204223 16217 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d [sys.catalog]: configured and running, proceeding with master startup.
I20250622 13:57:54.215987 16223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03985fa59a0545208b4dbdc9eb16232d. Latest consensus state: current_term: 2 leader_uuid: "03985fa59a0545208b4dbdc9eb16232d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03985fa59a0545208b4dbdc9eb16232d" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 43047 } } }
I20250622 13:57:54.218360 16223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d [sys.catalog]: This master's current role is: LEADER
I20250622 13:57:54.217583 16222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "03985fa59a0545208b4dbdc9eb16232d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03985fa59a0545208b4dbdc9eb16232d" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 43047 } } }
I20250622 13:57:54.220430 16222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d [sys.catalog]: This master's current role is: LEADER
I20250622 13:57:54.225193 16229 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 13:57:54.236553 16229 catalog_manager.cc:671] Loaded metadata for table TestTable [id=32a0ea662d144a3fb8deb57c3d9f6c66]
I20250622 13:57:54.243841 16229 tablet_loader.cc:96] loaded metadata for tablet 429c18b60ef7410d8b6aa93578dc576f (table TestTable [id=32a0ea662d144a3fb8deb57c3d9f6c66])
I20250622 13:57:54.245401 16229 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 13:57:54.250167 16229 catalog_manager.cc:1261] Loaded cluster ID: bd494f3f44514c85b8ced9d73eeda8f3
I20250622 13:57:54.250517 16229 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 13:57:54.258087 16229 catalog_manager.cc:1506] Loading token signing keys...
I20250622 13:57:54.262920 16229 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 03985fa59a0545208b4dbdc9eb16232d: Loaded TSK: 0
I20250622 13:57:54.264568 16229 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250622 13:57:54.413419 16219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:57:54.414011 16219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:57:54.442513 16219 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250622 13:57:54.474581 15917 raft_consensus.cc:1273] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [term 2 FOLLOWER]: Refusing update from remote peer bf2d062dd668430dbb1809685aef8190: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250622 13:57:54.476202 16243 consensus_queue.cc:1035] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 36419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:57:54.512557 15962 heartbeater.cc:344] Connected to a master server at 127.15.25.126:43047
I20250622 13:57:54.521358 16182 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" instance_seqno: 1750600669840782) as {username='slave'} at 127.15.25.67:40537; Asking this server to re-register.
I20250622 13:57:54.523573 15962 heartbeater.cc:461] Registering TS with master...
I20250622 13:57:54.524220 15962 heartbeater.cc:507] Master 127.15.25.126:43047 requested a full tablet report, sending...
I20250622 13:57:54.527889 16182 ts_manager.cc:194] Registered new tserver with Master: d01741a82fc94c0598d6dc3f8a7ec34f (127.15.25.67:36419)
I20250622 13:57:54.533644 16182 catalog_manager.cc:5582] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f reported cstate change: term changed from 1 to 2, leader changed from e62fcd99af494fd68cdc66266dc46077 (127.15.25.66) to bf2d062dd668430dbb1809685aef8190 (127.15.25.65). New cstate: current_term: 2 leader_uuid: "bf2d062dd668430dbb1809685aef8190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62fcd99af494fd68cdc66266dc46077" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 43433 } } peers { permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 36419 } } peers { permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 } } }
W20250622 13:57:54.555842 15585 consensus_peers.cc:487] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190 -> Peer e62fcd99af494fd68cdc66266dc46077 (127.15.25.66:43433): Couldn't send request to peer e62fcd99af494fd68cdc66266dc46077. Status: Network error: Client connection negotiation failed: client connection to 127.15.25.66:43433: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250622 13:57:54.580734 15696 heartbeater.cc:344] Connected to a master server at 127.15.25.126:43047
I20250622 13:57:54.584368 16181 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "bf2d062dd668430dbb1809685aef8190" instance_seqno: 1750600666413012) as {username='slave'} at 127.15.25.65:59269; Asking this server to re-register.
I20250622 13:57:54.585911 15696 heartbeater.cc:461] Registering TS with master...
I20250622 13:57:54.586475 15696 heartbeater.cc:507] Master 127.15.25.126:43047 requested a full tablet report, sending...
I20250622 13:57:54.589345 16181 ts_manager.cc:194] Registered new tserver with Master: bf2d062dd668430dbb1809685aef8190 (127.15.25.65:44059)
I20250622 13:57:54.627012 16095 heartbeater.cc:344] Connected to a master server at 127.15.25.126:43047
I20250622 13:57:54.627470 16095 heartbeater.cc:499] Master 127.15.25.126:43047 was elected leader, sending a full tablet report...
I20250622 13:57:54.629521 16180 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "1afa76052465463bae6da1e8fc63fe63" instance_seqno: 1750600671530155) as {username='slave'} at 127.15.25.68:37665; Asking this server to re-register.
I20250622 13:57:54.631142 16095 heartbeater.cc:461] Registering TS with master...
I20250622 13:57:54.631726 16095 heartbeater.cc:507] Master 127.15.25.126:43047 requested a full tablet report, sending...
I20250622 13:57:54.633656 16180 ts_manager.cc:194] Registered new tserver with Master: 1afa76052465463bae6da1e8fc63fe63 (127.15.25.68:43453)
W20250622 13:57:55.597172 15585 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.25.66:43433, user_credentials={real_user=slave}} blocked reactor thread for 45639.8us
W20250622 13:57:55.852543 16219 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.290s user 0.510s sys 0.734s
W20250622 13:57:55.852881 16219 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.290s user 0.510s sys 0.734s
I20250622 13:57:55.948117 15917 tablet_service.cc:1905] Received UnsafeChangeConfig RPC: dest_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f"
tablet_id: "429c18b60ef7410d8b6aa93578dc576f"
caller_id: "kudu-tools"
new_config {
peers {
permanent_uuid: "bf2d062dd668430dbb1809685aef8190"
}
peers {
permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f"
}
}
from {username='slave'} at 127.0.0.1:45008
W20250622 13:57:55.949465 15917 raft_consensus.cc:2216] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [term 2 FOLLOWER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62fcd99af494fd68cdc66266dc46077" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 43433 } } peers { permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 36419 } } peers { permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 } }NEW CONFIG: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 36419 } } peers { permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 } } unsafe_config_change: true
I20250622 13:57:55.950707 15917 raft_consensus.cc:3058] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [term 2 FOLLOWER]: Advancing to term 3
I20250622 13:57:56.070516 15917 raft_consensus.cc:1238] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [term 3 FOLLOWER]: Rejecting Update request from peer bf2d062dd668430dbb1809685aef8190 for earlier term 2. Current term is 3. Ops: []
I20250622 13:57:56.071875 16263 consensus_queue.cc:1046] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 36419 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250622 13:57:56.073324 16276 raft_consensus.cc:3053] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190 [term 2 LEADER]: Stepping down as leader of term 2
I20250622 13:57:56.073642 16276 raft_consensus.cc:738] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bf2d062dd668430dbb1809685aef8190, State: Running, Role: LEADER
I20250622 13:57:56.074358 16276 consensus_queue.cc:260] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62fcd99af494fd68cdc66266dc46077" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 43433 } } peers { permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 36419 } } peers { permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 } }
I20250622 13:57:56.075259 16276 raft_consensus.cc:3058] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190 [term 2 FOLLOWER]: Advancing to term 3
I20250622 13:57:57.460090 16279 raft_consensus.cc:491] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools)
I20250622 13:57:57.460510 16279 raft_consensus.cc:513] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 36419 } } peers { permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 } } unsafe_config_change: true
I20250622 13:57:57.461548 16279 leader_election.cc:290] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bf2d062dd668430dbb1809685aef8190 (127.15.25.65:44059)
I20250622 13:57:57.462819 15651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "429c18b60ef7410d8b6aa93578dc576f" candidate_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bf2d062dd668430dbb1809685aef8190" is_pre_election: true
I20250622 13:57:57.463305 15651 raft_consensus.cc:2466] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d01741a82fc94c0598d6dc3f8a7ec34f in term 3.
I20250622 13:57:57.464200 15851 leader_election.cc:304] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: bf2d062dd668430dbb1809685aef8190, d01741a82fc94c0598d6dc3f8a7ec34f; no voters:
I20250622 13:57:57.464742 16279 raft_consensus.cc:2802] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250622 13:57:57.465027 16279 raft_consensus.cc:491] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [term 3 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools)
I20250622 13:57:57.465214 16279 raft_consensus.cc:3058] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [term 3 FOLLOWER]: Advancing to term 4
I20250622 13:57:57.468806 16279 raft_consensus.cc:513] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 36419 } } peers { permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 } } unsafe_config_change: true
I20250622 13:57:57.469655 16279 leader_election.cc:290] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [CANDIDATE]: Term 4 election: Requested vote from peers bf2d062dd668430dbb1809685aef8190 (127.15.25.65:44059)
I20250622 13:57:57.470563 15651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "429c18b60ef7410d8b6aa93578dc576f" candidate_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bf2d062dd668430dbb1809685aef8190"
I20250622 13:57:57.470942 15651 raft_consensus.cc:3058] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190 [term 3 FOLLOWER]: Advancing to term 4
I20250622 13:57:57.474872 15651 raft_consensus.cc:2466] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d01741a82fc94c0598d6dc3f8a7ec34f in term 4.
I20250622 13:57:57.475553 15851 leader_election.cc:304] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: bf2d062dd668430dbb1809685aef8190, d01741a82fc94c0598d6dc3f8a7ec34f; no voters:
I20250622 13:57:57.476038 16279 raft_consensus.cc:2802] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [term 4 FOLLOWER]: Leader election won for term 4
I20250622 13:57:57.476707 16279 raft_consensus.cc:695] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [term 4 LEADER]: Becoming Leader. State: Replica: d01741a82fc94c0598d6dc3f8a7ec34f, State: Running, Role: LEADER
I20250622 13:57:57.477420 16279 consensus_queue.cc:237] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 36419 } } peers { permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 } } unsafe_config_change: true
I20250622 13:57:57.482689 16182 catalog_manager.cc:5582] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f reported cstate change: term changed from 2 to 4, leader changed from bf2d062dd668430dbb1809685aef8190 (127.15.25.65) to d01741a82fc94c0598d6dc3f8a7ec34f (127.15.25.67), now has a pending config: VOTER d01741a82fc94c0598d6dc3f8a7ec34f (127.15.25.67), VOTER bf2d062dd668430dbb1809685aef8190 (127.15.25.65). New cstate: current_term: 4 leader_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62fcd99af494fd68cdc66266dc46077" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 43433 } } peers { permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 36419 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 36419 } } peers { permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 } } unsafe_config_change: true }
I20250622 13:57:57.952872 15651 raft_consensus.cc:1273] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190 [term 4 FOLLOWER]: Refusing update from remote peer d01741a82fc94c0598d6dc3f8a7ec34f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250622 13:57:57.954147 16279 consensus_queue.cc:1035] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250622 13:57:57.961084 16280 raft_consensus.cc:2953] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [term 4 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER e62fcd99af494fd68cdc66266dc46077 (127.15.25.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 36419 } } peers { permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 } } unsafe_config_change: true }
I20250622 13:57:57.962599 15651 raft_consensus.cc:2953] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190 [term 4 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER e62fcd99af494fd68cdc66266dc46077 (127.15.25.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 36419 } } peers { permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 } } unsafe_config_change: true }
I20250622 13:57:57.971233 16182 catalog_manager.cc:5582] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f reported cstate change: config changed from index -1 to 3, VOTER e62fcd99af494fd68cdc66266dc46077 (127.15.25.66) evicted, no longer has a pending config: VOTER d01741a82fc94c0598d6dc3f8a7ec34f (127.15.25.67), VOTER bf2d062dd668430dbb1809685aef8190 (127.15.25.65). New cstate: current_term: 4 leader_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 36419 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
W20250622 13:57:57.977793 16182 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet 429c18b60ef7410d8b6aa93578dc576f on TS e62fcd99af494fd68cdc66266dc46077: Not found: failed to reset TS proxy: Could not find TS for UUID e62fcd99af494fd68cdc66266dc46077
I20250622 13:57:57.991724 15917 consensus_queue.cc:237] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 36419 } } peers { permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 } } peers { permanent_uuid: "1afa76052465463bae6da1e8fc63fe63" member_type: NON_VOTER last_known_addr { host: "127.15.25.68" port: 43453 } attrs { promote: true } } unsafe_config_change: true
I20250622 13:57:57.995548 15651 raft_consensus.cc:1273] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190 [term 4 FOLLOWER]: Refusing update from remote peer d01741a82fc94c0598d6dc3f8a7ec34f: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250622 13:57:57.996966 16279 consensus_queue.cc:1035] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250622 13:57:58.002251 16280 raft_consensus.cc:2953] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER 1afa76052465463bae6da1e8fc63fe63 (127.15.25.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 36419 } } peers { permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 } } peers { permanent_uuid: "1afa76052465463bae6da1e8fc63fe63" member_type: NON_VOTER last_known_addr { host: "127.15.25.68" port: 43453 } attrs { promote: true } } unsafe_config_change: true }
I20250622 13:57:58.004411 15651 raft_consensus.cc:2953] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER 1afa76052465463bae6da1e8fc63fe63 (127.15.25.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 36419 } } peers { permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 } } peers { permanent_uuid: "1afa76052465463bae6da1e8fc63fe63" member_type: NON_VOTER last_known_addr { host: "127.15.25.68" port: 43453 } attrs { promote: true } } unsafe_config_change: true }
W20250622 13:57:58.007609 15853 consensus_peers.cc:487] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f -> Peer 1afa76052465463bae6da1e8fc63fe63 (127.15.25.68:43453): Couldn't send request to peer 1afa76052465463bae6da1e8fc63fe63. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 429c18b60ef7410d8b6aa93578dc576f. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:57:58.009425 16168 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 429c18b60ef7410d8b6aa93578dc576f with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250622 13:57:58.012576 16182 catalog_manager.cc:5582] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190 reported cstate change: config changed from index 3 to 5, NON_VOTER 1afa76052465463bae6da1e8fc63fe63 (127.15.25.68) added. New cstate: current_term: 4 leader_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 36419 } } peers { permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 } } peers { permanent_uuid: "1afa76052465463bae6da1e8fc63fe63" member_type: NON_VOTER last_known_addr { host: "127.15.25.68" port: 43453 } attrs { promote: true } } unsafe_config_change: true }
W20250622 13:57:58.030350 16167 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 429c18b60ef7410d8b6aa93578dc576f on TS e62fcd99af494fd68cdc66266dc46077 failed: Not found: failed to reset TS proxy: Could not find TS for UUID e62fcd99af494fd68cdc66266dc46077
I20250622 13:57:58.568583 16296 ts_tablet_manager.cc:927] T 429c18b60ef7410d8b6aa93578dc576f P 1afa76052465463bae6da1e8fc63fe63: Initiating tablet copy from peer d01741a82fc94c0598d6dc3f8a7ec34f (127.15.25.67:36419)
I20250622 13:57:58.571309 16296 tablet_copy_client.cc:323] T 429c18b60ef7410d8b6aa93578dc576f P 1afa76052465463bae6da1e8fc63fe63: tablet copy: Beginning tablet copy session from remote peer at address 127.15.25.67:36419
I20250622 13:57:58.582926 15937 tablet_copy_service.cc:140] P d01741a82fc94c0598d6dc3f8a7ec34f: Received BeginTabletCopySession request for tablet 429c18b60ef7410d8b6aa93578dc576f from peer 1afa76052465463bae6da1e8fc63fe63 ({username='slave'} at 127.15.25.68:45447)
I20250622 13:57:58.583428 15937 tablet_copy_service.cc:161] P d01741a82fc94c0598d6dc3f8a7ec34f: Beginning new tablet copy session on tablet 429c18b60ef7410d8b6aa93578dc576f from peer 1afa76052465463bae6da1e8fc63fe63 at {username='slave'} at 127.15.25.68:45447: session id = 1afa76052465463bae6da1e8fc63fe63-429c18b60ef7410d8b6aa93578dc576f
I20250622 13:57:58.588492 15937 tablet_copy_source_session.cc:215] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 13:57:58.593101 16296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 429c18b60ef7410d8b6aa93578dc576f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:57:58.610096 16296 tablet_copy_client.cc:806] T 429c18b60ef7410d8b6aa93578dc576f P 1afa76052465463bae6da1e8fc63fe63: tablet copy: Starting download of 0 data blocks...
I20250622 13:57:58.610610 16296 tablet_copy_client.cc:670] T 429c18b60ef7410d8b6aa93578dc576f P 1afa76052465463bae6da1e8fc63fe63: tablet copy: Starting download of 1 WAL segments...
I20250622 13:57:58.613883 16296 tablet_copy_client.cc:538] T 429c18b60ef7410d8b6aa93578dc576f P 1afa76052465463bae6da1e8fc63fe63: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 13:57:58.619216 16296 tablet_bootstrap.cc:492] T 429c18b60ef7410d8b6aa93578dc576f P 1afa76052465463bae6da1e8fc63fe63: Bootstrap starting.
I20250622 13:57:58.630470 16296 log.cc:826] T 429c18b60ef7410d8b6aa93578dc576f P 1afa76052465463bae6da1e8fc63fe63: Log is configured to *not* fsync() on all Append() calls
I20250622 13:57:58.641139 16296 tablet_bootstrap.cc:492] T 429c18b60ef7410d8b6aa93578dc576f P 1afa76052465463bae6da1e8fc63fe63: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:57:58.641873 16296 tablet_bootstrap.cc:492] T 429c18b60ef7410d8b6aa93578dc576f P 1afa76052465463bae6da1e8fc63fe63: Bootstrap complete.
I20250622 13:57:58.642410 16296 ts_tablet_manager.cc:1397] T 429c18b60ef7410d8b6aa93578dc576f P 1afa76052465463bae6da1e8fc63fe63: Time spent bootstrapping tablet: real 0.023s user 0.018s sys 0.005s
I20250622 13:57:58.660148 16296 raft_consensus.cc:357] T 429c18b60ef7410d8b6aa93578dc576f P 1afa76052465463bae6da1e8fc63fe63 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 36419 } } peers { permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 } } peers { permanent_uuid: "1afa76052465463bae6da1e8fc63fe63" member_type: NON_VOTER last_known_addr { host: "127.15.25.68" port: 43453 } attrs { promote: true } } unsafe_config_change: true
I20250622 13:57:58.660964 16296 raft_consensus.cc:738] T 429c18b60ef7410d8b6aa93578dc576f P 1afa76052465463bae6da1e8fc63fe63 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 1afa76052465463bae6da1e8fc63fe63, State: Initialized, Role: LEARNER
I20250622 13:57:58.661574 16296 consensus_queue.cc:260] T 429c18b60ef7410d8b6aa93578dc576f P 1afa76052465463bae6da1e8fc63fe63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 36419 } } peers { permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 } } peers { permanent_uuid: "1afa76052465463bae6da1e8fc63fe63" member_type: NON_VOTER last_known_addr { host: "127.15.25.68" port: 43453 } attrs { promote: true } } unsafe_config_change: true
I20250622 13:57:58.664881 16296 ts_tablet_manager.cc:1428] T 429c18b60ef7410d8b6aa93578dc576f P 1afa76052465463bae6da1e8fc63fe63: Time spent starting tablet: real 0.022s user 0.022s sys 0.000s
I20250622 13:57:58.666420 15937 tablet_copy_service.cc:342] P d01741a82fc94c0598d6dc3f8a7ec34f: Request end of tablet copy session 1afa76052465463bae6da1e8fc63fe63-429c18b60ef7410d8b6aa93578dc576f received from {username='slave'} at 127.15.25.68:45447
I20250622 13:57:58.666790 15937 tablet_copy_service.cc:434] P d01741a82fc94c0598d6dc3f8a7ec34f: ending tablet copy session 1afa76052465463bae6da1e8fc63fe63-429c18b60ef7410d8b6aa93578dc576f on tablet 429c18b60ef7410d8b6aa93578dc576f with peer 1afa76052465463bae6da1e8fc63fe63
I20250622 13:57:59.184909 16050 raft_consensus.cc:1215] T 429c18b60ef7410d8b6aa93578dc576f P 1afa76052465463bae6da1e8fc63fe63 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.4->[4.5-4.5] Dedup: 4.5->[]
W20250622 13:57:59.197194 16167 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 429c18b60ef7410d8b6aa93578dc576f on TS e62fcd99af494fd68cdc66266dc46077 failed: Not found: failed to reset TS proxy: Could not find TS for UUID e62fcd99af494fd68cdc66266dc46077
I20250622 13:57:59.622051 16303 raft_consensus.cc:1062] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f: attempting to promote NON_VOTER 1afa76052465463bae6da1e8fc63fe63 to VOTER
I20250622 13:57:59.623668 16303 consensus_queue.cc:237] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 36419 } } peers { permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 } } peers { permanent_uuid: "1afa76052465463bae6da1e8fc63fe63" member_type: VOTER last_known_addr { host: "127.15.25.68" port: 43453 } attrs { promote: false } } unsafe_config_change: true
I20250622 13:57:59.627919 15651 raft_consensus.cc:1273] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190 [term 4 FOLLOWER]: Refusing update from remote peer d01741a82fc94c0598d6dc3f8a7ec34f: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250622 13:57:59.627995 16050 raft_consensus.cc:1273] T 429c18b60ef7410d8b6aa93578dc576f P 1afa76052465463bae6da1e8fc63fe63 [term 4 LEARNER]: Refusing update from remote peer d01741a82fc94c0598d6dc3f8a7ec34f: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250622 13:57:59.629026 16280 consensus_queue.cc:1035] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250622 13:57:59.629603 16303 consensus_queue.cc:1035] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1afa76052465463bae6da1e8fc63fe63" member_type: VOTER last_known_addr { host: "127.15.25.68" port: 43453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250622 13:57:59.634946 16280 raft_consensus.cc:2953] T 429c18b60ef7410d8b6aa93578dc576f P d01741a82fc94c0598d6dc3f8a7ec34f [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 1afa76052465463bae6da1e8fc63fe63 (127.15.25.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 36419 } } peers { permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 } } peers { permanent_uuid: "1afa76052465463bae6da1e8fc63fe63" member_type: VOTER last_known_addr { host: "127.15.25.68" port: 43453 } attrs { promote: false } } unsafe_config_change: true }
I20250622 13:57:59.636340 15651 raft_consensus.cc:2953] T 429c18b60ef7410d8b6aa93578dc576f P bf2d062dd668430dbb1809685aef8190 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 1afa76052465463bae6da1e8fc63fe63 (127.15.25.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 36419 } } peers { permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 } } peers { permanent_uuid: "1afa76052465463bae6da1e8fc63fe63" member_type: VOTER last_known_addr { host: "127.15.25.68" port: 43453 } attrs { promote: false } } unsafe_config_change: true }
I20250622 13:57:59.637200 16050 raft_consensus.cc:2953] T 429c18b60ef7410d8b6aa93578dc576f P 1afa76052465463bae6da1e8fc63fe63 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 1afa76052465463bae6da1e8fc63fe63 (127.15.25.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 36419 } } peers { permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 } } peers { permanent_uuid: "1afa76052465463bae6da1e8fc63fe63" member_type: VOTER last_known_addr { host: "127.15.25.68" port: 43453 } attrs { promote: false } } unsafe_config_change: true }
I20250622 13:57:59.646538 16181 catalog_manager.cc:5582] T 429c18b60ef7410d8b6aa93578dc576f P 1afa76052465463bae6da1e8fc63fe63 reported cstate change: config changed from index 5 to 6, 1afa76052465463bae6da1e8fc63fe63 (127.15.25.68) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 36419 } } peers { permanent_uuid: "bf2d062dd668430dbb1809685aef8190" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44059 } } peers { permanent_uuid: "1afa76052465463bae6da1e8fc63fe63" member_type: VOTER last_known_addr { host: "127.15.25.68" port: 43453 } attrs { promote: false } } unsafe_config_change: true }
I20250622 13:57:59.747206 15461 kudu-admin-test.cc:751] Waiting for Master to see new config...
I20250622 13:57:59.760190 15461 kudu-admin-test.cc:756] Tablet locations:
tablet_locations {
tablet_id: "429c18b60ef7410d8b6aa93578dc576f"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: LEADER
}
interned_replicas {
ts_info_idx: 1
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "d01741a82fc94c0598d6dc3f8a7ec34f"
rpc_addresses {
host: "127.15.25.67"
port: 36419
}
}
ts_infos {
permanent_uuid: "bf2d062dd668430dbb1809685aef8190"
rpc_addresses {
host: "127.15.25.65"
port: 44059
}
}
ts_infos {
permanent_uuid: "1afa76052465463bae6da1e8fc63fe63"
rpc_addresses {
host: "127.15.25.68"
port: 43453
}
}
I20250622 13:57:59.762363 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 15567
I20250622 13:57:59.785492 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 15833
I20250622 13:57:59.810385 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 15966
I20250622 13:57:59.835043 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 16134
2025-06-22T13:57:59Z chronyd exiting
[ OK ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes (18098 ms)
[ RUN ] AdminCliTest.TestGracefulSpecificLeaderStepDown
I20250622 13:57:59.892858 15461 test_util.cc:276] Using random seed: 2074814306
I20250622 13:57:59.898342 15461 ts_itest-base.cc:115] Starting cluster with:
I20250622 13:57:59.898492 15461 ts_itest-base.cc:116] --------------
I20250622 13:57:59.898597 15461 ts_itest-base.cc:117] 3 tablet servers
I20250622 13:57:59.898695 15461 ts_itest-base.cc:118] 3 replicas per TS
I20250622 13:57:59.898797 15461 ts_itest-base.cc:119] --------------
2025-06-22T13:57:59Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-22T13:57:59Z Disabled control of system clock
I20250622 13:57:59.931381 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.25.126:39933
--webserver_interface=127.15.25.126
--webserver_port=0
--builtin_ntp_servers=127.15.25.84:39511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.25.126:39933
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250622 13:58:00.204207 16322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:58:00.204766 16322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 13:58:00.205232 16322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:58:00.234043 16322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250622 13:58:00.234294 16322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 13:58:00.234614 16322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250622 13:58:00.234898 16322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250622 13:58:00.268800 16322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:39511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.15.25.126:39933
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.25.126:39933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.25.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 13:58:00.270103 16322 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 13:58:00.271677 16322 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:58:00.287025 16328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:58:00.287053 16329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:58:00.289108 16322 server_base.cc:1048] running on GCE node
W20250622 13:58:00.287190 16331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:58:01.401930 16322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 13:58:01.404867 16322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 13:58:01.406244 16322 hybrid_clock.cc:648] HybridClock initialized: now 1750600681406206 us; error 58 us; skew 500 ppm
I20250622 13:58:01.406950 16322 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:58:01.416229 16322 webserver.cc:469] Webserver started at http://127.15.25.126:35543/ using document root <none> and password file <none>
I20250622 13:58:01.417119 16322 fs_manager.cc:362] Metadata directory not provided
I20250622 13:58:01.417305 16322 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:58:01.417797 16322 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:58:01.422055 16322 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "0107dbd58759409d8e24c233c803ac5f"
format_stamp: "Formatted at 2025-06-22 13:58:01 on dist-test-slave-6tph"
I20250622 13:58:01.423136 16322 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "0107dbd58759409d8e24c233c803ac5f"
format_stamp: "Formatted at 2025-06-22 13:58:01 on dist-test-slave-6tph"
I20250622 13:58:01.429636 16322 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250622 13:58:01.435035 16338 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:01.435976 16322 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250622 13:58:01.436296 16322 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
uuid: "0107dbd58759409d8e24c233c803ac5f"
format_stamp: "Formatted at 2025-06-22 13:58:01 on dist-test-slave-6tph"
I20250622 13:58:01.436595 16322 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:58:01.498063 16322 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:58:01.499392 16322 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 13:58:01.499826 16322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:58:01.566046 16322 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.126:39933
I20250622 13:58:01.566110 16389 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.126:39933 every 8 connection(s)
I20250622 13:58:01.568509 16322 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250622 13:58:01.572453 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 16322
I20250622 13:58:01.572929 15461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250622 13:58:01.573928 16390 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:58:01.597182 16390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0107dbd58759409d8e24c233c803ac5f: Bootstrap starting.
I20250622 13:58:01.601959 16390 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0107dbd58759409d8e24c233c803ac5f: Neither blocks nor log segments found. Creating new log.
I20250622 13:58:01.603442 16390 log.cc:826] T 00000000000000000000000000000000 P 0107dbd58759409d8e24c233c803ac5f: Log is configured to *not* fsync() on all Append() calls
I20250622 13:58:01.607206 16390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0107dbd58759409d8e24c233c803ac5f: No bootstrap required, opened a new log
I20250622 13:58:01.621893 16390 raft_consensus.cc:357] T 00000000000000000000000000000000 P 0107dbd58759409d8e24c233c803ac5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0107dbd58759409d8e24c233c803ac5f" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 39933 } }
I20250622 13:58:01.622516 16390 raft_consensus.cc:383] T 00000000000000000000000000000000 P 0107dbd58759409d8e24c233c803ac5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:58:01.622795 16390 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0107dbd58759409d8e24c233c803ac5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0107dbd58759409d8e24c233c803ac5f, State: Initialized, Role: FOLLOWER
I20250622 13:58:01.623610 16390 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0107dbd58759409d8e24c233c803ac5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0107dbd58759409d8e24c233c803ac5f" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 39933 } }
I20250622 13:58:01.624122 16390 raft_consensus.cc:397] T 00000000000000000000000000000000 P 0107dbd58759409d8e24c233c803ac5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 13:58:01.624375 16390 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0107dbd58759409d8e24c233c803ac5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 13:58:01.624629 16390 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 0107dbd58759409d8e24c233c803ac5f [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:58:01.627960 16390 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0107dbd58759409d8e24c233c803ac5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0107dbd58759409d8e24c233c803ac5f" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 39933 } }
I20250622 13:58:01.628535 16390 leader_election.cc:304] T 00000000000000000000000000000000 P 0107dbd58759409d8e24c233c803ac5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0107dbd58759409d8e24c233c803ac5f; no voters:
I20250622 13:58:01.630087 16390 leader_election.cc:290] T 00000000000000000000000000000000 P 0107dbd58759409d8e24c233c803ac5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 13:58:01.630757 16395 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 0107dbd58759409d8e24c233c803ac5f [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:58:01.632665 16395 raft_consensus.cc:695] T 00000000000000000000000000000000 P 0107dbd58759409d8e24c233c803ac5f [term 1 LEADER]: Becoming Leader. State: Replica: 0107dbd58759409d8e24c233c803ac5f, State: Running, Role: LEADER
I20250622 13:58:01.633432 16395 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0107dbd58759409d8e24c233c803ac5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0107dbd58759409d8e24c233c803ac5f" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 39933 } }
I20250622 13:58:01.634953 16390 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0107dbd58759409d8e24c233c803ac5f [sys.catalog]: configured and running, proceeding with master startup.
I20250622 13:58:01.643812 16397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0107dbd58759409d8e24c233c803ac5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0107dbd58759409d8e24c233c803ac5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0107dbd58759409d8e24c233c803ac5f" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 39933 } } }
I20250622 13:58:01.646091 16397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0107dbd58759409d8e24c233c803ac5f [sys.catalog]: This master's current role is: LEADER
I20250622 13:58:01.645256 16396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0107dbd58759409d8e24c233c803ac5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0107dbd58759409d8e24c233c803ac5f. Latest consensus state: current_term: 1 leader_uuid: "0107dbd58759409d8e24c233c803ac5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0107dbd58759409d8e24c233c803ac5f" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 39933 } } }
I20250622 13:58:01.647256 16396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0107dbd58759409d8e24c233c803ac5f [sys.catalog]: This master's current role is: LEADER
I20250622 13:58:01.650344 16404 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 13:58:01.659996 16404 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 13:58:01.675163 16404 catalog_manager.cc:1349] Generated new cluster ID: d1e0ab52dde243b8af508c43b5c3f4b9
I20250622 13:58:01.675475 16404 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 13:58:01.719532 16404 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 13:58:01.721462 16404 catalog_manager.cc:1506] Loading token signing keys...
I20250622 13:58:01.737990 16404 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 0107dbd58759409d8e24c233c803ac5f: Generated new TSK 0
I20250622 13:58:01.739074 16404 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 13:58:01.763857 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.25.65:0
--local_ip_for_outbound_sockets=127.15.25.65
--webserver_interface=127.15.25.65
--webserver_port=0
--tserver_master_addrs=127.15.25.126:39933
--builtin_ntp_servers=127.15.25.84:39511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250622 13:58:02.040059 16414 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250622 13:58:02.040700 16414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:58:02.040997 16414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 13:58:02.041489 16414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:58:02.071444 16414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 13:58:02.072278 16414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.25.65
I20250622 13:58:02.105683 16414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:39511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.25.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.15.25.65
--webserver_port=0
--tserver_master_addrs=127.15.25.126:39933
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.25.65
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 13:58:02.107025 16414 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 13:58:02.108453 16414 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:58:02.124342 16421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:58:02.124513 16422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:58:03.267901 16423 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250622 13:58:03.267920 16414 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250622 13:58:02.127780 16424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:58:03.271631 16414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 13:58:03.274143 16414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 13:58:03.275530 16414 hybrid_clock.cc:648] HybridClock initialized: now 1750600683275476 us; error 71 us; skew 500 ppm
I20250622 13:58:03.276250 16414 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:58:03.282857 16414 webserver.cc:469] Webserver started at http://127.15.25.65:42433/ using document root <none> and password file <none>
I20250622 13:58:03.283780 16414 fs_manager.cc:362] Metadata directory not provided
I20250622 13:58:03.283967 16414 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:58:03.284399 16414 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:58:03.288394 16414 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "3833939ed3d44b71813f154d2743ca34"
format_stamp: "Formatted at 2025-06-22 13:58:03 on dist-test-slave-6tph"
I20250622 13:58:03.289415 16414 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "3833939ed3d44b71813f154d2743ca34"
format_stamp: "Formatted at 2025-06-22 13:58:03 on dist-test-slave-6tph"
I20250622 13:58:03.296030 16414 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250622 13:58:03.301448 16431 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:03.302500 16414 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250622 13:58:03.302798 16414 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "3833939ed3d44b71813f154d2743ca34"
format_stamp: "Formatted at 2025-06-22 13:58:03 on dist-test-slave-6tph"
I20250622 13:58:03.303071 16414 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:58:03.361583 16414 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:58:03.362926 16414 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 13:58:03.363315 16414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:58:03.365556 16414 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:58:03.369426 16414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:58:03.369616 16414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:03.369896 16414 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:58:03.370049 16414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:03.488358 16414 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.65:43999
I20250622 13:58:03.488451 16543 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.65:43999 every 8 connection(s)
I20250622 13:58:03.490744 16414 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250622 13:58:03.497464 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 16414
I20250622 13:58:03.497891 15461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250622 13:58:03.503501 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.25.66:0
--local_ip_for_outbound_sockets=127.15.25.66
--webserver_interface=127.15.25.66
--webserver_port=0
--tserver_master_addrs=127.15.25.126:39933
--builtin_ntp_servers=127.15.25.84:39511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250622 13:58:03.509410 16544 heartbeater.cc:344] Connected to a master server at 127.15.25.126:39933
I20250622 13:58:03.509860 16544 heartbeater.cc:461] Registering TS with master...
I20250622 13:58:03.510807 16544 heartbeater.cc:507] Master 127.15.25.126:39933 requested a full tablet report, sending...
I20250622 13:58:03.513025 16355 ts_manager.cc:194] Registered new tserver with Master: 3833939ed3d44b71813f154d2743ca34 (127.15.25.65:43999)
I20250622 13:58:03.514801 16355 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.25.65:39621
W20250622 13:58:03.783025 16548 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250622 13:58:03.783747 16548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:58:03.784082 16548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 13:58:03.784580 16548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:58:03.812855 16548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 13:58:03.813603 16548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.25.66
I20250622 13:58:03.844164 16548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:39511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.25.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.15.25.66
--webserver_port=0
--tserver_master_addrs=127.15.25.126:39933
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.25.66
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 13:58:03.845355 16548 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 13:58:03.846824 16548 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:58:03.862385 16555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:58:04.517686 16544 heartbeater.cc:499] Master 127.15.25.126:39933 was elected leader, sending a full tablet report...
W20250622 13:58:03.862426 16557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:58:03.862466 16554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:58:03.863829 16548 server_base.cc:1048] running on GCE node
I20250622 13:58:04.942780 16548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 13:58:04.944928 16548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 13:58:04.946275 16548 hybrid_clock.cc:648] HybridClock initialized: now 1750600684946245 us; error 57 us; skew 500 ppm
I20250622 13:58:04.947057 16548 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:58:04.953007 16548 webserver.cc:469] Webserver started at http://127.15.25.66:42359/ using document root <none> and password file <none>
I20250622 13:58:04.953934 16548 fs_manager.cc:362] Metadata directory not provided
I20250622 13:58:04.954160 16548 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:58:04.954613 16548 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:58:04.958880 16548 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "aee733dd7e33474eac3bd30dab37cc4c"
format_stamp: "Formatted at 2025-06-22 13:58:04 on dist-test-slave-6tph"
I20250622 13:58:04.959996 16548 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "aee733dd7e33474eac3bd30dab37cc4c"
format_stamp: "Formatted at 2025-06-22 13:58:04 on dist-test-slave-6tph"
I20250622 13:58:04.966780 16548 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20250622 13:58:04.972334 16564 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:04.973383 16548 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20250622 13:58:04.973680 16548 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "aee733dd7e33474eac3bd30dab37cc4c"
format_stamp: "Formatted at 2025-06-22 13:58:04 on dist-test-slave-6tph"
I20250622 13:58:04.974057 16548 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:58:05.020648 16548 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:58:05.022143 16548 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 13:58:05.022557 16548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:58:05.024946 16548 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:58:05.028759 16548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:58:05.028972 16548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:05.029203 16548 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:58:05.029358 16548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:05.161532 16548 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.66:43459
I20250622 13:58:05.161623 16676 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.66:43459 every 8 connection(s)
I20250622 13:58:05.163996 16548 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250622 13:58:05.169698 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 16548
I20250622 13:58:05.170073 15461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250622 13:58:05.175287 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.25.67:0
--local_ip_for_outbound_sockets=127.15.25.67
--webserver_interface=127.15.25.67
--webserver_port=0
--tserver_master_addrs=127.15.25.126:39933
--builtin_ntp_servers=127.15.25.84:39511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250622 13:58:05.183743 16677 heartbeater.cc:344] Connected to a master server at 127.15.25.126:39933
I20250622 13:58:05.184269 16677 heartbeater.cc:461] Registering TS with master...
I20250622 13:58:05.185506 16677 heartbeater.cc:507] Master 127.15.25.126:39933 requested a full tablet report, sending...
I20250622 13:58:05.187670 16355 ts_manager.cc:194] Registered new tserver with Master: aee733dd7e33474eac3bd30dab37cc4c (127.15.25.66:43459)
I20250622 13:58:05.188922 16355 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.25.66:58887
W20250622 13:58:05.448665 16681 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250622 13:58:05.449271 16681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:58:05.449529 16681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 13:58:05.449994 16681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:58:05.478040 16681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 13:58:05.478848 16681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.25.67
I20250622 13:58:05.510257 16681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:39511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.25.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.15.25.67
--webserver_port=0
--tserver_master_addrs=127.15.25.126:39933
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.25.67
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 13:58:05.511477 16681 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 13:58:05.513017 16681 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:58:05.527926 16687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:58:06.192615 16677 heartbeater.cc:499] Master 127.15.25.126:39933 was elected leader, sending a full tablet report...
W20250622 13:58:05.528076 16690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:58:05.528090 16688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:58:06.612200 16689 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250622 13:58:06.612356 16681 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 13:58:06.615929 16681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 13:58:06.618422 16681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 13:58:06.619834 16681 hybrid_clock.cc:648] HybridClock initialized: now 1750600686619799 us; error 56 us; skew 500 ppm
I20250622 13:58:06.620594 16681 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:58:06.630383 16681 webserver.cc:469] Webserver started at http://127.15.25.67:33791/ using document root <none> and password file <none>
I20250622 13:58:06.631177 16681 fs_manager.cc:362] Metadata directory not provided
I20250622 13:58:06.631345 16681 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:58:06.631701 16681 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:58:06.635690 16681 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "e31befb687f54204821f206e98c1caa0"
format_stamp: "Formatted at 2025-06-22 13:58:06 on dist-test-slave-6tph"
I20250622 13:58:06.636703 16681 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "e31befb687f54204821f206e98c1caa0"
format_stamp: "Formatted at 2025-06-22 13:58:06 on dist-test-slave-6tph"
I20250622 13:58:06.642997 16681 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.000s sys 0.008s
I20250622 13:58:06.647807 16697 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:06.648732 16681 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250622 13:58:06.649032 16681 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e31befb687f54204821f206e98c1caa0"
format_stamp: "Formatted at 2025-06-22 13:58:06 on dist-test-slave-6tph"
I20250622 13:58:06.649346 16681 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:58:06.700260 16681 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:58:06.701689 16681 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 13:58:06.702124 16681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:58:06.704427 16681 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:58:06.708377 16681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:58:06.708586 16681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:06.708837 16681 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:58:06.709002 16681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:06.836997 16681 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.67:32799
I20250622 13:58:06.837091 16809 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.67:32799 every 8 connection(s)
I20250622 13:58:06.839504 16681 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250622 13:58:06.840458 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 16681
I20250622 13:58:06.840817 15461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250622 13:58:06.858073 16810 heartbeater.cc:344] Connected to a master server at 127.15.25.126:39933
I20250622 13:58:06.858546 16810 heartbeater.cc:461] Registering TS with master...
I20250622 13:58:06.859570 16810 heartbeater.cc:507] Master 127.15.25.126:39933 requested a full tablet report, sending...
I20250622 13:58:06.861444 16355 ts_manager.cc:194] Registered new tserver with Master: e31befb687f54204821f206e98c1caa0 (127.15.25.67:32799)
I20250622 13:58:06.862612 16355 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.25.67:56633
I20250622 13:58:06.873222 15461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250622 13:58:06.905977 16355 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50574:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250622 13:58:06.924391 16355 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250622 13:58:06.971556 16745 tablet_service.cc:1468] Processing CreateTablet for tablet 388221e6c3d2406e85c6641537398ee0 (DEFAULT_TABLE table=TestTable [id=de9d4dee29e84c46b0706455e4713e54]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 13:58:06.972491 16479 tablet_service.cc:1468] Processing CreateTablet for tablet 388221e6c3d2406e85c6641537398ee0 (DEFAULT_TABLE table=TestTable [id=de9d4dee29e84c46b0706455e4713e54]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 13:58:06.973461 16745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 388221e6c3d2406e85c6641537398ee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:58:06.974426 16479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 388221e6c3d2406e85c6641537398ee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:58:06.976562 16612 tablet_service.cc:1468] Processing CreateTablet for tablet 388221e6c3d2406e85c6641537398ee0 (DEFAULT_TABLE table=TestTable [id=de9d4dee29e84c46b0706455e4713e54]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 13:58:06.978201 16612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 388221e6c3d2406e85c6641537398ee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:58:06.994510 16829 tablet_bootstrap.cc:492] T 388221e6c3d2406e85c6641537398ee0 P e31befb687f54204821f206e98c1caa0: Bootstrap starting.
I20250622 13:58:07.000524 16829 tablet_bootstrap.cc:654] T 388221e6c3d2406e85c6641537398ee0 P e31befb687f54204821f206e98c1caa0: Neither blocks nor log segments found. Creating new log.
I20250622 13:58:07.002738 16830 tablet_bootstrap.cc:492] T 388221e6c3d2406e85c6641537398ee0 P 3833939ed3d44b71813f154d2743ca34: Bootstrap starting.
I20250622 13:58:07.003540 16829 log.cc:826] T 388221e6c3d2406e85c6641537398ee0 P e31befb687f54204821f206e98c1caa0: Log is configured to *not* fsync() on all Append() calls
I20250622 13:58:07.005787 16831 tablet_bootstrap.cc:492] T 388221e6c3d2406e85c6641537398ee0 P aee733dd7e33474eac3bd30dab37cc4c: Bootstrap starting.
I20250622 13:58:07.011883 16831 tablet_bootstrap.cc:654] T 388221e6c3d2406e85c6641537398ee0 P aee733dd7e33474eac3bd30dab37cc4c: Neither blocks nor log segments found. Creating new log.
I20250622 13:58:07.011888 16830 tablet_bootstrap.cc:654] T 388221e6c3d2406e85c6641537398ee0 P 3833939ed3d44b71813f154d2743ca34: Neither blocks nor log segments found. Creating new log.
I20250622 13:58:07.014008 16831 log.cc:826] T 388221e6c3d2406e85c6641537398ee0 P aee733dd7e33474eac3bd30dab37cc4c: Log is configured to *not* fsync() on all Append() calls
I20250622 13:58:07.014050 16830 log.cc:826] T 388221e6c3d2406e85c6641537398ee0 P 3833939ed3d44b71813f154d2743ca34: Log is configured to *not* fsync() on all Append() calls
I20250622 13:58:07.015352 16829 tablet_bootstrap.cc:492] T 388221e6c3d2406e85c6641537398ee0 P e31befb687f54204821f206e98c1caa0: No bootstrap required, opened a new log
I20250622 13:58:07.016001 16829 ts_tablet_manager.cc:1397] T 388221e6c3d2406e85c6641537398ee0 P e31befb687f54204821f206e98c1caa0: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.011s
I20250622 13:58:07.019140 16830 tablet_bootstrap.cc:492] T 388221e6c3d2406e85c6641537398ee0 P 3833939ed3d44b71813f154d2743ca34: No bootstrap required, opened a new log
I20250622 13:58:07.019387 16831 tablet_bootstrap.cc:492] T 388221e6c3d2406e85c6641537398ee0 P aee733dd7e33474eac3bd30dab37cc4c: No bootstrap required, opened a new log
I20250622 13:58:07.019547 16830 ts_tablet_manager.cc:1397] T 388221e6c3d2406e85c6641537398ee0 P 3833939ed3d44b71813f154d2743ca34: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.004s
I20250622 13:58:07.019865 16831 ts_tablet_manager.cc:1397] T 388221e6c3d2406e85c6641537398ee0 P aee733dd7e33474eac3bd30dab37cc4c: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.011s
I20250622 13:58:07.038672 16830 raft_consensus.cc:357] T 388221e6c3d2406e85c6641537398ee0 P 3833939ed3d44b71813f154d2743ca34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31befb687f54204821f206e98c1caa0" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 32799 } } peers { permanent_uuid: "3833939ed3d44b71813f154d2743ca34" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 43999 } } peers { permanent_uuid: "aee733dd7e33474eac3bd30dab37cc4c" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 43459 } }
I20250622 13:58:07.039376 16830 raft_consensus.cc:738] T 388221e6c3d2406e85c6641537398ee0 P 3833939ed3d44b71813f154d2743ca34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3833939ed3d44b71813f154d2743ca34, State: Initialized, Role: FOLLOWER
I20250622 13:58:07.039996 16830 consensus_queue.cc:260] T 388221e6c3d2406e85c6641537398ee0 P 3833939ed3d44b71813f154d2743ca34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31befb687f54204821f206e98c1caa0" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 32799 } } peers { permanent_uuid: "3833939ed3d44b71813f154d2743ca34" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 43999 } } peers { permanent_uuid: "aee733dd7e33474eac3bd30dab37cc4c" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 43459 } }
I20250622 13:58:07.042675 16829 raft_consensus.cc:357] T 388221e6c3d2406e85c6641537398ee0 P e31befb687f54204821f206e98c1caa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31befb687f54204821f206e98c1caa0" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 32799 } } peers { permanent_uuid: "3833939ed3d44b71813f154d2743ca34" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 43999 } } peers { permanent_uuid: "aee733dd7e33474eac3bd30dab37cc4c" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 43459 } }
I20250622 13:58:07.043529 16829 raft_consensus.cc:738] T 388221e6c3d2406e85c6641537398ee0 P e31befb687f54204821f206e98c1caa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e31befb687f54204821f206e98c1caa0, State: Initialized, Role: FOLLOWER
I20250622 13:58:07.044207 16829 consensus_queue.cc:260] T 388221e6c3d2406e85c6641537398ee0 P e31befb687f54204821f206e98c1caa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31befb687f54204821f206e98c1caa0" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 32799 } } peers { permanent_uuid: "3833939ed3d44b71813f154d2743ca34" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 43999 } } peers { permanent_uuid: "aee733dd7e33474eac3bd30dab37cc4c" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 43459 } }
I20250622 13:58:07.045677 16831 raft_consensus.cc:357] T 388221e6c3d2406e85c6641537398ee0 P aee733dd7e33474eac3bd30dab37cc4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31befb687f54204821f206e98c1caa0" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 32799 } } peers { permanent_uuid: "3833939ed3d44b71813f154d2743ca34" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 43999 } } peers { permanent_uuid: "aee733dd7e33474eac3bd30dab37cc4c" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 43459 } }
I20250622 13:58:07.046720 16831 raft_consensus.cc:738] T 388221e6c3d2406e85c6641537398ee0 P aee733dd7e33474eac3bd30dab37cc4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aee733dd7e33474eac3bd30dab37cc4c, State: Initialized, Role: FOLLOWER
I20250622 13:58:07.047534 16831 consensus_queue.cc:260] T 388221e6c3d2406e85c6641537398ee0 P aee733dd7e33474eac3bd30dab37cc4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31befb687f54204821f206e98c1caa0" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 32799 } } peers { permanent_uuid: "3833939ed3d44b71813f154d2743ca34" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 43999 } } peers { permanent_uuid: "aee733dd7e33474eac3bd30dab37cc4c" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 43459 } }
I20250622 13:58:07.049384 16830 ts_tablet_manager.cc:1428] T 388221e6c3d2406e85c6641537398ee0 P 3833939ed3d44b71813f154d2743ca34: Time spent starting tablet: real 0.030s user 0.018s sys 0.011s
I20250622 13:58:07.050763 16810 heartbeater.cc:499] Master 127.15.25.126:39933 was elected leader, sending a full tablet report...
I20250622 13:58:07.052135 16829 ts_tablet_manager.cc:1428] T 388221e6c3d2406e85c6641537398ee0 P e31befb687f54204821f206e98c1caa0: Time spent starting tablet: real 0.036s user 0.028s sys 0.004s
I20250622 13:58:07.054920 16831 ts_tablet_manager.cc:1428] T 388221e6c3d2406e85c6641537398ee0 P aee733dd7e33474eac3bd30dab37cc4c: Time spent starting tablet: real 0.035s user 0.032s sys 0.001s
I20250622 13:58:07.082433 15461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250622 13:58:07.085243 15461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 3833939ed3d44b71813f154d2743ca34 to finish bootstrapping
W20250622 13:58:07.094381 16811 tablet.cc:2378] T 388221e6c3d2406e85c6641537398ee0 P e31befb687f54204821f206e98c1caa0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 13:58:07.097226 15461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver aee733dd7e33474eac3bd30dab37cc4c to finish bootstrapping
I20250622 13:58:07.110471 15461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver e31befb687f54204821f206e98c1caa0 to finish bootstrapping
I20250622 13:58:07.153697 16499 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "388221e6c3d2406e85c6641537398ee0"
dest_uuid: "3833939ed3d44b71813f154d2743ca34"
from {username='slave'} at 127.0.0.1:45678
I20250622 13:58:07.154326 16499 raft_consensus.cc:491] T 388221e6c3d2406e85c6641537398ee0 P 3833939ed3d44b71813f154d2743ca34 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 13:58:07.154592 16499 raft_consensus.cc:3058] T 388221e6c3d2406e85c6641537398ee0 P 3833939ed3d44b71813f154d2743ca34 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:58:07.158784 16499 raft_consensus.cc:513] T 388221e6c3d2406e85c6641537398ee0 P 3833939ed3d44b71813f154d2743ca34 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31befb687f54204821f206e98c1caa0" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 32799 } } peers { permanent_uuid: "3833939ed3d44b71813f154d2743ca34" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 43999 } } peers { permanent_uuid: "aee733dd7e33474eac3bd30dab37cc4c" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 43459 } }
I20250622 13:58:07.161712 16499 leader_election.cc:290] T 388221e6c3d2406e85c6641537398ee0 P 3833939ed3d44b71813f154d2743ca34 [CANDIDATE]: Term 1 election: Requested vote from peers e31befb687f54204821f206e98c1caa0 (127.15.25.67:32799), aee733dd7e33474eac3bd30dab37cc4c (127.15.25.66:43459)
W20250622 13:58:07.169859 16678 tablet.cc:2378] T 388221e6c3d2406e85c6641537398ee0 P aee733dd7e33474eac3bd30dab37cc4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 13:58:07.171171 15461 cluster_itest_util.cc:257] Not converged past 1 yet: 0.0 0.0 0.0
I20250622 13:58:07.174101 16765 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "388221e6c3d2406e85c6641537398ee0" candidate_uuid: "3833939ed3d44b71813f154d2743ca34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e31befb687f54204821f206e98c1caa0"
I20250622 13:58:07.174623 16765 raft_consensus.cc:3058] T 388221e6c3d2406e85c6641537398ee0 P e31befb687f54204821f206e98c1caa0 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:58:07.178460 16632 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "388221e6c3d2406e85c6641537398ee0" candidate_uuid: "3833939ed3d44b71813f154d2743ca34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "aee733dd7e33474eac3bd30dab37cc4c"
I20250622 13:58:07.179203 16632 raft_consensus.cc:3058] T 388221e6c3d2406e85c6641537398ee0 P aee733dd7e33474eac3bd30dab37cc4c [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:58:07.180486 16765 raft_consensus.cc:2466] T 388221e6c3d2406e85c6641537398ee0 P e31befb687f54204821f206e98c1caa0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3833939ed3d44b71813f154d2743ca34 in term 1.
I20250622 13:58:07.181775 16435 leader_election.cc:304] T 388221e6c3d2406e85c6641537398ee0 P 3833939ed3d44b71813f154d2743ca34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3833939ed3d44b71813f154d2743ca34, e31befb687f54204821f206e98c1caa0; no voters:
I20250622 13:58:07.182739 16835 raft_consensus.cc:2802] T 388221e6c3d2406e85c6641537398ee0 P 3833939ed3d44b71813f154d2743ca34 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:58:07.184535 16835 raft_consensus.cc:695] T 388221e6c3d2406e85c6641537398ee0 P 3833939ed3d44b71813f154d2743ca34 [term 1 LEADER]: Becoming Leader. State: Replica: 3833939ed3d44b71813f154d2743ca34, State: Running, Role: LEADER
I20250622 13:58:07.185062 16632 raft_consensus.cc:2466] T 388221e6c3d2406e85c6641537398ee0 P aee733dd7e33474eac3bd30dab37cc4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3833939ed3d44b71813f154d2743ca34 in term 1.
I20250622 13:58:07.185366 16835 consensus_queue.cc:237] T 388221e6c3d2406e85c6641537398ee0 P 3833939ed3d44b71813f154d2743ca34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31befb687f54204821f206e98c1caa0" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 32799 } } peers { permanent_uuid: "3833939ed3d44b71813f154d2743ca34" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 43999 } } peers { permanent_uuid: "aee733dd7e33474eac3bd30dab37cc4c" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 43459 } }
I20250622 13:58:07.196820 16353 catalog_manager.cc:5582] T 388221e6c3d2406e85c6641537398ee0 P 3833939ed3d44b71813f154d2743ca34 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3833939ed3d44b71813f154d2743ca34 (127.15.25.65). New cstate: current_term: 1 leader_uuid: "3833939ed3d44b71813f154d2743ca34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31befb687f54204821f206e98c1caa0" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 32799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3833939ed3d44b71813f154d2743ca34" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 43999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aee733dd7e33474eac3bd30dab37cc4c" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 43459 } health_report { overall_health: UNKNOWN } } }
W20250622 13:58:07.248769 16545 tablet.cc:2378] T 388221e6c3d2406e85c6641537398ee0 P 3833939ed3d44b71813f154d2743ca34: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 13:58:07.276096 15461 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250622 13:58:07.481192 15461 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250622 13:58:07.638768 16835 consensus_queue.cc:1035] T 388221e6c3d2406e85c6641537398ee0 P 3833939ed3d44b71813f154d2743ca34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e31befb687f54204821f206e98c1caa0" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 32799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:58:07.654605 16842 consensus_queue.cc:1035] T 388221e6c3d2406e85c6641537398ee0 P 3833939ed3d44b71813f154d2743ca34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aee733dd7e33474eac3bd30dab37cc4c" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 43459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:58:09.345685 16499 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "388221e6c3d2406e85c6641537398ee0"
dest_uuid: "3833939ed3d44b71813f154d2743ca34"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45684
I20250622 13:58:09.346170 16499 raft_consensus.cc:604] T 388221e6c3d2406e85c6641537398ee0 P 3833939ed3d44b71813f154d2743ca34 [term 1 LEADER]: Received request to transfer leadership
I20250622 13:58:09.402266 16873 raft_consensus.cc:991] T 388221e6c3d2406e85c6641537398ee0 P 3833939ed3d44b71813f154d2743ca34: : Instructing follower e31befb687f54204821f206e98c1caa0 to start an election
I20250622 13:58:09.402631 16873 raft_consensus.cc:1079] T 388221e6c3d2406e85c6641537398ee0 P 3833939ed3d44b71813f154d2743ca34 [term 1 LEADER]: Signalling peer e31befb687f54204821f206e98c1caa0 to start an election
I20250622 13:58:09.403808 16765 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "388221e6c3d2406e85c6641537398ee0"
dest_uuid: "e31befb687f54204821f206e98c1caa0"
from {username='slave'} at 127.15.25.65:42033
I20250622 13:58:09.404271 16765 raft_consensus.cc:491] T 388221e6c3d2406e85c6641537398ee0 P e31befb687f54204821f206e98c1caa0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 13:58:09.404512 16765 raft_consensus.cc:3058] T 388221e6c3d2406e85c6641537398ee0 P e31befb687f54204821f206e98c1caa0 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:58:09.408394 16765 raft_consensus.cc:513] T 388221e6c3d2406e85c6641537398ee0 P e31befb687f54204821f206e98c1caa0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31befb687f54204821f206e98c1caa0" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 32799 } } peers { permanent_uuid: "3833939ed3d44b71813f154d2743ca34" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 43999 } } peers { permanent_uuid: "aee733dd7e33474eac3bd30dab37cc4c" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 43459 } }
I20250622 13:58:09.410413 16765 leader_election.cc:290] T 388221e6c3d2406e85c6641537398ee0 P e31befb687f54204821f206e98c1caa0 [CANDIDATE]: Term 2 election: Requested vote from peers 3833939ed3d44b71813f154d2743ca34 (127.15.25.65:43999), aee733dd7e33474eac3bd30dab37cc4c (127.15.25.66:43459)
I20250622 13:58:09.420449 16499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "388221e6c3d2406e85c6641537398ee0" candidate_uuid: "e31befb687f54204821f206e98c1caa0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3833939ed3d44b71813f154d2743ca34"
I20250622 13:58:09.420957 16499 raft_consensus.cc:3053] T 388221e6c3d2406e85c6641537398ee0 P 3833939ed3d44b71813f154d2743ca34 [term 1 LEADER]: Stepping down as leader of term 1
I20250622 13:58:09.421200 16499 raft_consensus.cc:738] T 388221e6c3d2406e85c6641537398ee0 P 3833939ed3d44b71813f154d2743ca34 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3833939ed3d44b71813f154d2743ca34, State: Running, Role: LEADER
I20250622 13:58:09.421898 16632 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "388221e6c3d2406e85c6641537398ee0" candidate_uuid: "e31befb687f54204821f206e98c1caa0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "aee733dd7e33474eac3bd30dab37cc4c"
I20250622 13:58:09.421829 16499 consensus_queue.cc:260] T 388221e6c3d2406e85c6641537398ee0 P 3833939ed3d44b71813f154d2743ca34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31befb687f54204821f206e98c1caa0" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 32799 } } peers { permanent_uuid: "3833939ed3d44b71813f154d2743ca34" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 43999 } } peers { permanent_uuid: "aee733dd7e33474eac3bd30dab37cc4c" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 43459 } }
I20250622 13:58:09.422302 16632 raft_consensus.cc:3058] T 388221e6c3d2406e85c6641537398ee0 P aee733dd7e33474eac3bd30dab37cc4c [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:58:09.422729 16499 raft_consensus.cc:3058] T 388221e6c3d2406e85c6641537398ee0 P 3833939ed3d44b71813f154d2743ca34 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:58:09.425827 16632 raft_consensus.cc:2466] T 388221e6c3d2406e85c6641537398ee0 P aee733dd7e33474eac3bd30dab37cc4c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e31befb687f54204821f206e98c1caa0 in term 2.
I20250622 13:58:09.426182 16499 raft_consensus.cc:2466] T 388221e6c3d2406e85c6641537398ee0 P 3833939ed3d44b71813f154d2743ca34 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e31befb687f54204821f206e98c1caa0 in term 2.
I20250622 13:58:09.426726 16701 leader_election.cc:304] T 388221e6c3d2406e85c6641537398ee0 P e31befb687f54204821f206e98c1caa0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aee733dd7e33474eac3bd30dab37cc4c, e31befb687f54204821f206e98c1caa0; no voters:
I20250622 13:58:09.428408 16877 raft_consensus.cc:2802] T 388221e6c3d2406e85c6641537398ee0 P e31befb687f54204821f206e98c1caa0 [term 2 FOLLOWER]: Leader election won for term 2
I20250622 13:58:09.429611 16877 raft_consensus.cc:695] T 388221e6c3d2406e85c6641537398ee0 P e31befb687f54204821f206e98c1caa0 [term 2 LEADER]: Becoming Leader. State: Replica: e31befb687f54204821f206e98c1caa0, State: Running, Role: LEADER
I20250622 13:58:09.430287 16877 consensus_queue.cc:237] T 388221e6c3d2406e85c6641537398ee0 P e31befb687f54204821f206e98c1caa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31befb687f54204821f206e98c1caa0" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 32799 } } peers { permanent_uuid: "3833939ed3d44b71813f154d2743ca34" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 43999 } } peers { permanent_uuid: "aee733dd7e33474eac3bd30dab37cc4c" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 43459 } }
I20250622 13:58:09.436806 16355 catalog_manager.cc:5582] T 388221e6c3d2406e85c6641537398ee0 P e31befb687f54204821f206e98c1caa0 reported cstate change: term changed from 1 to 2, leader changed from 3833939ed3d44b71813f154d2743ca34 (127.15.25.65) to e31befb687f54204821f206e98c1caa0 (127.15.25.67). New cstate: current_term: 2 leader_uuid: "e31befb687f54204821f206e98c1caa0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31befb687f54204821f206e98c1caa0" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 32799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3833939ed3d44b71813f154d2743ca34" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 43999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aee733dd7e33474eac3bd30dab37cc4c" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 43459 } health_report { overall_health: UNKNOWN } } }
I20250622 13:58:09.992746 16632 raft_consensus.cc:1273] T 388221e6c3d2406e85c6641537398ee0 P aee733dd7e33474eac3bd30dab37cc4c [term 2 FOLLOWER]: Refusing update from remote peer e31befb687f54204821f206e98c1caa0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250622 13:58:09.993892 16884 consensus_queue.cc:1035] T 388221e6c3d2406e85c6641537398ee0 P e31befb687f54204821f206e98c1caa0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aee733dd7e33474eac3bd30dab37cc4c" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 43459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250622 13:58:10.004874 16499 raft_consensus.cc:1273] T 388221e6c3d2406e85c6641537398ee0 P 3833939ed3d44b71813f154d2743ca34 [term 2 FOLLOWER]: Refusing update from remote peer e31befb687f54204821f206e98c1caa0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250622 13:58:10.006299 16884 consensus_queue.cc:1035] T 388221e6c3d2406e85c6641537398ee0 P e31befb687f54204821f206e98c1caa0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3833939ed3d44b71813f154d2743ca34" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 43999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250622 13:58:11.961392 16499 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "388221e6c3d2406e85c6641537398ee0"
dest_uuid: "3833939ed3d44b71813f154d2743ca34"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49148
I20250622 13:58:11.962148 16499 raft_consensus.cc:604] T 388221e6c3d2406e85c6641537398ee0 P 3833939ed3d44b71813f154d2743ca34 [term 2 FOLLOWER]: Received request to transfer leadership
I20250622 13:58:11.962499 16499 raft_consensus.cc:612] T 388221e6c3d2406e85c6641537398ee0 P 3833939ed3d44b71813f154d2743ca34 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250622 13:58:12.997530 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 16414
I20250622 13:58:13.021574 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 16548
I20250622 13:58:13.047010 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 16681
W20250622 13:58:13.047015 16340 connection.cc:537] server connection from 127.15.25.66:58887 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250622 13:58:13.070983 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 16322
2025-06-22T13:58:13Z chronyd exiting
[ OK ] AdminCliTest.TestGracefulSpecificLeaderStepDown (13230 ms)
[ RUN ] AdminCliTest.TestDescribeTableColumnFlags
I20250622 13:58:13.123673 15461 test_util.cc:276] Using random seed: 2088045120
I20250622 13:58:13.127552 15461 ts_itest-base.cc:115] Starting cluster with:
I20250622 13:58:13.127712 15461 ts_itest-base.cc:116] --------------
I20250622 13:58:13.127840 15461 ts_itest-base.cc:117] 3 tablet servers
I20250622 13:58:13.127955 15461 ts_itest-base.cc:118] 3 replicas per TS
I20250622 13:58:13.128059 15461 ts_itest-base.cc:119] --------------
2025-06-22T13:58:13Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-22T13:58:13Z Disabled control of system clock
I20250622 13:58:13.161554 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.25.126:40051
--webserver_interface=127.15.25.126
--webserver_port=0
--builtin_ntp_servers=127.15.25.84:36469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.25.126:40051 with env {}
W20250622 13:58:13.435920 16925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:58:13.436422 16925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 13:58:13.436828 16925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:58:13.464838 16925 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250622 13:58:13.465081 16925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 13:58:13.465250 16925 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250622 13:58:13.465427 16925 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250622 13:58:13.496362 16925 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:36469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.25.126:40051
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.25.126:40051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.25.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 13:58:13.497635 16925 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 13:58:13.499229 16925 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:58:13.511312 16932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:58:13.513583 16934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:58:13.511399 16931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:58:14.615324 16933 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1102 milliseconds
I20250622 13:58:14.615437 16925 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 13:58:14.616662 16925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 13:58:14.619143 16925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 13:58:14.620496 16925 hybrid_clock.cc:648] HybridClock initialized: now 1750600694620443 us; error 81 us; skew 500 ppm
I20250622 13:58:14.621249 16925 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:58:14.626999 16925 webserver.cc:469] Webserver started at http://127.15.25.126:34261/ using document root <none> and password file <none>
I20250622 13:58:14.627897 16925 fs_manager.cc:362] Metadata directory not provided
I20250622 13:58:14.628095 16925 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:58:14.628496 16925 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:58:14.632279 16925 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "e68ebae275e7475f859ecc27dfaa4fc9"
format_stamp: "Formatted at 2025-06-22 13:58:14 on dist-test-slave-6tph"
I20250622 13:58:14.633214 16925 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "e68ebae275e7475f859ecc27dfaa4fc9"
format_stamp: "Formatted at 2025-06-22 13:58:14 on dist-test-slave-6tph"
I20250622 13:58:14.639577 16925 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20250622 13:58:14.644513 16941 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:14.645364 16925 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250622 13:58:14.645637 16925 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
uuid: "e68ebae275e7475f859ecc27dfaa4fc9"
format_stamp: "Formatted at 2025-06-22 13:58:14 on dist-test-slave-6tph"
I20250622 13:58:14.645956 16925 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:58:14.691555 16925 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:58:14.692885 16925 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 13:58:14.693300 16925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:58:14.757362 16925 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.126:40051
I20250622 13:58:14.757440 16992 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.126:40051 every 8 connection(s)
I20250622 13:58:14.759896 16925 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250622 13:58:14.762651 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 16925
I20250622 13:58:14.763105 15461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250622 13:58:14.765645 16993 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:58:14.793594 16993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e68ebae275e7475f859ecc27dfaa4fc9: Bootstrap starting.
I20250622 13:58:14.799693 16993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e68ebae275e7475f859ecc27dfaa4fc9: Neither blocks nor log segments found. Creating new log.
I20250622 13:58:14.801229 16993 log.cc:826] T 00000000000000000000000000000000 P e68ebae275e7475f859ecc27dfaa4fc9: Log is configured to *not* fsync() on all Append() calls
I20250622 13:58:14.805680 16993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e68ebae275e7475f859ecc27dfaa4fc9: No bootstrap required, opened a new log
I20250622 13:58:14.822173 16993 raft_consensus.cc:357] T 00000000000000000000000000000000 P e68ebae275e7475f859ecc27dfaa4fc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68ebae275e7475f859ecc27dfaa4fc9" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 40051 } }
I20250622 13:58:14.822764 16993 raft_consensus.cc:383] T 00000000000000000000000000000000 P e68ebae275e7475f859ecc27dfaa4fc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:58:14.822981 16993 raft_consensus.cc:738] T 00000000000000000000000000000000 P e68ebae275e7475f859ecc27dfaa4fc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e68ebae275e7475f859ecc27dfaa4fc9, State: Initialized, Role: FOLLOWER
I20250622 13:58:14.823580 16993 consensus_queue.cc:260] T 00000000000000000000000000000000 P e68ebae275e7475f859ecc27dfaa4fc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68ebae275e7475f859ecc27dfaa4fc9" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 40051 } }
I20250622 13:58:14.824074 16993 raft_consensus.cc:397] T 00000000000000000000000000000000 P e68ebae275e7475f859ecc27dfaa4fc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 13:58:14.824330 16993 raft_consensus.cc:491] T 00000000000000000000000000000000 P e68ebae275e7475f859ecc27dfaa4fc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 13:58:14.824666 16993 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e68ebae275e7475f859ecc27dfaa4fc9 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:58:14.828585 16993 raft_consensus.cc:513] T 00000000000000000000000000000000 P e68ebae275e7475f859ecc27dfaa4fc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68ebae275e7475f859ecc27dfaa4fc9" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 40051 } }
I20250622 13:58:14.829206 16993 leader_election.cc:304] T 00000000000000000000000000000000 P e68ebae275e7475f859ecc27dfaa4fc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e68ebae275e7475f859ecc27dfaa4fc9; no voters:
I20250622 13:58:14.830695 16993 leader_election.cc:290] T 00000000000000000000000000000000 P e68ebae275e7475f859ecc27dfaa4fc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 13:58:14.831457 16998 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e68ebae275e7475f859ecc27dfaa4fc9 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:58:14.833215 16998 raft_consensus.cc:695] T 00000000000000000000000000000000 P e68ebae275e7475f859ecc27dfaa4fc9 [term 1 LEADER]: Becoming Leader. State: Replica: e68ebae275e7475f859ecc27dfaa4fc9, State: Running, Role: LEADER
I20250622 13:58:14.834007 16998 consensus_queue.cc:237] T 00000000000000000000000000000000 P e68ebae275e7475f859ecc27dfaa4fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68ebae275e7475f859ecc27dfaa4fc9" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 40051 } }
I20250622 13:58:14.834347 16993 sys_catalog.cc:564] T 00000000000000000000000000000000 P e68ebae275e7475f859ecc27dfaa4fc9 [sys.catalog]: configured and running, proceeding with master startup.
I20250622 13:58:14.844812 16999 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68ebae275e7475f859ecc27dfaa4fc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e68ebae275e7475f859ecc27dfaa4fc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68ebae275e7475f859ecc27dfaa4fc9" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 40051 } } }
I20250622 13:58:14.845597 16999 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68ebae275e7475f859ecc27dfaa4fc9 [sys.catalog]: This master's current role is: LEADER
I20250622 13:58:14.846082 17000 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68ebae275e7475f859ecc27dfaa4fc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e68ebae275e7475f859ecc27dfaa4fc9. Latest consensus state: current_term: 1 leader_uuid: "e68ebae275e7475f859ecc27dfaa4fc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68ebae275e7475f859ecc27dfaa4fc9" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 40051 } } }
I20250622 13:58:14.846873 17000 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68ebae275e7475f859ecc27dfaa4fc9 [sys.catalog]: This master's current role is: LEADER
I20250622 13:58:14.848572 17008 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 13:58:14.858605 17008 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 13:58:14.872015 17008 catalog_manager.cc:1349] Generated new cluster ID: 1f735d8f2fba4b2bb23924fa6ebddf79
I20250622 13:58:14.872295 17008 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 13:58:14.891634 17008 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 13:58:14.892966 17008 catalog_manager.cc:1506] Loading token signing keys...
I20250622 13:58:14.916289 17008 catalog_manager.cc:5955] T 00000000000000000000000000000000 P e68ebae275e7475f859ecc27dfaa4fc9: Generated new TSK 0
I20250622 13:58:14.917145 17008 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 13:58:14.930382 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.25.65:0
--local_ip_for_outbound_sockets=127.15.25.65
--webserver_interface=127.15.25.65
--webserver_port=0
--tserver_master_addrs=127.15.25.126:40051
--builtin_ntp_servers=127.15.25.84:36469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250622 13:58:15.223191 17017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:58:15.223683 17017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 13:58:15.224161 17017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:58:15.254703 17017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 13:58:15.255517 17017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.25.65
I20250622 13:58:15.288326 17017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:36469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.25.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.15.25.65
--webserver_port=0
--tserver_master_addrs=127.15.25.126:40051
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.25.65
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 13:58:15.289628 17017 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 13:58:15.291142 17017 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:58:15.307890 17024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:58:15.310376 17017 server_base.cc:1048] running on GCE node
W20250622 13:58:15.309065 17026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:58:15.308143 17023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:58:16.426196 17017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 13:58:16.428853 17017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 13:58:16.430240 17017 hybrid_clock.cc:648] HybridClock initialized: now 1750600696430187 us; error 72 us; skew 500 ppm
I20250622 13:58:16.430971 17017 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:58:16.437279 17017 webserver.cc:469] Webserver started at http://127.15.25.65:33627/ using document root <none> and password file <none>
I20250622 13:58:16.438189 17017 fs_manager.cc:362] Metadata directory not provided
I20250622 13:58:16.438383 17017 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:58:16.438853 17017 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:58:16.443187 17017 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "34d8d90a37d94f3299d957f2ac1ef076"
format_stamp: "Formatted at 2025-06-22 13:58:16 on dist-test-slave-6tph"
I20250622 13:58:16.444268 17017 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "34d8d90a37d94f3299d957f2ac1ef076"
format_stamp: "Formatted at 2025-06-22 13:58:16 on dist-test-slave-6tph"
I20250622 13:58:16.451614 17017 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20250622 13:58:16.457545 17033 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:16.458693 17017 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250622 13:58:16.458997 17017 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "34d8d90a37d94f3299d957f2ac1ef076"
format_stamp: "Formatted at 2025-06-22 13:58:16 on dist-test-slave-6tph"
I20250622 13:58:16.459311 17017 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:58:16.512145 17017 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:58:16.513569 17017 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 13:58:16.514008 17017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:58:16.516402 17017 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:58:16.520347 17017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:58:16.520568 17017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:16.520803 17017 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:58:16.520956 17017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:16.642308 17017 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.65:44117
I20250622 13:58:16.642397 17145 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.65:44117 every 8 connection(s)
I20250622 13:58:16.644857 17017 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250622 13:58:16.648113 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 17017
I20250622 13:58:16.648495 15461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250622 13:58:16.655201 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.25.66:0
--local_ip_for_outbound_sockets=127.15.25.66
--webserver_interface=127.15.25.66
--webserver_port=0
--tserver_master_addrs=127.15.25.126:40051
--builtin_ntp_servers=127.15.25.84:36469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 13:58:16.665668 17146 heartbeater.cc:344] Connected to a master server at 127.15.25.126:40051
I20250622 13:58:16.666173 17146 heartbeater.cc:461] Registering TS with master...
I20250622 13:58:16.667484 17146 heartbeater.cc:507] Master 127.15.25.126:40051 requested a full tablet report, sending...
I20250622 13:58:16.669909 16958 ts_manager.cc:194] Registered new tserver with Master: 34d8d90a37d94f3299d957f2ac1ef076 (127.15.25.65:44117)
I20250622 13:58:16.671612 16958 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.25.65:56679
W20250622 13:58:16.948146 17150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:58:16.948678 17150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 13:58:16.949234 17150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:58:16.981113 17150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 13:58:16.982009 17150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.25.66
I20250622 13:58:17.016160 17150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:36469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.25.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.15.25.66
--webserver_port=0
--tserver_master_addrs=127.15.25.126:40051
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.25.66
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 13:58:17.017482 17150 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 13:58:17.019203 17150 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:58:17.034461 17157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:58:17.675343 17146 heartbeater.cc:499] Master 127.15.25.126:40051 was elected leader, sending a full tablet report...
W20250622 13:58:17.034530 17156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:58:17.036458 17150 server_base.cc:1048] running on GCE node
W20250622 13:58:17.034570 17159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:58:18.115617 17150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 13:58:18.118323 17150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 13:58:18.119727 17150 hybrid_clock.cc:648] HybridClock initialized: now 1750600698119699 us; error 49 us; skew 500 ppm
I20250622 13:58:18.120491 17150 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:58:18.126492 17150 webserver.cc:469] Webserver started at http://127.15.25.66:36427/ using document root <none> and password file <none>
I20250622 13:58:18.127370 17150 fs_manager.cc:362] Metadata directory not provided
I20250622 13:58:18.127542 17150 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:58:18.128000 17150 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:58:18.132126 17150 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "d50f6ac76fe14fb3b4345735b6281e4a"
format_stamp: "Formatted at 2025-06-22 13:58:18 on dist-test-slave-6tph"
I20250622 13:58:18.133076 17150 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "d50f6ac76fe14fb3b4345735b6281e4a"
format_stamp: "Formatted at 2025-06-22 13:58:18 on dist-test-slave-6tph"
I20250622 13:58:18.139637 17150 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20250622 13:58:18.144428 17166 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:18.145321 17150 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250622 13:58:18.145583 17150 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d50f6ac76fe14fb3b4345735b6281e4a"
format_stamp: "Formatted at 2025-06-22 13:58:18 on dist-test-slave-6tph"
I20250622 13:58:18.145939 17150 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:58:18.200559 17150 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:58:18.202018 17150 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 13:58:18.202443 17150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:58:18.204939 17150 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:58:18.208892 17150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:58:18.209085 17150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:18.209300 17150 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:58:18.209442 17150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:18.332752 17150 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.66:37901
I20250622 13:58:18.332849 17278 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.66:37901 every 8 connection(s)
I20250622 13:58:18.335112 17150 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250622 13:58:18.343712 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 17150
I20250622 13:58:18.344059 15461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250622 13:58:18.350204 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.25.67:0
--local_ip_for_outbound_sockets=127.15.25.67
--webserver_interface=127.15.25.67
--webserver_port=0
--tserver_master_addrs=127.15.25.126:40051
--builtin_ntp_servers=127.15.25.84:36469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 13:58:18.354418 17279 heartbeater.cc:344] Connected to a master server at 127.15.25.126:40051
I20250622 13:58:18.354776 17279 heartbeater.cc:461] Registering TS with master...
I20250622 13:58:18.355676 17279 heartbeater.cc:507] Master 127.15.25.126:40051 requested a full tablet report, sending...
I20250622 13:58:18.357661 16958 ts_manager.cc:194] Registered new tserver with Master: d50f6ac76fe14fb3b4345735b6281e4a (127.15.25.66:37901)
I20250622 13:58:18.358866 16958 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.25.66:58679
W20250622 13:58:18.631446 17283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:58:18.631915 17283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 13:58:18.632349 17283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:58:18.661715 17283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 13:58:18.662492 17283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.25.67
I20250622 13:58:18.694422 17283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:36469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.25.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.15.25.67
--webserver_port=0
--tserver_master_addrs=127.15.25.126:40051
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.25.67
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 13:58:18.695719 17283 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 13:58:18.697417 17283 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:58:18.712004 17290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:58:19.361588 17279 heartbeater.cc:499] Master 127.15.25.126:40051 was elected leader, sending a full tablet report...
W20250622 13:58:18.712231 17289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:58:18.713999 17292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:58:19.800413 17291 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250622 13:58:19.800462 17283 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 13:58:19.803690 17283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 13:58:19.805665 17283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 13:58:19.807044 17283 hybrid_clock.cc:648] HybridClock initialized: now 1750600699807004 us; error 54 us; skew 500 ppm
I20250622 13:58:19.807848 17283 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:58:19.813428 17283 webserver.cc:469] Webserver started at http://127.15.25.67:39007/ using document root <none> and password file <none>
I20250622 13:58:19.814262 17283 fs_manager.cc:362] Metadata directory not provided
I20250622 13:58:19.814450 17283 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:58:19.814857 17283 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:58:19.819129 17283 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "ea9534d24449465f8cdd99ec57b54d34"
format_stamp: "Formatted at 2025-06-22 13:58:19 on dist-test-slave-6tph"
I20250622 13:58:19.820065 17283 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "ea9534d24449465f8cdd99ec57b54d34"
format_stamp: "Formatted at 2025-06-22 13:58:19 on dist-test-slave-6tph"
I20250622 13:58:19.826480 17283 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20250622 13:58:19.831547 17299 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:19.832428 17283 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20250622 13:58:19.832702 17283 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ea9534d24449465f8cdd99ec57b54d34"
format_stamp: "Formatted at 2025-06-22 13:58:19 on dist-test-slave-6tph"
I20250622 13:58:19.832998 17283 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:58:19.881955 17283 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:58:19.883273 17283 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 13:58:19.883672 17283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:58:19.885970 17283 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:58:19.889575 17283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:58:19.889779 17283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250622 13:58:19.890002 17283 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:58:19.890147 17283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:20.013264 17283 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.67:42935
I20250622 13:58:20.013353 17411 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.67:42935 every 8 connection(s)
I20250622 13:58:20.015472 17283 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250622 13:58:20.025360 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 17283
I20250622 13:58:20.025808 15461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250622 13:58:20.034579 17412 heartbeater.cc:344] Connected to a master server at 127.15.25.126:40051
I20250622 13:58:20.034942 17412 heartbeater.cc:461] Registering TS with master...
I20250622 13:58:20.035852 17412 heartbeater.cc:507] Master 127.15.25.126:40051 requested a full tablet report, sending...
I20250622 13:58:20.037690 16958 ts_manager.cc:194] Registered new tserver with Master: ea9534d24449465f8cdd99ec57b54d34 (127.15.25.67:42935)
I20250622 13:58:20.038909 16958 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.25.67:43859
I20250622 13:58:20.044143 15461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250622 13:58:20.069705 16958 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35238:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250622 13:58:20.086499 16958 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250622 13:58:20.129805 17347 tablet_service.cc:1468] Processing CreateTablet for tablet a847dfbf80594e97a21b5985d4686b55 (DEFAULT_TABLE table=TestTable [id=9af10dd67a814da89e1424fa6aa8a326]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 13:58:20.131546 17347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a847dfbf80594e97a21b5985d4686b55. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:58:20.132241 17081 tablet_service.cc:1468] Processing CreateTablet for tablet a847dfbf80594e97a21b5985d4686b55 (DEFAULT_TABLE table=TestTable [id=9af10dd67a814da89e1424fa6aa8a326]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 13:58:20.133898 17081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a847dfbf80594e97a21b5985d4686b55. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:58:20.133371 17214 tablet_service.cc:1468] Processing CreateTablet for tablet a847dfbf80594e97a21b5985d4686b55 (DEFAULT_TABLE table=TestTable [id=9af10dd67a814da89e1424fa6aa8a326]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 13:58:20.135028 17214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a847dfbf80594e97a21b5985d4686b55. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:58:20.154012 17431 tablet_bootstrap.cc:492] T a847dfbf80594e97a21b5985d4686b55 P 34d8d90a37d94f3299d957f2ac1ef076: Bootstrap starting.
I20250622 13:58:20.157361 17432 tablet_bootstrap.cc:492] T a847dfbf80594e97a21b5985d4686b55 P ea9534d24449465f8cdd99ec57b54d34: Bootstrap starting.
I20250622 13:58:20.160244 17433 tablet_bootstrap.cc:492] T a847dfbf80594e97a21b5985d4686b55 P d50f6ac76fe14fb3b4345735b6281e4a: Bootstrap starting.
I20250622 13:58:20.163719 17431 tablet_bootstrap.cc:654] T a847dfbf80594e97a21b5985d4686b55 P 34d8d90a37d94f3299d957f2ac1ef076: Neither blocks nor log segments found. Creating new log.
I20250622 13:58:20.164500 17432 tablet_bootstrap.cc:654] T a847dfbf80594e97a21b5985d4686b55 P ea9534d24449465f8cdd99ec57b54d34: Neither blocks nor log segments found. Creating new log.
I20250622 13:58:20.165184 17433 tablet_bootstrap.cc:654] T a847dfbf80594e97a21b5985d4686b55 P d50f6ac76fe14fb3b4345735b6281e4a: Neither blocks nor log segments found. Creating new log.
I20250622 13:58:20.165999 17431 log.cc:826] T a847dfbf80594e97a21b5985d4686b55 P 34d8d90a37d94f3299d957f2ac1ef076: Log is configured to *not* fsync() on all Append() calls
I20250622 13:58:20.166779 17432 log.cc:826] T a847dfbf80594e97a21b5985d4686b55 P ea9534d24449465f8cdd99ec57b54d34: Log is configured to *not* fsync() on all Append() calls
I20250622 13:58:20.166913 17433 log.cc:826] T a847dfbf80594e97a21b5985d4686b55 P d50f6ac76fe14fb3b4345735b6281e4a: Log is configured to *not* fsync() on all Append() calls
I20250622 13:58:20.172255 17431 tablet_bootstrap.cc:492] T a847dfbf80594e97a21b5985d4686b55 P 34d8d90a37d94f3299d957f2ac1ef076: No bootstrap required, opened a new log
I20250622 13:58:20.172871 17431 ts_tablet_manager.cc:1397] T a847dfbf80594e97a21b5985d4686b55 P 34d8d90a37d94f3299d957f2ac1ef076: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.005s
I20250622 13:58:20.173592 17433 tablet_bootstrap.cc:492] T a847dfbf80594e97a21b5985d4686b55 P d50f6ac76fe14fb3b4345735b6281e4a: No bootstrap required, opened a new log
I20250622 13:58:20.174046 17433 ts_tablet_manager.cc:1397] T a847dfbf80594e97a21b5985d4686b55 P d50f6ac76fe14fb3b4345735b6281e4a: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.005s
I20250622 13:58:20.175706 17432 tablet_bootstrap.cc:492] T a847dfbf80594e97a21b5985d4686b55 P ea9534d24449465f8cdd99ec57b54d34: No bootstrap required, opened a new log
I20250622 13:58:20.176173 17432 ts_tablet_manager.cc:1397] T a847dfbf80594e97a21b5985d4686b55 P ea9534d24449465f8cdd99ec57b54d34: Time spent bootstrapping tablet: real 0.019s user 0.018s sys 0.000s
I20250622 13:58:20.191742 17433 raft_consensus.cc:357] T a847dfbf80594e97a21b5985d4686b55 P d50f6ac76fe14fb3b4345735b6281e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9534d24449465f8cdd99ec57b54d34" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 42935 } } peers { permanent_uuid: "34d8d90a37d94f3299d957f2ac1ef076" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44117 } } peers { permanent_uuid: "d50f6ac76fe14fb3b4345735b6281e4a" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 37901 } }
I20250622 13:58:20.192595 17433 raft_consensus.cc:383] T a847dfbf80594e97a21b5985d4686b55 P d50f6ac76fe14fb3b4345735b6281e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:58:20.192842 17433 raft_consensus.cc:738] T a847dfbf80594e97a21b5985d4686b55 P d50f6ac76fe14fb3b4345735b6281e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d50f6ac76fe14fb3b4345735b6281e4a, State: Initialized, Role: FOLLOWER
I20250622 13:58:20.193574 17433 consensus_queue.cc:260] T a847dfbf80594e97a21b5985d4686b55 P d50f6ac76fe14fb3b4345735b6281e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9534d24449465f8cdd99ec57b54d34" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 42935 } } peers { permanent_uuid: "34d8d90a37d94f3299d957f2ac1ef076" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44117 } } peers { permanent_uuid: "d50f6ac76fe14fb3b4345735b6281e4a" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 37901 } }
I20250622 13:58:20.197618 17433 ts_tablet_manager.cc:1428] T a847dfbf80594e97a21b5985d4686b55 P d50f6ac76fe14fb3b4345735b6281e4a: Time spent starting tablet: real 0.023s user 0.019s sys 0.003s
I20250622 13:58:20.199142 17431 raft_consensus.cc:357] T a847dfbf80594e97a21b5985d4686b55 P 34d8d90a37d94f3299d957f2ac1ef076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9534d24449465f8cdd99ec57b54d34" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 42935 } } peers { permanent_uuid: "34d8d90a37d94f3299d957f2ac1ef076" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44117 } } peers { permanent_uuid: "d50f6ac76fe14fb3b4345735b6281e4a" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 37901 } }
I20250622 13:58:20.200130 17431 raft_consensus.cc:383] T a847dfbf80594e97a21b5985d4686b55 P 34d8d90a37d94f3299d957f2ac1ef076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:58:20.200471 17431 raft_consensus.cc:738] T a847dfbf80594e97a21b5985d4686b55 P 34d8d90a37d94f3299d957f2ac1ef076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34d8d90a37d94f3299d957f2ac1ef076, State: Initialized, Role: FOLLOWER
I20250622 13:58:20.201555 17431 consensus_queue.cc:260] T a847dfbf80594e97a21b5985d4686b55 P 34d8d90a37d94f3299d957f2ac1ef076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9534d24449465f8cdd99ec57b54d34" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 42935 } } peers { permanent_uuid: "34d8d90a37d94f3299d957f2ac1ef076" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44117 } } peers { permanent_uuid: "d50f6ac76fe14fb3b4345735b6281e4a" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 37901 } }
I20250622 13:58:20.205901 17431 ts_tablet_manager.cc:1428] T a847dfbf80594e97a21b5985d4686b55 P 34d8d90a37d94f3299d957f2ac1ef076: Time spent starting tablet: real 0.033s user 0.024s sys 0.009s
I20250622 13:58:20.206813 17432 raft_consensus.cc:357] T a847dfbf80594e97a21b5985d4686b55 P ea9534d24449465f8cdd99ec57b54d34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9534d24449465f8cdd99ec57b54d34" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 42935 } } peers { permanent_uuid: "34d8d90a37d94f3299d957f2ac1ef076" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44117 } } peers { permanent_uuid: "d50f6ac76fe14fb3b4345735b6281e4a" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 37901 } }
I20250622 13:58:20.207679 17432 raft_consensus.cc:383] T a847dfbf80594e97a21b5985d4686b55 P ea9534d24449465f8cdd99ec57b54d34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:58:20.207966 17432 raft_consensus.cc:738] T a847dfbf80594e97a21b5985d4686b55 P ea9534d24449465f8cdd99ec57b54d34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea9534d24449465f8cdd99ec57b54d34, State: Initialized, Role: FOLLOWER
I20250622 13:58:20.208786 17432 consensus_queue.cc:260] T a847dfbf80594e97a21b5985d4686b55 P ea9534d24449465f8cdd99ec57b54d34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9534d24449465f8cdd99ec57b54d34" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 42935 } } peers { permanent_uuid: "34d8d90a37d94f3299d957f2ac1ef076" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44117 } } peers { permanent_uuid: "d50f6ac76fe14fb3b4345735b6281e4a" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 37901 } }
I20250622 13:58:20.212033 17412 heartbeater.cc:499] Master 127.15.25.126:40051 was elected leader, sending a full tablet report...
I20250622 13:58:20.212846 17432 ts_tablet_manager.cc:1428] T a847dfbf80594e97a21b5985d4686b55 P ea9534d24449465f8cdd99ec57b54d34: Time spent starting tablet: real 0.036s user 0.024s sys 0.009s
I20250622 13:58:20.246819 17438 raft_consensus.cc:491] T a847dfbf80594e97a21b5985d4686b55 P 34d8d90a37d94f3299d957f2ac1ef076 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:58:20.247275 17438 raft_consensus.cc:513] T a847dfbf80594e97a21b5985d4686b55 P 34d8d90a37d94f3299d957f2ac1ef076 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9534d24449465f8cdd99ec57b54d34" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 42935 } } peers { permanent_uuid: "34d8d90a37d94f3299d957f2ac1ef076" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44117 } } peers { permanent_uuid: "d50f6ac76fe14fb3b4345735b6281e4a" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 37901 } }
I20250622 13:58:20.249807 17438 leader_election.cc:290] T a847dfbf80594e97a21b5985d4686b55 P 34d8d90a37d94f3299d957f2ac1ef076 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea9534d24449465f8cdd99ec57b54d34 (127.15.25.67:42935), d50f6ac76fe14fb3b4345735b6281e4a (127.15.25.66:37901)
I20250622 13:58:20.263180 17234 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a847dfbf80594e97a21b5985d4686b55" candidate_uuid: "34d8d90a37d94f3299d957f2ac1ef076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d50f6ac76fe14fb3b4345735b6281e4a" is_pre_election: true
I20250622 13:58:20.263172 17367 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a847dfbf80594e97a21b5985d4686b55" candidate_uuid: "34d8d90a37d94f3299d957f2ac1ef076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea9534d24449465f8cdd99ec57b54d34" is_pre_election: true
I20250622 13:58:20.264098 17234 raft_consensus.cc:2466] T a847dfbf80594e97a21b5985d4686b55 P d50f6ac76fe14fb3b4345735b6281e4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34d8d90a37d94f3299d957f2ac1ef076 in term 0.
I20250622 13:58:20.264202 17367 raft_consensus.cc:2466] T a847dfbf80594e97a21b5985d4686b55 P ea9534d24449465f8cdd99ec57b54d34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34d8d90a37d94f3299d957f2ac1ef076 in term 0.
I20250622 13:58:20.265453 17036 leader_election.cc:304] T a847dfbf80594e97a21b5985d4686b55 P 34d8d90a37d94f3299d957f2ac1ef076 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34d8d90a37d94f3299d957f2ac1ef076, ea9534d24449465f8cdd99ec57b54d34; no voters:
I20250622 13:58:20.266213 17438 raft_consensus.cc:2802] T a847dfbf80594e97a21b5985d4686b55 P 34d8d90a37d94f3299d957f2ac1ef076 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:58:20.266469 17438 raft_consensus.cc:491] T a847dfbf80594e97a21b5985d4686b55 P 34d8d90a37d94f3299d957f2ac1ef076 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:58:20.266721 17438 raft_consensus.cc:3058] T a847dfbf80594e97a21b5985d4686b55 P 34d8d90a37d94f3299d957f2ac1ef076 [term 0 FOLLOWER]: Advancing to term 1
W20250622 13:58:20.269845 17413 tablet.cc:2378] T a847dfbf80594e97a21b5985d4686b55 P ea9534d24449465f8cdd99ec57b54d34: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 13:58:20.271495 17438 raft_consensus.cc:513] T a847dfbf80594e97a21b5985d4686b55 P 34d8d90a37d94f3299d957f2ac1ef076 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9534d24449465f8cdd99ec57b54d34" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 42935 } } peers { permanent_uuid: "34d8d90a37d94f3299d957f2ac1ef076" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44117 } } peers { permanent_uuid: "d50f6ac76fe14fb3b4345735b6281e4a" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 37901 } }
I20250622 13:58:20.272735 17438 leader_election.cc:290] T a847dfbf80594e97a21b5985d4686b55 P 34d8d90a37d94f3299d957f2ac1ef076 [CANDIDATE]: Term 1 election: Requested vote from peers ea9534d24449465f8cdd99ec57b54d34 (127.15.25.67:42935), d50f6ac76fe14fb3b4345735b6281e4a (127.15.25.66:37901)
I20250622 13:58:20.273500 17367 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a847dfbf80594e97a21b5985d4686b55" candidate_uuid: "34d8d90a37d94f3299d957f2ac1ef076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea9534d24449465f8cdd99ec57b54d34"
I20250622 13:58:20.273555 17234 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a847dfbf80594e97a21b5985d4686b55" candidate_uuid: "34d8d90a37d94f3299d957f2ac1ef076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d50f6ac76fe14fb3b4345735b6281e4a"
I20250622 13:58:20.273933 17367 raft_consensus.cc:3058] T a847dfbf80594e97a21b5985d4686b55 P ea9534d24449465f8cdd99ec57b54d34 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:58:20.274014 17234 raft_consensus.cc:3058] T a847dfbf80594e97a21b5985d4686b55 P d50f6ac76fe14fb3b4345735b6281e4a [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:58:20.278389 17367 raft_consensus.cc:2466] T a847dfbf80594e97a21b5985d4686b55 P ea9534d24449465f8cdd99ec57b54d34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34d8d90a37d94f3299d957f2ac1ef076 in term 1.
I20250622 13:58:20.278496 17234 raft_consensus.cc:2466] T a847dfbf80594e97a21b5985d4686b55 P d50f6ac76fe14fb3b4345735b6281e4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34d8d90a37d94f3299d957f2ac1ef076 in term 1.
I20250622 13:58:20.279160 17036 leader_election.cc:304] T a847dfbf80594e97a21b5985d4686b55 P 34d8d90a37d94f3299d957f2ac1ef076 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34d8d90a37d94f3299d957f2ac1ef076, ea9534d24449465f8cdd99ec57b54d34; no voters:
I20250622 13:58:20.279686 17438 raft_consensus.cc:2802] T a847dfbf80594e97a21b5985d4686b55 P 34d8d90a37d94f3299d957f2ac1ef076 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:58:20.281117 17438 raft_consensus.cc:695] T a847dfbf80594e97a21b5985d4686b55 P 34d8d90a37d94f3299d957f2ac1ef076 [term 1 LEADER]: Becoming Leader. State: Replica: 34d8d90a37d94f3299d957f2ac1ef076, State: Running, Role: LEADER
I20250622 13:58:20.281818 17438 consensus_queue.cc:237] T a847dfbf80594e97a21b5985d4686b55 P 34d8d90a37d94f3299d957f2ac1ef076 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9534d24449465f8cdd99ec57b54d34" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 42935 } } peers { permanent_uuid: "34d8d90a37d94f3299d957f2ac1ef076" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44117 } } peers { permanent_uuid: "d50f6ac76fe14fb3b4345735b6281e4a" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 37901 } }
I20250622 13:58:20.292346 16958 catalog_manager.cc:5582] T a847dfbf80594e97a21b5985d4686b55 P 34d8d90a37d94f3299d957f2ac1ef076 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34d8d90a37d94f3299d957f2ac1ef076 (127.15.25.65). New cstate: current_term: 1 leader_uuid: "34d8d90a37d94f3299d957f2ac1ef076" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea9534d24449465f8cdd99ec57b54d34" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 42935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34d8d90a37d94f3299d957f2ac1ef076" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d50f6ac76fe14fb3b4345735b6281e4a" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 37901 } health_report { overall_health: UNKNOWN } } }
I20250622 13:58:20.330426 15461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250622 13:58:20.334036 15461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 34d8d90a37d94f3299d957f2ac1ef076 to finish bootstrapping
W20250622 13:58:20.341547 17280 tablet.cc:2378] T a847dfbf80594e97a21b5985d4686b55 P d50f6ac76fe14fb3b4345735b6281e4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 13:58:20.348256 15461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver d50f6ac76fe14fb3b4345735b6281e4a to finish bootstrapping
I20250622 13:58:20.359889 15461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver ea9534d24449465f8cdd99ec57b54d34 to finish bootstrapping
I20250622 13:58:20.373661 16958 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35238:
name: "TestAnotherTable"
schema {
columns {
name: "foo"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "bar"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
comment: "comment for bar"
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "foo"
}
}
}
W20250622 13:58:20.374994 16958 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAnotherTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250622 13:58:20.389612 17081 tablet_service.cc:1468] Processing CreateTablet for tablet 1bb38fa4eb21427db77e29d30ee1449d (DEFAULT_TABLE table=TestAnotherTable [id=aafddf89aea04d919c3f924b186c30ee]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250622 13:58:20.390080 17214 tablet_service.cc:1468] Processing CreateTablet for tablet 1bb38fa4eb21427db77e29d30ee1449d (DEFAULT_TABLE table=TestAnotherTable [id=aafddf89aea04d919c3f924b186c30ee]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250622 13:58:20.390655 17081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bb38fa4eb21427db77e29d30ee1449d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:58:20.390826 17347 tablet_service.cc:1468] Processing CreateTablet for tablet 1bb38fa4eb21427db77e29d30ee1449d (DEFAULT_TABLE table=TestAnotherTable [id=aafddf89aea04d919c3f924b186c30ee]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250622 13:58:20.390991 17214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bb38fa4eb21427db77e29d30ee1449d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:58:20.391563 17347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bb38fa4eb21427db77e29d30ee1449d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:58:20.401726 17432 tablet_bootstrap.cc:492] T 1bb38fa4eb21427db77e29d30ee1449d P ea9534d24449465f8cdd99ec57b54d34: Bootstrap starting.
W20250622 13:58:20.403010 17147 tablet.cc:2378] T a847dfbf80594e97a21b5985d4686b55 P 34d8d90a37d94f3299d957f2ac1ef076: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 13:58:20.404312 17431 tablet_bootstrap.cc:492] T 1bb38fa4eb21427db77e29d30ee1449d P 34d8d90a37d94f3299d957f2ac1ef076: Bootstrap starting.
I20250622 13:58:20.405609 17433 tablet_bootstrap.cc:492] T 1bb38fa4eb21427db77e29d30ee1449d P d50f6ac76fe14fb3b4345735b6281e4a: Bootstrap starting.
I20250622 13:58:20.407275 17432 tablet_bootstrap.cc:654] T 1bb38fa4eb21427db77e29d30ee1449d P ea9534d24449465f8cdd99ec57b54d34: Neither blocks nor log segments found. Creating new log.
I20250622 13:58:20.409478 17431 tablet_bootstrap.cc:654] T 1bb38fa4eb21427db77e29d30ee1449d P 34d8d90a37d94f3299d957f2ac1ef076: Neither blocks nor log segments found. Creating new log.
I20250622 13:58:20.411448 17433 tablet_bootstrap.cc:654] T 1bb38fa4eb21427db77e29d30ee1449d P d50f6ac76fe14fb3b4345735b6281e4a: Neither blocks nor log segments found. Creating new log.
I20250622 13:58:20.412895 17432 tablet_bootstrap.cc:492] T 1bb38fa4eb21427db77e29d30ee1449d P ea9534d24449465f8cdd99ec57b54d34: No bootstrap required, opened a new log
I20250622 13:58:20.413347 17432 ts_tablet_manager.cc:1397] T 1bb38fa4eb21427db77e29d30ee1449d P ea9534d24449465f8cdd99ec57b54d34: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20250622 13:58:20.415151 17431 tablet_bootstrap.cc:492] T 1bb38fa4eb21427db77e29d30ee1449d P 34d8d90a37d94f3299d957f2ac1ef076: No bootstrap required, opened a new log
I20250622 13:58:20.415586 17431 ts_tablet_manager.cc:1397] T 1bb38fa4eb21427db77e29d30ee1449d P 34d8d90a37d94f3299d957f2ac1ef076: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.003s
I20250622 13:58:20.415798 17432 raft_consensus.cc:357] T 1bb38fa4eb21427db77e29d30ee1449d P ea9534d24449465f8cdd99ec57b54d34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d8d90a37d94f3299d957f2ac1ef076" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44117 } } peers { permanent_uuid: "d50f6ac76fe14fb3b4345735b6281e4a" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 37901 } } peers { permanent_uuid: "ea9534d24449465f8cdd99ec57b54d34" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 42935 } }
I20250622 13:58:20.416517 17432 raft_consensus.cc:383] T 1bb38fa4eb21427db77e29d30ee1449d P ea9534d24449465f8cdd99ec57b54d34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:58:20.416839 17432 raft_consensus.cc:738] T 1bb38fa4eb21427db77e29d30ee1449d P ea9534d24449465f8cdd99ec57b54d34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea9534d24449465f8cdd99ec57b54d34, State: Initialized, Role: FOLLOWER
I20250622 13:58:20.417064 17433 tablet_bootstrap.cc:492] T 1bb38fa4eb21427db77e29d30ee1449d P d50f6ac76fe14fb3b4345735b6281e4a: No bootstrap required, opened a new log
I20250622 13:58:20.417464 17433 ts_tablet_manager.cc:1397] T 1bb38fa4eb21427db77e29d30ee1449d P d50f6ac76fe14fb3b4345735b6281e4a: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.003s
I20250622 13:58:20.417686 17432 consensus_queue.cc:260] T 1bb38fa4eb21427db77e29d30ee1449d P ea9534d24449465f8cdd99ec57b54d34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d8d90a37d94f3299d957f2ac1ef076" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44117 } } peers { permanent_uuid: "d50f6ac76fe14fb3b4345735b6281e4a" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 37901 } } peers { permanent_uuid: "ea9534d24449465f8cdd99ec57b54d34" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 42935 } }
I20250622 13:58:20.418206 17431 raft_consensus.cc:357] T 1bb38fa4eb21427db77e29d30ee1449d P 34d8d90a37d94f3299d957f2ac1ef076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d8d90a37d94f3299d957f2ac1ef076" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44117 } } peers { permanent_uuid: "d50f6ac76fe14fb3b4345735b6281e4a" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 37901 } } peers { permanent_uuid: "ea9534d24449465f8cdd99ec57b54d34" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 42935 } }
I20250622 13:58:20.418845 17431 raft_consensus.cc:383] T 1bb38fa4eb21427db77e29d30ee1449d P 34d8d90a37d94f3299d957f2ac1ef076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:58:20.419134 17431 raft_consensus.cc:738] T 1bb38fa4eb21427db77e29d30ee1449d P 34d8d90a37d94f3299d957f2ac1ef076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34d8d90a37d94f3299d957f2ac1ef076, State: Initialized, Role: FOLLOWER
I20250622 13:58:20.419914 17432 ts_tablet_manager.cc:1428] T 1bb38fa4eb21427db77e29d30ee1449d P ea9534d24449465f8cdd99ec57b54d34: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20250622 13:58:20.419749 17431 consensus_queue.cc:260] T 1bb38fa4eb21427db77e29d30ee1449d P 34d8d90a37d94f3299d957f2ac1ef076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d8d90a37d94f3299d957f2ac1ef076" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44117 } } peers { permanent_uuid: "d50f6ac76fe14fb3b4345735b6281e4a" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 37901 } } peers { permanent_uuid: "ea9534d24449465f8cdd99ec57b54d34" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 42935 } }
I20250622 13:58:20.420059 17433 raft_consensus.cc:357] T 1bb38fa4eb21427db77e29d30ee1449d P d50f6ac76fe14fb3b4345735b6281e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d8d90a37d94f3299d957f2ac1ef076" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44117 } } peers { permanent_uuid: "d50f6ac76fe14fb3b4345735b6281e4a" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 37901 } } peers { permanent_uuid: "ea9534d24449465f8cdd99ec57b54d34" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 42935 } }
I20250622 13:58:20.421301 17433 raft_consensus.cc:383] T 1bb38fa4eb21427db77e29d30ee1449d P d50f6ac76fe14fb3b4345735b6281e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:58:20.421559 17433 raft_consensus.cc:738] T 1bb38fa4eb21427db77e29d30ee1449d P d50f6ac76fe14fb3b4345735b6281e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d50f6ac76fe14fb3b4345735b6281e4a, State: Initialized, Role: FOLLOWER
I20250622 13:58:20.422149 17433 consensus_queue.cc:260] T 1bb38fa4eb21427db77e29d30ee1449d P d50f6ac76fe14fb3b4345735b6281e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d8d90a37d94f3299d957f2ac1ef076" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44117 } } peers { permanent_uuid: "d50f6ac76fe14fb3b4345735b6281e4a" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 37901 } } peers { permanent_uuid: "ea9534d24449465f8cdd99ec57b54d34" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 42935 } }
I20250622 13:58:20.425159 17433 ts_tablet_manager.cc:1428] T 1bb38fa4eb21427db77e29d30ee1449d P d50f6ac76fe14fb3b4345735b6281e4a: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250622 13:58:20.426532 17431 ts_tablet_manager.cc:1428] T 1bb38fa4eb21427db77e29d30ee1449d P 34d8d90a37d94f3299d957f2ac1ef076: Time spent starting tablet: real 0.011s user 0.004s sys 0.001s
I20250622 13:58:20.502698 17437 raft_consensus.cc:491] T 1bb38fa4eb21427db77e29d30ee1449d P d50f6ac76fe14fb3b4345735b6281e4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:58:20.503072 17437 raft_consensus.cc:513] T 1bb38fa4eb21427db77e29d30ee1449d P d50f6ac76fe14fb3b4345735b6281e4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d8d90a37d94f3299d957f2ac1ef076" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44117 } } peers { permanent_uuid: "d50f6ac76fe14fb3b4345735b6281e4a" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 37901 } } peers { permanent_uuid: "ea9534d24449465f8cdd99ec57b54d34" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 42935 } }
I20250622 13:58:20.505263 17437 leader_election.cc:290] T 1bb38fa4eb21427db77e29d30ee1449d P d50f6ac76fe14fb3b4345735b6281e4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34d8d90a37d94f3299d957f2ac1ef076 (127.15.25.65:44117), ea9534d24449465f8cdd99ec57b54d34 (127.15.25.67:42935)
I20250622 13:58:20.518399 17367 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bb38fa4eb21427db77e29d30ee1449d" candidate_uuid: "d50f6ac76fe14fb3b4345735b6281e4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea9534d24449465f8cdd99ec57b54d34" is_pre_election: true
I20250622 13:58:20.518265 17101 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bb38fa4eb21427db77e29d30ee1449d" candidate_uuid: "d50f6ac76fe14fb3b4345735b6281e4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34d8d90a37d94f3299d957f2ac1ef076" is_pre_election: true
I20250622 13:58:20.518986 17367 raft_consensus.cc:2466] T 1bb38fa4eb21427db77e29d30ee1449d P ea9534d24449465f8cdd99ec57b54d34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d50f6ac76fe14fb3b4345735b6281e4a in term 0.
I20250622 13:58:20.519039 17101 raft_consensus.cc:2466] T 1bb38fa4eb21427db77e29d30ee1449d P 34d8d90a37d94f3299d957f2ac1ef076 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d50f6ac76fe14fb3b4345735b6281e4a in term 0.
I20250622 13:58:20.520052 17169 leader_election.cc:304] T 1bb38fa4eb21427db77e29d30ee1449d P d50f6ac76fe14fb3b4345735b6281e4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d50f6ac76fe14fb3b4345735b6281e4a, ea9534d24449465f8cdd99ec57b54d34; no voters:
I20250622 13:58:20.521231 17437 raft_consensus.cc:2802] T 1bb38fa4eb21427db77e29d30ee1449d P d50f6ac76fe14fb3b4345735b6281e4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:58:20.521493 17437 raft_consensus.cc:491] T 1bb38fa4eb21427db77e29d30ee1449d P d50f6ac76fe14fb3b4345735b6281e4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:58:20.521754 17437 raft_consensus.cc:3058] T 1bb38fa4eb21427db77e29d30ee1449d P d50f6ac76fe14fb3b4345735b6281e4a [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:58:20.526187 17437 raft_consensus.cc:513] T 1bb38fa4eb21427db77e29d30ee1449d P d50f6ac76fe14fb3b4345735b6281e4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d8d90a37d94f3299d957f2ac1ef076" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44117 } } peers { permanent_uuid: "d50f6ac76fe14fb3b4345735b6281e4a" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 37901 } } peers { permanent_uuid: "ea9534d24449465f8cdd99ec57b54d34" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 42935 } }
I20250622 13:58:20.527498 17437 leader_election.cc:290] T 1bb38fa4eb21427db77e29d30ee1449d P d50f6ac76fe14fb3b4345735b6281e4a [CANDIDATE]: Term 1 election: Requested vote from peers 34d8d90a37d94f3299d957f2ac1ef076 (127.15.25.65:44117), ea9534d24449465f8cdd99ec57b54d34 (127.15.25.67:42935)
I20250622 13:58:20.528158 17101 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bb38fa4eb21427db77e29d30ee1449d" candidate_uuid: "d50f6ac76fe14fb3b4345735b6281e4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34d8d90a37d94f3299d957f2ac1ef076"
I20250622 13:58:20.528362 17367 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bb38fa4eb21427db77e29d30ee1449d" candidate_uuid: "d50f6ac76fe14fb3b4345735b6281e4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea9534d24449465f8cdd99ec57b54d34"
I20250622 13:58:20.528545 17101 raft_consensus.cc:3058] T 1bb38fa4eb21427db77e29d30ee1449d P 34d8d90a37d94f3299d957f2ac1ef076 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:58:20.528787 17367 raft_consensus.cc:3058] T 1bb38fa4eb21427db77e29d30ee1449d P ea9534d24449465f8cdd99ec57b54d34 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:58:20.532581 17367 raft_consensus.cc:2466] T 1bb38fa4eb21427db77e29d30ee1449d P ea9534d24449465f8cdd99ec57b54d34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d50f6ac76fe14fb3b4345735b6281e4a in term 1.
I20250622 13:58:20.532755 17101 raft_consensus.cc:2466] T 1bb38fa4eb21427db77e29d30ee1449d P 34d8d90a37d94f3299d957f2ac1ef076 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d50f6ac76fe14fb3b4345735b6281e4a in term 1.
I20250622 13:58:20.533360 17169 leader_election.cc:304] T 1bb38fa4eb21427db77e29d30ee1449d P d50f6ac76fe14fb3b4345735b6281e4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d50f6ac76fe14fb3b4345735b6281e4a, ea9534d24449465f8cdd99ec57b54d34; no voters:
I20250622 13:58:20.534060 17437 raft_consensus.cc:2802] T 1bb38fa4eb21427db77e29d30ee1449d P d50f6ac76fe14fb3b4345735b6281e4a [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:58:20.535562 17437 raft_consensus.cc:695] T 1bb38fa4eb21427db77e29d30ee1449d P d50f6ac76fe14fb3b4345735b6281e4a [term 1 LEADER]: Becoming Leader. State: Replica: d50f6ac76fe14fb3b4345735b6281e4a, State: Running, Role: LEADER
I20250622 13:58:20.536300 17437 consensus_queue.cc:237] T 1bb38fa4eb21427db77e29d30ee1449d P d50f6ac76fe14fb3b4345735b6281e4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d8d90a37d94f3299d957f2ac1ef076" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44117 } } peers { permanent_uuid: "d50f6ac76fe14fb3b4345735b6281e4a" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 37901 } } peers { permanent_uuid: "ea9534d24449465f8cdd99ec57b54d34" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 42935 } }
I20250622 13:58:20.547477 16956 catalog_manager.cc:5582] T 1bb38fa4eb21427db77e29d30ee1449d P d50f6ac76fe14fb3b4345735b6281e4a reported cstate change: term changed from 0 to 1, leader changed from <none> to d50f6ac76fe14fb3b4345735b6281e4a (127.15.25.66). New cstate: current_term: 1 leader_uuid: "d50f6ac76fe14fb3b4345735b6281e4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d8d90a37d94f3299d957f2ac1ef076" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d50f6ac76fe14fb3b4345735b6281e4a" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 37901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea9534d24449465f8cdd99ec57b54d34" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 42935 } health_report { overall_health: UNKNOWN } } }
I20250622 13:58:20.685002 17438 consensus_queue.cc:1035] T a847dfbf80594e97a21b5985d4686b55 P 34d8d90a37d94f3299d957f2ac1ef076 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea9534d24449465f8cdd99ec57b54d34" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 42935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 13:58:20.693789 17438 consensus_queue.cc:1035] T a847dfbf80594e97a21b5985d4686b55 P 34d8d90a37d94f3299d957f2ac1ef076 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d50f6ac76fe14fb3b4345735b6281e4a" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 37901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250622 13:58:20.886276 17452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:58:20.886865 17452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:58:20.914093 17452 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250622 13:58:20.993257 17437 consensus_queue.cc:1035] T 1bb38fa4eb21427db77e29d30ee1449d P d50f6ac76fe14fb3b4345735b6281e4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "34d8d90a37d94f3299d957f2ac1ef076" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250622 13:58:21.013531 17450 consensus_queue.cc:1035] T 1bb38fa4eb21427db77e29d30ee1449d P d50f6ac76fe14fb3b4345735b6281e4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea9534d24449465f8cdd99ec57b54d34" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 42935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250622 13:58:22.159574 17452 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.203s user 0.413s sys 0.783s
W20250622 13:58:22.159818 17452 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.203s user 0.413s sys 0.783s
W20250622 13:58:23.507826 17477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:58:23.508405 17477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:58:23.537204 17477 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250622 13:58:24.761003 17477 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.184s user 0.474s sys 0.707s
W20250622 13:58:24.761443 17477 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.185s user 0.474s sys 0.707s
W20250622 13:58:26.142807 17493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:58:26.143406 17493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:58:26.174153 17493 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250622 13:58:27.458811 17493 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.242s user 0.462s sys 0.778s
W20250622 13:58:27.459187 17493 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.243s user 0.462s sys 0.778s
W20250622 13:58:28.825400 17508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:58:28.826004 17508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:58:28.856881 17508 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250622 13:58:30.118577 17508 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.220s user 0.490s sys 0.729s
W20250622 13:58:30.118849 17508 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.221s user 0.490s sys 0.729s
I20250622 13:58:31.190532 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 17017
I20250622 13:58:31.217190 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 17150
I20250622 13:58:31.241458 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 17283
I20250622 13:58:31.268615 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 16925
2025-06-22T13:58:31Z chronyd exiting
[ OK ] AdminCliTest.TestDescribeTableColumnFlags (18202 ms)
[ RUN ] AdminCliTest.TestAuthzResetCacheNotAuthorized
I20250622 13:58:31.326992 15461 test_util.cc:276] Using random seed: 2106248438
I20250622 13:58:31.331357 15461 ts_itest-base.cc:115] Starting cluster with:
I20250622 13:58:31.331518 15461 ts_itest-base.cc:116] --------------
I20250622 13:58:31.331645 15461 ts_itest-base.cc:117] 3 tablet servers
I20250622 13:58:31.331758 15461 ts_itest-base.cc:118] 3 replicas per TS
I20250622 13:58:31.331880 15461 ts_itest-base.cc:119] --------------
2025-06-22T13:58:31Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-22T13:58:31Z Disabled control of system clock
I20250622 13:58:31.366010 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.25.126:36385
--webserver_interface=127.15.25.126
--webserver_port=0
--builtin_ntp_servers=127.15.25.84:44677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.25.126:36385
--superuser_acl=no-such-user with env {}
W20250622 13:58:31.652765 17529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:58:31.653298 17529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 13:58:31.653848 17529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:58:31.684693 17529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250622 13:58:31.684983 17529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 13:58:31.685196 17529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250622 13:58:31.685387 17529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250622 13:58:31.719166 17529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:44677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.25.126:36385
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.25.126:36385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.25.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 13:58:31.720420 17529 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 13:58:31.721984 17529 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:58:31.735221 17536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:58:31.736212 17535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:58:31.736775 17538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:58:31.738036 17529 server_base.cc:1048] running on GCE node
I20250622 13:58:32.883327 17529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 13:58:32.885996 17529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 13:58:32.887356 17529 hybrid_clock.cc:648] HybridClock initialized: now 1750600712887320 us; error 54 us; skew 500 ppm
I20250622 13:58:32.888109 17529 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:58:32.898779 17529 webserver.cc:469] Webserver started at http://127.15.25.126:35791/ using document root <none> and password file <none>
I20250622 13:58:32.899659 17529 fs_manager.cc:362] Metadata directory not provided
I20250622 13:58:32.899852 17529 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:58:32.900296 17529 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:58:32.904557 17529 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "9b3bd4a7d03946e7985be266aafd7bdd"
format_stamp: "Formatted at 2025-06-22 13:58:32 on dist-test-slave-6tph"
I20250622 13:58:32.905608 17529 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "9b3bd4a7d03946e7985be266aafd7bdd"
format_stamp: "Formatted at 2025-06-22 13:58:32 on dist-test-slave-6tph"
I20250622 13:58:32.912173 17529 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250622 13:58:32.917405 17546 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:32.918503 17529 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250622 13:58:32.918817 17529 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
uuid: "9b3bd4a7d03946e7985be266aafd7bdd"
format_stamp: "Formatted at 2025-06-22 13:58:32 on dist-test-slave-6tph"
I20250622 13:58:32.919162 17529 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:58:32.965718 17529 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:58:32.967096 17529 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 13:58:32.967501 17529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:58:33.035115 17529 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.126:36385
I20250622 13:58:33.035193 17597 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.126:36385 every 8 connection(s)
I20250622 13:58:33.037850 17529 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250622 13:58:33.040192 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 17529
I20250622 13:58:33.040994 15461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250622 13:58:33.043156 17598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:58:33.067667 17598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b3bd4a7d03946e7985be266aafd7bdd: Bootstrap starting.
I20250622 13:58:33.072762 17598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b3bd4a7d03946e7985be266aafd7bdd: Neither blocks nor log segments found. Creating new log.
I20250622 13:58:33.074944 17598 log.cc:826] T 00000000000000000000000000000000 P 9b3bd4a7d03946e7985be266aafd7bdd: Log is configured to *not* fsync() on all Append() calls
I20250622 13:58:33.078826 17598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b3bd4a7d03946e7985be266aafd7bdd: No bootstrap required, opened a new log
I20250622 13:58:33.095598 17598 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9b3bd4a7d03946e7985be266aafd7bdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b3bd4a7d03946e7985be266aafd7bdd" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 36385 } }
I20250622 13:58:33.096202 17598 raft_consensus.cc:383] T 00000000000000000000000000000000 P 9b3bd4a7d03946e7985be266aafd7bdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:58:33.096400 17598 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9b3bd4a7d03946e7985be266aafd7bdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b3bd4a7d03946e7985be266aafd7bdd, State: Initialized, Role: FOLLOWER
I20250622 13:58:33.097045 17598 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b3bd4a7d03946e7985be266aafd7bdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b3bd4a7d03946e7985be266aafd7bdd" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 36385 } }
I20250622 13:58:33.097499 17598 raft_consensus.cc:397] T 00000000000000000000000000000000 P 9b3bd4a7d03946e7985be266aafd7bdd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 13:58:33.097791 17598 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9b3bd4a7d03946e7985be266aafd7bdd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 13:58:33.098170 17598 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9b3bd4a7d03946e7985be266aafd7bdd [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:58:33.102563 17598 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9b3bd4a7d03946e7985be266aafd7bdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b3bd4a7d03946e7985be266aafd7bdd" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 36385 } }
I20250622 13:58:33.103228 17598 leader_election.cc:304] T 00000000000000000000000000000000 P 9b3bd4a7d03946e7985be266aafd7bdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b3bd4a7d03946e7985be266aafd7bdd; no voters:
I20250622 13:58:33.104761 17598 leader_election.cc:290] T 00000000000000000000000000000000 P 9b3bd4a7d03946e7985be266aafd7bdd [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 13:58:33.105507 17603 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9b3bd4a7d03946e7985be266aafd7bdd [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:58:33.107589 17603 raft_consensus.cc:695] T 00000000000000000000000000000000 P 9b3bd4a7d03946e7985be266aafd7bdd [term 1 LEADER]: Becoming Leader. State: Replica: 9b3bd4a7d03946e7985be266aafd7bdd, State: Running, Role: LEADER
I20250622 13:58:33.108209 17603 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b3bd4a7d03946e7985be266aafd7bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b3bd4a7d03946e7985be266aafd7bdd" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 36385 } }
I20250622 13:58:33.108544 17598 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b3bd4a7d03946e7985be266aafd7bdd [sys.catalog]: configured and running, proceeding with master startup.
I20250622 13:58:33.121235 17605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b3bd4a7d03946e7985be266aafd7bdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b3bd4a7d03946e7985be266aafd7bdd. Latest consensus state: current_term: 1 leader_uuid: "9b3bd4a7d03946e7985be266aafd7bdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b3bd4a7d03946e7985be266aafd7bdd" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 36385 } } }
I20250622 13:58:33.121174 17604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b3bd4a7d03946e7985be266aafd7bdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b3bd4a7d03946e7985be266aafd7bdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b3bd4a7d03946e7985be266aafd7bdd" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 36385 } } }
I20250622 13:58:33.121920 17605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b3bd4a7d03946e7985be266aafd7bdd [sys.catalog]: This master's current role is: LEADER
I20250622 13:58:33.122051 17604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b3bd4a7d03946e7985be266aafd7bdd [sys.catalog]: This master's current role is: LEADER
I20250622 13:58:33.125128 17612 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 13:58:33.136293 17612 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 13:58:33.152384 17612 catalog_manager.cc:1349] Generated new cluster ID: ca31a97704134c2e886ebc7f8ef2bc87
I20250622 13:58:33.152678 17612 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 13:58:33.188210 17612 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 13:58:33.190141 17612 catalog_manager.cc:1506] Loading token signing keys...
I20250622 13:58:33.204278 17612 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 9b3bd4a7d03946e7985be266aafd7bdd: Generated new TSK 0
I20250622 13:58:33.205334 17612 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 13:58:33.216998 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.25.65:0
--local_ip_for_outbound_sockets=127.15.25.65
--webserver_interface=127.15.25.65
--webserver_port=0
--tserver_master_addrs=127.15.25.126:36385
--builtin_ntp_servers=127.15.25.84:44677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250622 13:58:33.508092 17622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:58:33.508612 17622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 13:58:33.509109 17622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:58:33.538851 17622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 13:58:33.539669 17622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.25.65
I20250622 13:58:33.573369 17622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:44677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.25.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.15.25.65
--webserver_port=0
--tserver_master_addrs=127.15.25.126:36385
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.25.65
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 13:58:33.574769 17622 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 13:58:33.576370 17622 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:58:33.593036 17629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:58:33.593250 17628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:58:33.593189 17631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:58:35.119958 17630 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1525 milliseconds
I20250622 13:58:35.120072 17622 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 13:58:35.121263 17622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 13:58:35.123473 17622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 13:58:35.124790 17622 hybrid_clock.cc:648] HybridClock initialized: now 1750600715124767 us; error 45 us; skew 500 ppm
I20250622 13:58:35.125556 17622 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:58:35.132210 17622 webserver.cc:469] Webserver started at http://127.15.25.65:43439/ using document root <none> and password file <none>
I20250622 13:58:35.133059 17622 fs_manager.cc:362] Metadata directory not provided
I20250622 13:58:35.133241 17622 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:58:35.133683 17622 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:58:35.137974 17622 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "0cd531c740744c498883fd80cb33b4b2"
format_stamp: "Formatted at 2025-06-22 13:58:35 on dist-test-slave-6tph"
I20250622 13:58:35.138998 17622 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "0cd531c740744c498883fd80cb33b4b2"
format_stamp: "Formatted at 2025-06-22 13:58:35 on dist-test-slave-6tph"
I20250622 13:58:35.145571 17622 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20250622 13:58:35.150676 17638 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:35.151603 17622 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20250622 13:58:35.151887 17622 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "0cd531c740744c498883fd80cb33b4b2"
format_stamp: "Formatted at 2025-06-22 13:58:35 on dist-test-slave-6tph"
I20250622 13:58:35.152172 17622 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:58:35.198670 17622 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:58:35.200017 17622 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 13:58:35.200420 17622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:58:35.202713 17622 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:58:35.206554 17622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:58:35.206733 17622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:35.206991 17622 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:58:35.207142 17622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:35.334173 17622 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.65:38473
I20250622 13:58:35.334273 17750 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.65:38473 every 8 connection(s)
I20250622 13:58:35.336736 17622 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250622 13:58:35.344494 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 17622
I20250622 13:58:35.344980 15461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250622 13:58:35.352154 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.25.66:0
--local_ip_for_outbound_sockets=127.15.25.66
--webserver_interface=127.15.25.66
--webserver_port=0
--tserver_master_addrs=127.15.25.126:36385
--builtin_ntp_servers=127.15.25.84:44677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 13:58:35.359397 17751 heartbeater.cc:344] Connected to a master server at 127.15.25.126:36385
I20250622 13:58:35.359891 17751 heartbeater.cc:461] Registering TS with master...
I20250622 13:58:35.361110 17751 heartbeater.cc:507] Master 127.15.25.126:36385 requested a full tablet report, sending...
I20250622 13:58:35.363775 17563 ts_manager.cc:194] Registered new tserver with Master: 0cd531c740744c498883fd80cb33b4b2 (127.15.25.65:38473)
I20250622 13:58:35.365620 17563 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.25.65:42489
W20250622 13:58:35.630674 17755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:58:35.631155 17755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 13:58:35.631655 17755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:58:35.661567 17755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 13:58:35.662420 17755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.25.66
I20250622 13:58:35.696589 17755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:44677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.25.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.15.25.66
--webserver_port=0
--tserver_master_addrs=127.15.25.126:36385
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.25.66
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 13:58:35.697876 17755 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 13:58:35.699406 17755 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:58:35.715214 17761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:58:36.369269 17751 heartbeater.cc:499] Master 127.15.25.126:36385 was elected leader, sending a full tablet report...
W20250622 13:58:35.715255 17762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:58:35.715890 17764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:58:35.717259 17755 server_base.cc:1048] running on GCE node
I20250622 13:58:36.809051 17755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 13:58:36.811208 17755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 13:58:36.812551 17755 hybrid_clock.cc:648] HybridClock initialized: now 1750600716812512 us; error 50 us; skew 500 ppm
I20250622 13:58:36.813300 17755 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:58:36.819159 17755 webserver.cc:469] Webserver started at http://127.15.25.66:40023/ using document root <none> and password file <none>
I20250622 13:58:36.819998 17755 fs_manager.cc:362] Metadata directory not provided
I20250622 13:58:36.820204 17755 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:58:36.820605 17755 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:58:36.824699 17755 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "c5e3538dc5524f3b924d47ab6946596c"
format_stamp: "Formatted at 2025-06-22 13:58:36 on dist-test-slave-6tph"
I20250622 13:58:36.825930 17755 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "c5e3538dc5524f3b924d47ab6946596c"
format_stamp: "Formatted at 2025-06-22 13:58:36 on dist-test-slave-6tph"
I20250622 13:58:36.832445 17755 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250622 13:58:36.837431 17771 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:36.838346 17755 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250622 13:58:36.838605 17755 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "c5e3538dc5524f3b924d47ab6946596c"
format_stamp: "Formatted at 2025-06-22 13:58:36 on dist-test-slave-6tph"
I20250622 13:58:36.838886 17755 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:58:36.895723 17755 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:58:36.897132 17755 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 13:58:36.897531 17755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:58:36.899752 17755 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:58:36.903376 17755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:58:36.903600 17755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:36.903820 17755 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:58:36.903977 17755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:37.032488 17755 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.66:45445
I20250622 13:58:37.032588 17883 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.66:45445 every 8 connection(s)
I20250622 13:58:37.035912 17755 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250622 13:58:37.040066 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 17755
I20250622 13:58:37.040575 15461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250622 13:58:37.047410 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.25.67:0
--local_ip_for_outbound_sockets=127.15.25.67
--webserver_interface=127.15.25.67
--webserver_port=0
--tserver_master_addrs=127.15.25.126:36385
--builtin_ntp_servers=127.15.25.84:44677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 13:58:37.058085 17884 heartbeater.cc:344] Connected to a master server at 127.15.25.126:36385
I20250622 13:58:37.058584 17884 heartbeater.cc:461] Registering TS with master...
I20250622 13:58:37.059765 17884 heartbeater.cc:507] Master 127.15.25.126:36385 requested a full tablet report, sending...
I20250622 13:58:37.062033 17563 ts_manager.cc:194] Registered new tserver with Master: c5e3538dc5524f3b924d47ab6946596c (127.15.25.66:45445)
I20250622 13:58:37.063474 17563 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.25.66:34641
W20250622 13:58:37.347035 17888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:58:37.347672 17888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 13:58:37.348176 17888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:58:37.379720 17888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 13:58:37.380553 17888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.25.67
I20250622 13:58:37.413921 17888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:44677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.25.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.15.25.67
--webserver_port=0
--tserver_master_addrs=127.15.25.126:36385
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.25.67
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 13:58:37.415237 17888 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 13:58:37.416821 17888 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:58:37.431384 17895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:58:38.067202 17884 heartbeater.cc:499] Master 127.15.25.126:36385 was elected leader, sending a full tablet report...
W20250622 13:58:37.432024 17894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:58:37.433017 17897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:58:37.433386 17888 server_base.cc:1048] running on GCE node
I20250622 13:58:38.535547 17888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 13:58:38.538194 17888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 13:58:38.539604 17888 hybrid_clock.cc:648] HybridClock initialized: now 1750600718539552 us; error 68 us; skew 500 ppm
I20250622 13:58:38.540364 17888 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:58:38.550706 17888 webserver.cc:469] Webserver started at http://127.15.25.67:40689/ using document root <none> and password file <none>
I20250622 13:58:38.551548 17888 fs_manager.cc:362] Metadata directory not provided
I20250622 13:58:38.551728 17888 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:58:38.552119 17888 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:58:38.556488 17888 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "9910089f26514b2e9198126c7de57467"
format_stamp: "Formatted at 2025-06-22 13:58:38 on dist-test-slave-6tph"
I20250622 13:58:38.557427 17888 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "9910089f26514b2e9198126c7de57467"
format_stamp: "Formatted at 2025-06-22 13:58:38 on dist-test-slave-6tph"
I20250622 13:58:38.563500 17888 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20250622 13:58:38.568516 17904 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:38.569458 17888 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250622 13:58:38.569767 17888 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9910089f26514b2e9198126c7de57467"
format_stamp: "Formatted at 2025-06-22 13:58:38 on dist-test-slave-6tph"
I20250622 13:58:38.570055 17888 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:58:38.614817 17888 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:58:38.616184 17888 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 13:58:38.616551 17888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:58:38.618736 17888 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:58:38.622185 17888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:58:38.622363 17888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:38.622588 17888 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:58:38.622773 17888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:38.748306 17888 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.67:33721
I20250622 13:58:38.748404 18016 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.67:33721 every 8 connection(s)
I20250622 13:58:38.750646 17888 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250622 13:58:38.756016 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 17888
I20250622 13:58:38.756546 15461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250622 13:58:38.771317 18017 heartbeater.cc:344] Connected to a master server at 127.15.25.126:36385
I20250622 13:58:38.771811 18017 heartbeater.cc:461] Registering TS with master...
I20250622 13:58:38.772933 18017 heartbeater.cc:507] Master 127.15.25.126:36385 requested a full tablet report, sending...
I20250622 13:58:38.775009 17562 ts_manager.cc:194] Registered new tserver with Master: 9910089f26514b2e9198126c7de57467 (127.15.25.67:33721)
I20250622 13:58:38.776357 17562 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.25.67:44873
I20250622 13:58:38.788173 15461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250622 13:58:38.817003 17562 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60598:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250622 13:58:38.834473 17562 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250622 13:58:38.894182 17819 tablet_service.cc:1468] Processing CreateTablet for tablet d26877634c8542a09547dbe8c896dd8d (DEFAULT_TABLE table=TestTable [id=05aae5d2066c4e72834a4cca41876315]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 13:58:38.896157 17819 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d26877634c8542a09547dbe8c896dd8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:58:38.896241 17952 tablet_service.cc:1468] Processing CreateTablet for tablet d26877634c8542a09547dbe8c896dd8d (DEFAULT_TABLE table=TestTable [id=05aae5d2066c4e72834a4cca41876315]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 13:58:38.898054 17952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d26877634c8542a09547dbe8c896dd8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:58:38.898738 17686 tablet_service.cc:1468] Processing CreateTablet for tablet d26877634c8542a09547dbe8c896dd8d (DEFAULT_TABLE table=TestTable [id=05aae5d2066c4e72834a4cca41876315]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 13:58:38.900049 17686 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d26877634c8542a09547dbe8c896dd8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:58:38.922458 18036 tablet_bootstrap.cc:492] T d26877634c8542a09547dbe8c896dd8d P 0cd531c740744c498883fd80cb33b4b2: Bootstrap starting.
I20250622 13:58:38.923090 18037 tablet_bootstrap.cc:492] T d26877634c8542a09547dbe8c896dd8d P c5e3538dc5524f3b924d47ab6946596c: Bootstrap starting.
I20250622 13:58:38.926461 18038 tablet_bootstrap.cc:492] T d26877634c8542a09547dbe8c896dd8d P 9910089f26514b2e9198126c7de57467: Bootstrap starting.
I20250622 13:58:38.931044 18037 tablet_bootstrap.cc:654] T d26877634c8542a09547dbe8c896dd8d P c5e3538dc5524f3b924d47ab6946596c: Neither blocks nor log segments found. Creating new log.
I20250622 13:58:38.931589 18036 tablet_bootstrap.cc:654] T d26877634c8542a09547dbe8c896dd8d P 0cd531c740744c498883fd80cb33b4b2: Neither blocks nor log segments found. Creating new log.
I20250622 13:58:38.932714 18038 tablet_bootstrap.cc:654] T d26877634c8542a09547dbe8c896dd8d P 9910089f26514b2e9198126c7de57467: Neither blocks nor log segments found. Creating new log.
I20250622 13:58:38.933529 18037 log.cc:826] T d26877634c8542a09547dbe8c896dd8d P c5e3538dc5524f3b924d47ab6946596c: Log is configured to *not* fsync() on all Append() calls
I20250622 13:58:38.934015 18036 log.cc:826] T d26877634c8542a09547dbe8c896dd8d P 0cd531c740744c498883fd80cb33b4b2: Log is configured to *not* fsync() on all Append() calls
I20250622 13:58:38.934482 18038 log.cc:826] T d26877634c8542a09547dbe8c896dd8d P 9910089f26514b2e9198126c7de57467: Log is configured to *not* fsync() on all Append() calls
I20250622 13:58:38.939261 18037 tablet_bootstrap.cc:492] T d26877634c8542a09547dbe8c896dd8d P c5e3538dc5524f3b924d47ab6946596c: No bootstrap required, opened a new log
I20250622 13:58:38.939677 18037 ts_tablet_manager.cc:1397] T d26877634c8542a09547dbe8c896dd8d P c5e3538dc5524f3b924d47ab6946596c: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.005s
I20250622 13:58:38.940335 18038 tablet_bootstrap.cc:492] T d26877634c8542a09547dbe8c896dd8d P 9910089f26514b2e9198126c7de57467: No bootstrap required, opened a new log
I20250622 13:58:38.940361 18036 tablet_bootstrap.cc:492] T d26877634c8542a09547dbe8c896dd8d P 0cd531c740744c498883fd80cb33b4b2: No bootstrap required, opened a new log
I20250622 13:58:38.940812 18036 ts_tablet_manager.cc:1397] T d26877634c8542a09547dbe8c896dd8d P 0cd531c740744c498883fd80cb33b4b2: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.007s
I20250622 13:58:38.940809 18038 ts_tablet_manager.cc:1397] T d26877634c8542a09547dbe8c896dd8d P 9910089f26514b2e9198126c7de57467: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.007s
I20250622 13:58:38.957955 18037 raft_consensus.cc:357] T d26877634c8542a09547dbe8c896dd8d P c5e3538dc5524f3b924d47ab6946596c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e3538dc5524f3b924d47ab6946596c" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45445 } } peers { permanent_uuid: "9910089f26514b2e9198126c7de57467" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33721 } } peers { permanent_uuid: "0cd531c740744c498883fd80cb33b4b2" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 38473 } }
I20250622 13:58:38.958662 18037 raft_consensus.cc:383] T d26877634c8542a09547dbe8c896dd8d P c5e3538dc5524f3b924d47ab6946596c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:58:38.959020 18037 raft_consensus.cc:738] T d26877634c8542a09547dbe8c896dd8d P c5e3538dc5524f3b924d47ab6946596c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e3538dc5524f3b924d47ab6946596c, State: Initialized, Role: FOLLOWER
I20250622 13:58:38.959790 18037 consensus_queue.cc:260] T d26877634c8542a09547dbe8c896dd8d P c5e3538dc5524f3b924d47ab6946596c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e3538dc5524f3b924d47ab6946596c" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45445 } } peers { permanent_uuid: "9910089f26514b2e9198126c7de57467" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33721 } } peers { permanent_uuid: "0cd531c740744c498883fd80cb33b4b2" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 38473 } }
I20250622 13:58:38.963835 18037 ts_tablet_manager.cc:1428] T d26877634c8542a09547dbe8c896dd8d P c5e3538dc5524f3b924d47ab6946596c: Time spent starting tablet: real 0.024s user 0.021s sys 0.003s
I20250622 13:58:38.967180 18036 raft_consensus.cc:357] T d26877634c8542a09547dbe8c896dd8d P 0cd531c740744c498883fd80cb33b4b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e3538dc5524f3b924d47ab6946596c" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45445 } } peers { permanent_uuid: "9910089f26514b2e9198126c7de57467" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33721 } } peers { permanent_uuid: "0cd531c740744c498883fd80cb33b4b2" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 38473 } }
I20250622 13:58:38.967180 18038 raft_consensus.cc:357] T d26877634c8542a09547dbe8c896dd8d P 9910089f26514b2e9198126c7de57467 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e3538dc5524f3b924d47ab6946596c" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45445 } } peers { permanent_uuid: "9910089f26514b2e9198126c7de57467" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33721 } } peers { permanent_uuid: "0cd531c740744c498883fd80cb33b4b2" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 38473 } }
I20250622 13:58:38.968036 18038 raft_consensus.cc:383] T d26877634c8542a09547dbe8c896dd8d P 9910089f26514b2e9198126c7de57467 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:58:38.968142 18036 raft_consensus.cc:383] T d26877634c8542a09547dbe8c896dd8d P 0cd531c740744c498883fd80cb33b4b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:58:38.968362 18038 raft_consensus.cc:738] T d26877634c8542a09547dbe8c896dd8d P 9910089f26514b2e9198126c7de57467 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9910089f26514b2e9198126c7de57467, State: Initialized, Role: FOLLOWER
I20250622 13:58:38.968449 18036 raft_consensus.cc:738] T d26877634c8542a09547dbe8c896dd8d P 0cd531c740744c498883fd80cb33b4b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cd531c740744c498883fd80cb33b4b2, State: Initialized, Role: FOLLOWER
I20250622 13:58:38.969202 18038 consensus_queue.cc:260] T d26877634c8542a09547dbe8c896dd8d P 9910089f26514b2e9198126c7de57467 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e3538dc5524f3b924d47ab6946596c" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45445 } } peers { permanent_uuid: "9910089f26514b2e9198126c7de57467" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33721 } } peers { permanent_uuid: "0cd531c740744c498883fd80cb33b4b2" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 38473 } }
I20250622 13:58:38.969710 18036 consensus_queue.cc:260] T d26877634c8542a09547dbe8c896dd8d P 0cd531c740744c498883fd80cb33b4b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e3538dc5524f3b924d47ab6946596c" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45445 } } peers { permanent_uuid: "9910089f26514b2e9198126c7de57467" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33721 } } peers { permanent_uuid: "0cd531c740744c498883fd80cb33b4b2" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 38473 } }
I20250622 13:58:38.974560 18036 ts_tablet_manager.cc:1428] T d26877634c8542a09547dbe8c896dd8d P 0cd531c740744c498883fd80cb33b4b2: Time spent starting tablet: real 0.033s user 0.033s sys 0.002s
I20250622 13:58:38.977155 18017 heartbeater.cc:499] Master 127.15.25.126:36385 was elected leader, sending a full tablet report...
I20250622 13:58:38.978163 18038 ts_tablet_manager.cc:1428] T d26877634c8542a09547dbe8c896dd8d P 9910089f26514b2e9198126c7de57467: Time spent starting tablet: real 0.037s user 0.031s sys 0.008s
W20250622 13:58:39.004812 18018 tablet.cc:2378] T d26877634c8542a09547dbe8c896dd8d P 9910089f26514b2e9198126c7de57467: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250622 13:58:39.043080 17885 tablet.cc:2378] T d26877634c8542a09547dbe8c896dd8d P c5e3538dc5524f3b924d47ab6946596c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250622 13:58:39.094575 17752 tablet.cc:2378] T d26877634c8542a09547dbe8c896dd8d P 0cd531c740744c498883fd80cb33b4b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 13:58:39.208006 18043 raft_consensus.cc:491] T d26877634c8542a09547dbe8c896dd8d P 9910089f26514b2e9198126c7de57467 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:58:39.208447 18043 raft_consensus.cc:513] T d26877634c8542a09547dbe8c896dd8d P 9910089f26514b2e9198126c7de57467 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e3538dc5524f3b924d47ab6946596c" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45445 } } peers { permanent_uuid: "9910089f26514b2e9198126c7de57467" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33721 } } peers { permanent_uuid: "0cd531c740744c498883fd80cb33b4b2" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 38473 } }
I20250622 13:58:39.210675 18043 leader_election.cc:290] T d26877634c8542a09547dbe8c896dd8d P 9910089f26514b2e9198126c7de57467 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5e3538dc5524f3b924d47ab6946596c (127.15.25.66:45445), 0cd531c740744c498883fd80cb33b4b2 (127.15.25.65:38473)
I20250622 13:58:39.218147 17839 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d26877634c8542a09547dbe8c896dd8d" candidate_uuid: "9910089f26514b2e9198126c7de57467" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e3538dc5524f3b924d47ab6946596c" is_pre_election: true
I20250622 13:58:39.218839 17839 raft_consensus.cc:2466] T d26877634c8542a09547dbe8c896dd8d P c5e3538dc5524f3b924d47ab6946596c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9910089f26514b2e9198126c7de57467 in term 0.
I20250622 13:58:39.220101 17908 leader_election.cc:304] T d26877634c8542a09547dbe8c896dd8d P 9910089f26514b2e9198126c7de57467 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9910089f26514b2e9198126c7de57467, c5e3538dc5524f3b924d47ab6946596c; no voters:
I20250622 13:58:39.220024 17706 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d26877634c8542a09547dbe8c896dd8d" candidate_uuid: "9910089f26514b2e9198126c7de57467" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cd531c740744c498883fd80cb33b4b2" is_pre_election: true
I20250622 13:58:39.220835 18043 raft_consensus.cc:2802] T d26877634c8542a09547dbe8c896dd8d P 9910089f26514b2e9198126c7de57467 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:58:39.220795 17706 raft_consensus.cc:2466] T d26877634c8542a09547dbe8c896dd8d P 0cd531c740744c498883fd80cb33b4b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9910089f26514b2e9198126c7de57467 in term 0.
I20250622 13:58:39.221088 18043 raft_consensus.cc:491] T d26877634c8542a09547dbe8c896dd8d P 9910089f26514b2e9198126c7de57467 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:58:39.221336 18043 raft_consensus.cc:3058] T d26877634c8542a09547dbe8c896dd8d P 9910089f26514b2e9198126c7de57467 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:58:39.225467 18043 raft_consensus.cc:513] T d26877634c8542a09547dbe8c896dd8d P 9910089f26514b2e9198126c7de57467 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e3538dc5524f3b924d47ab6946596c" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45445 } } peers { permanent_uuid: "9910089f26514b2e9198126c7de57467" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33721 } } peers { permanent_uuid: "0cd531c740744c498883fd80cb33b4b2" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 38473 } }
I20250622 13:58:39.226789 18043 leader_election.cc:290] T d26877634c8542a09547dbe8c896dd8d P 9910089f26514b2e9198126c7de57467 [CANDIDATE]: Term 1 election: Requested vote from peers c5e3538dc5524f3b924d47ab6946596c (127.15.25.66:45445), 0cd531c740744c498883fd80cb33b4b2 (127.15.25.65:38473)
I20250622 13:58:39.227445 17839 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d26877634c8542a09547dbe8c896dd8d" candidate_uuid: "9910089f26514b2e9198126c7de57467" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e3538dc5524f3b924d47ab6946596c"
I20250622 13:58:39.227557 17706 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d26877634c8542a09547dbe8c896dd8d" candidate_uuid: "9910089f26514b2e9198126c7de57467" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cd531c740744c498883fd80cb33b4b2"
I20250622 13:58:39.227804 17839 raft_consensus.cc:3058] T d26877634c8542a09547dbe8c896dd8d P c5e3538dc5524f3b924d47ab6946596c [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:58:39.227907 17706 raft_consensus.cc:3058] T d26877634c8542a09547dbe8c896dd8d P 0cd531c740744c498883fd80cb33b4b2 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:58:39.231909 17839 raft_consensus.cc:2466] T d26877634c8542a09547dbe8c896dd8d P c5e3538dc5524f3b924d47ab6946596c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9910089f26514b2e9198126c7de57467 in term 1.
I20250622 13:58:39.231956 17706 raft_consensus.cc:2466] T d26877634c8542a09547dbe8c896dd8d P 0cd531c740744c498883fd80cb33b4b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9910089f26514b2e9198126c7de57467 in term 1.
I20250622 13:58:39.232610 17908 leader_election.cc:304] T d26877634c8542a09547dbe8c896dd8d P 9910089f26514b2e9198126c7de57467 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9910089f26514b2e9198126c7de57467, c5e3538dc5524f3b924d47ab6946596c; no voters:
I20250622 13:58:39.233160 18043 raft_consensus.cc:2802] T d26877634c8542a09547dbe8c896dd8d P 9910089f26514b2e9198126c7de57467 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:58:39.234576 18043 raft_consensus.cc:695] T d26877634c8542a09547dbe8c896dd8d P 9910089f26514b2e9198126c7de57467 [term 1 LEADER]: Becoming Leader. State: Replica: 9910089f26514b2e9198126c7de57467, State: Running, Role: LEADER
I20250622 13:58:39.235399 18043 consensus_queue.cc:237] T d26877634c8542a09547dbe8c896dd8d P 9910089f26514b2e9198126c7de57467 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e3538dc5524f3b924d47ab6946596c" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45445 } } peers { permanent_uuid: "9910089f26514b2e9198126c7de57467" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33721 } } peers { permanent_uuid: "0cd531c740744c498883fd80cb33b4b2" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 38473 } }
I20250622 13:58:39.245508 17562 catalog_manager.cc:5582] T d26877634c8542a09547dbe8c896dd8d P 9910089f26514b2e9198126c7de57467 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9910089f26514b2e9198126c7de57467 (127.15.25.67). New cstate: current_term: 1 leader_uuid: "9910089f26514b2e9198126c7de57467" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5e3538dc5524f3b924d47ab6946596c" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9910089f26514b2e9198126c7de57467" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cd531c740744c498883fd80cb33b4b2" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 38473 } health_report { overall_health: UNKNOWN } } }
I20250622 13:58:39.337447 15461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250622 13:58:39.340476 15461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 0cd531c740744c498883fd80cb33b4b2 to finish bootstrapping
I20250622 13:58:39.352371 15461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver c5e3538dc5524f3b924d47ab6946596c to finish bootstrapping
I20250622 13:58:39.362950 15461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 9910089f26514b2e9198126c7de57467 to finish bootstrapping
I20250622 13:58:39.829468 18059 consensus_queue.cc:1035] T d26877634c8542a09547dbe8c896dd8d P 9910089f26514b2e9198126c7de57467 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e3538dc5524f3b924d47ab6946596c" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250622 13:58:39.836498 18059 consensus_queue.cc:1035] T d26877634c8542a09547dbe8c896dd8d P 9910089f26514b2e9198126c7de57467 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cd531c740744c498883fd80cb33b4b2" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 38473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250622 13:58:40.973501 17562 server_base.cc:1130] Unauthorized access attempt to method kudu.master.MasterService.RefreshAuthzCache from {username='slave'} at 127.0.0.1:34394
I20250622 13:58:42.002358 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 17622
W20250622 13:58:42.024250 17906 connection.cc:537] client connection to 127.15.25.65:38473 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250622 13:58:42.024758 17906 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250622 13:58:42.024832 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 17755
W20250622 13:58:42.029847 17906 consensus_peers.cc:487] T d26877634c8542a09547dbe8c896dd8d P 9910089f26514b2e9198126c7de57467 -> Peer 0cd531c740744c498883fd80cb33b4b2 (127.15.25.65:38473): Couldn't send request to peer 0cd531c740744c498883fd80cb33b4b2. Status: Network error: Client connection negotiation failed: client connection to 127.15.25.65:38473: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250622 13:58:42.054651 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 17888
I20250622 13:58:42.077584 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 17529
2025-06-22T13:58:42Z chronyd exiting
[ OK ] AdminCliTest.TestAuthzResetCacheNotAuthorized (10802 ms)
[ RUN ] AdminCliTest.TestRebuildTables
I20250622 13:58:42.129357 15461 test_util.cc:276] Using random seed: 2117050806
I20250622 13:58:42.133036 15461 ts_itest-base.cc:115] Starting cluster with:
I20250622 13:58:42.133167 15461 ts_itest-base.cc:116] --------------
I20250622 13:58:42.133312 15461 ts_itest-base.cc:117] 3 tablet servers
I20250622 13:58:42.133425 15461 ts_itest-base.cc:118] 3 replicas per TS
I20250622 13:58:42.133551 15461 ts_itest-base.cc:119] --------------
2025-06-22T13:58:42Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-22T13:58:42Z Disabled control of system clock
I20250622 13:58:42.166420 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.25.126:44459
--webserver_interface=127.15.25.126
--webserver_port=0
--builtin_ntp_servers=127.15.25.84:43331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.25.126:44459 with env {}
W20250622 13:58:42.444579 18089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:58:42.445124 18089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 13:58:42.445514 18089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:58:42.475461 18089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250622 13:58:42.475761 18089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 13:58:42.475963 18089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250622 13:58:42.476150 18089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250622 13:58:42.507932 18089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:43331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.25.126:44459
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.25.126:44459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.25.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 13:58:42.509173 18089 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 13:58:42.510766 18089 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:58:42.523811 18095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:58:42.523919 18096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:58:42.526517 18098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:58:43.638151 18097 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1111 milliseconds
I20250622 13:58:43.638252 18089 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 13:58:43.639384 18089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 13:58:43.642323 18089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 13:58:43.643721 18089 hybrid_clock.cc:648] HybridClock initialized: now 1750600723643679 us; error 53 us; skew 500 ppm
I20250622 13:58:43.644557 18089 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:58:43.654356 18089 webserver.cc:469] Webserver started at http://127.15.25.126:37457/ using document root <none> and password file <none>
I20250622 13:58:43.655280 18089 fs_manager.cc:362] Metadata directory not provided
I20250622 13:58:43.655496 18089 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:58:43.655979 18089 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:58:43.660223 18089 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "a43815bcdf224b538f20db87e79bfe97"
format_stamp: "Formatted at 2025-06-22 13:58:43 on dist-test-slave-6tph"
I20250622 13:58:43.661311 18089 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "a43815bcdf224b538f20db87e79bfe97"
format_stamp: "Formatted at 2025-06-22 13:58:43 on dist-test-slave-6tph"
I20250622 13:58:43.668045 18089 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250622 13:58:43.673187 18105 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:43.674149 18089 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250622 13:58:43.674464 18089 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
uuid: "a43815bcdf224b538f20db87e79bfe97"
format_stamp: "Formatted at 2025-06-22 13:58:43 on dist-test-slave-6tph"
I20250622 13:58:43.674785 18089 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:58:43.729264 18089 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:58:43.730679 18089 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 13:58:43.731096 18089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:58:43.793504 18089 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.126:44459
I20250622 13:58:43.793561 18156 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.126:44459 every 8 connection(s)
I20250622 13:58:43.796101 18089 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250622 13:58:43.797917 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 18089
I20250622 13:58:43.798403 15461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250622 13:58:43.801481 18157 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:58:43.824057 18157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97: Bootstrap starting.
I20250622 13:58:43.828894 18157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97: Neither blocks nor log segments found. Creating new log.
I20250622 13:58:43.830560 18157 log.cc:826] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97: Log is configured to *not* fsync() on all Append() calls
I20250622 13:58:43.834528 18157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97: No bootstrap required, opened a new log
I20250622 13:58:43.851572 18157 raft_consensus.cc:357] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43815bcdf224b538f20db87e79bfe97" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 44459 } }
I20250622 13:58:43.852166 18157 raft_consensus.cc:383] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:58:43.852365 18157 raft_consensus.cc:738] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a43815bcdf224b538f20db87e79bfe97, State: Initialized, Role: FOLLOWER
I20250622 13:58:43.852927 18157 consensus_queue.cc:260] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43815bcdf224b538f20db87e79bfe97" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 44459 } }
I20250622 13:58:43.853374 18157 raft_consensus.cc:397] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 13:58:43.853582 18157 raft_consensus.cc:491] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 13:58:43.853892 18157 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:58:43.857513 18157 raft_consensus.cc:513] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43815bcdf224b538f20db87e79bfe97" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 44459 } }
I20250622 13:58:43.858129 18157 leader_election.cc:304] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a43815bcdf224b538f20db87e79bfe97; no voters:
I20250622 13:58:43.859549 18157 leader_election.cc:290] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 13:58:43.860236 18162 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:58:43.863073 18162 raft_consensus.cc:695] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [term 1 LEADER]: Becoming Leader. State: Replica: a43815bcdf224b538f20db87e79bfe97, State: Running, Role: LEADER
I20250622 13:58:43.863227 18157 sys_catalog.cc:564] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [sys.catalog]: configured and running, proceeding with master startup.
I20250622 13:58:43.863756 18162 consensus_queue.cc:237] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43815bcdf224b538f20db87e79bfe97" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 44459 } }
I20250622 13:58:43.873356 18164 sys_catalog.cc:455] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a43815bcdf224b538f20db87e79bfe97. Latest consensus state: current_term: 1 leader_uuid: "a43815bcdf224b538f20db87e79bfe97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43815bcdf224b538f20db87e79bfe97" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 44459 } } }
I20250622 13:58:43.874184 18164 sys_catalog.cc:458] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [sys.catalog]: This master's current role is: LEADER
I20250622 13:58:43.874326 18163 sys_catalog.cc:455] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a43815bcdf224b538f20db87e79bfe97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43815bcdf224b538f20db87e79bfe97" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 44459 } } }
I20250622 13:58:43.874926 18163 sys_catalog.cc:458] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [sys.catalog]: This master's current role is: LEADER
I20250622 13:58:43.878180 18172 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 13:58:43.890395 18172 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 13:58:43.905382 18172 catalog_manager.cc:1349] Generated new cluster ID: 2915a2c530594219b4d14e5bb2fb25a5
I20250622 13:58:43.905678 18172 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 13:58:43.954763 18172 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 13:58:43.956127 18172 catalog_manager.cc:1506] Loading token signing keys...
I20250622 13:58:43.966869 18172 catalog_manager.cc:5955] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97: Generated new TSK 0
I20250622 13:58:43.967793 18172 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 13:58:43.990501 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.25.65:0
--local_ip_for_outbound_sockets=127.15.25.65
--webserver_interface=127.15.25.65
--webserver_port=0
--tserver_master_addrs=127.15.25.126:44459
--builtin_ntp_servers=127.15.25.84:43331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250622 13:58:44.271414 18181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:58:44.271919 18181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 13:58:44.272411 18181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:58:44.302688 18181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 13:58:44.303472 18181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.25.65
I20250622 13:58:44.335938 18181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:43331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.25.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.15.25.65
--webserver_port=0
--tserver_master_addrs=127.15.25.126:44459
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.25.65
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 13:58:44.337131 18181 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 13:58:44.338552 18181 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:58:44.353938 18188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:58:44.357121 18181 server_base.cc:1048] running on GCE node
W20250622 13:58:44.356670 18190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:58:44.357201 18187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:58:45.440778 18181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 13:58:45.443404 18181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 13:58:45.444816 18181 hybrid_clock.cc:648] HybridClock initialized: now 1750600725444770 us; error 44 us; skew 500 ppm
I20250622 13:58:45.445909 18181 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:58:45.452911 18181 webserver.cc:469] Webserver started at http://127.15.25.65:42715/ using document root <none> and password file <none>
I20250622 13:58:45.454123 18181 fs_manager.cc:362] Metadata directory not provided
I20250622 13:58:45.454382 18181 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:58:45.454931 18181 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:58:45.460598 18181 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "2ca9b618349c4fa6a2bad70473011c13"
format_stamp: "Formatted at 2025-06-22 13:58:45 on dist-test-slave-6tph"
I20250622 13:58:45.461956 18181 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "2ca9b618349c4fa6a2bad70473011c13"
format_stamp: "Formatted at 2025-06-22 13:58:45 on dist-test-slave-6tph"
I20250622 13:58:45.470861 18181 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20250622 13:58:45.478106 18197 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:45.479265 18181 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250622 13:58:45.479596 18181 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "2ca9b618349c4fa6a2bad70473011c13"
format_stamp: "Formatted at 2025-06-22 13:58:45 on dist-test-slave-6tph"
I20250622 13:58:45.479964 18181 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:58:45.538280 18181 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:58:45.539647 18181 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 13:58:45.539992 18181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:58:45.542129 18181 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:58:45.545779 18181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:58:45.545948 18181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:45.546121 18181 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:58:45.546248 18181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:45.666090 18181 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.65:35893
I20250622 13:58:45.666152 18309 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.65:35893 every 8 connection(s)
I20250622 13:58:45.668385 18181 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250622 13:58:45.676642 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 18181
I20250622 13:58:45.676980 15461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250622 13:58:45.682469 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.25.66:0
--local_ip_for_outbound_sockets=127.15.25.66
--webserver_interface=127.15.25.66
--webserver_port=0
--tserver_master_addrs=127.15.25.126:44459
--builtin_ntp_servers=127.15.25.84:43331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 13:58:45.688253 18310 heartbeater.cc:344] Connected to a master server at 127.15.25.126:44459
I20250622 13:58:45.688715 18310 heartbeater.cc:461] Registering TS with master...
I20250622 13:58:45.689865 18310 heartbeater.cc:507] Master 127.15.25.126:44459 requested a full tablet report, sending...
I20250622 13:58:45.692436 18122 ts_manager.cc:194] Registered new tserver with Master: 2ca9b618349c4fa6a2bad70473011c13 (127.15.25.65:35893)
I20250622 13:58:45.694480 18122 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.25.65:44247
W20250622 13:58:45.960517 18314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:58:45.961076 18314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 13:58:45.961594 18314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:58:45.990851 18314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 13:58:45.991667 18314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.25.66
I20250622 13:58:46.024602 18314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:43331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.25.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.15.25.66
--webserver_port=0
--tserver_master_addrs=127.15.25.126:44459
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.25.66
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 13:58:46.025897 18314 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 13:58:46.027402 18314 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:58:46.043030 18320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:58:46.697968 18310 heartbeater.cc:499] Master 127.15.25.126:44459 was elected leader, sending a full tablet report...
W20250622 13:58:46.043439 18321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:58:46.043653 18323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:58:46.044296 18314 server_base.cc:1048] running on GCE node
I20250622 13:58:47.131503 18314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 13:58:47.133592 18314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 13:58:47.134960 18314 hybrid_clock.cc:648] HybridClock initialized: now 1750600727134934 us; error 46 us; skew 500 ppm
I20250622 13:58:47.135715 18314 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:58:47.141691 18314 webserver.cc:469] Webserver started at http://127.15.25.66:37569/ using document root <none> and password file <none>
I20250622 13:58:47.142527 18314 fs_manager.cc:362] Metadata directory not provided
I20250622 13:58:47.142716 18314 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:58:47.143095 18314 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:58:47.148384 18314 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "fda16503a5c54ead9c668695832bd6fd"
format_stamp: "Formatted at 2025-06-22 13:58:47 on dist-test-slave-6tph"
I20250622 13:58:47.149312 18314 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "fda16503a5c54ead9c668695832bd6fd"
format_stamp: "Formatted at 2025-06-22 13:58:47 on dist-test-slave-6tph"
I20250622 13:58:47.155432 18314 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20250622 13:58:47.159996 18330 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:47.160892 18314 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250622 13:58:47.161123 18314 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "fda16503a5c54ead9c668695832bd6fd"
format_stamp: "Formatted at 2025-06-22 13:58:47 on dist-test-slave-6tph"
I20250622 13:58:47.161406 18314 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:58:47.217288 18314 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:58:47.218616 18314 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 13:58:47.218971 18314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:58:47.221164 18314 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:58:47.224788 18314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:58:47.224973 18314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:47.225152 18314 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:58:47.225296 18314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:47.358527 18314 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.66:39531
I20250622 13:58:47.358623 18442 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.66:39531 every 8 connection(s)
I20250622 13:58:47.361099 18314 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250622 13:58:47.369508 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 18314
I20250622 13:58:47.369881 15461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250622 13:58:47.375869 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.25.67:0
--local_ip_for_outbound_sockets=127.15.25.67
--webserver_interface=127.15.25.67
--webserver_port=0
--tserver_master_addrs=127.15.25.126:44459
--builtin_ntp_servers=127.15.25.84:43331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 13:58:47.382504 18443 heartbeater.cc:344] Connected to a master server at 127.15.25.126:44459
I20250622 13:58:47.382977 18443 heartbeater.cc:461] Registering TS with master...
I20250622 13:58:47.383960 18443 heartbeater.cc:507] Master 127.15.25.126:44459 requested a full tablet report, sending...
I20250622 13:58:47.386092 18122 ts_manager.cc:194] Registered new tserver with Master: fda16503a5c54ead9c668695832bd6fd (127.15.25.66:39531)
I20250622 13:58:47.387878 18122 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.25.66:42009
W20250622 13:58:47.647652 18447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:58:47.648092 18447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 13:58:47.648494 18447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:58:47.685659 18447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 13:58:47.686461 18447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.25.67
I20250622 13:58:47.719167 18447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:43331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.25.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.15.25.67
--webserver_port=0
--tserver_master_addrs=127.15.25.126:44459
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.25.67
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 13:58:47.720366 18447 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 13:58:47.721771 18447 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:58:47.735114 18453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:58:48.391038 18443 heartbeater.cc:499] Master 127.15.25.126:44459 was elected leader, sending a full tablet report...
W20250622 13:58:47.735158 18454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:58:47.736668 18456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:58:48.815210 18455 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250622 13:58:48.815310 18447 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 13:58:48.818403 18447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 13:58:48.820297 18447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 13:58:48.821625 18447 hybrid_clock.cc:648] HybridClock initialized: now 1750600728821609 us; error 40 us; skew 500 ppm
I20250622 13:58:48.822327 18447 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:58:48.832113 18447 webserver.cc:469] Webserver started at http://127.15.25.67:43387/ using document root <none> and password file <none>
I20250622 13:58:48.832949 18447 fs_manager.cc:362] Metadata directory not provided
I20250622 13:58:48.833158 18447 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:58:48.833557 18447 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:58:48.837633 18447 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "30361482e92a4445867fb9c448c67ea4"
format_stamp: "Formatted at 2025-06-22 13:58:48 on dist-test-slave-6tph"
I20250622 13:58:48.838579 18447 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "30361482e92a4445867fb9c448c67ea4"
format_stamp: "Formatted at 2025-06-22 13:58:48 on dist-test-slave-6tph"
I20250622 13:58:48.844843 18447 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250622 13:58:48.849825 18463 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:48.850730 18447 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250622 13:58:48.851011 18447 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "30361482e92a4445867fb9c448c67ea4"
format_stamp: "Formatted at 2025-06-22 13:58:48 on dist-test-slave-6tph"
I20250622 13:58:48.851294 18447 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:58:48.894831 18447 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:58:48.896169 18447 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 13:58:48.896580 18447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:58:48.898663 18447 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:58:48.902143 18447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:58:48.902321 18447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:48.902537 18447 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:58:48.902675 18447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:49.032544 18447 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.67:37633
I20250622 13:58:49.032624 18575 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.67:37633 every 8 connection(s)
I20250622 13:58:49.035270 18447 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250622 13:58:49.039781 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 18447
I20250622 13:58:49.040125 15461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250622 13:58:49.053958 18576 heartbeater.cc:344] Connected to a master server at 127.15.25.126:44459
I20250622 13:58:49.054356 18576 heartbeater.cc:461] Registering TS with master...
I20250622 13:58:49.055239 18576 heartbeater.cc:507] Master 127.15.25.126:44459 requested a full tablet report, sending...
I20250622 13:58:49.057165 18122 ts_manager.cc:194] Registered new tserver with Master: 30361482e92a4445867fb9c448c67ea4 (127.15.25.67:37633)
I20250622 13:58:49.058636 18122 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.25.67:37565
I20250622 13:58:49.059523 15461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250622 13:58:49.086982 15461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250622 13:58:49.087282 15461 test_util.cc:276] Using random seed: 2124008737
I20250622 13:58:49.124143 18122 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44198:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250622 13:58:49.162353 18245 tablet_service.cc:1468] Processing CreateTablet for tablet 596f3292519e417bbd324a7a53c500da (DEFAULT_TABLE table=TestTable [id=5218e01f0a4a44cea87e3db2807ab3e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 13:58:49.163618 18245 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 596f3292519e417bbd324a7a53c500da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:58:49.182128 18596 tablet_bootstrap.cc:492] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13: Bootstrap starting.
I20250622 13:58:49.187039 18596 tablet_bootstrap.cc:654] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13: Neither blocks nor log segments found. Creating new log.
I20250622 13:58:49.188578 18596 log.cc:826] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13: Log is configured to *not* fsync() on all Append() calls
I20250622 13:58:49.192723 18596 tablet_bootstrap.cc:492] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13: No bootstrap required, opened a new log
I20250622 13:58:49.193079 18596 ts_tablet_manager.cc:1397] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20250622 13:58:49.208849 18596 raft_consensus.cc:357] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } }
I20250622 13:58:49.209362 18596 raft_consensus.cc:383] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:58:49.209568 18596 raft_consensus.cc:738] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca9b618349c4fa6a2bad70473011c13, State: Initialized, Role: FOLLOWER
I20250622 13:58:49.210278 18596 consensus_queue.cc:260] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } }
I20250622 13:58:49.210732 18596 raft_consensus.cc:397] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 13:58:49.211019 18596 raft_consensus.cc:491] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 13:58:49.211311 18596 raft_consensus.cc:3058] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:58:49.215117 18596 raft_consensus.cc:513] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } }
I20250622 13:58:49.215831 18596 leader_election.cc:304] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ca9b618349c4fa6a2bad70473011c13; no voters:
I20250622 13:58:49.217741 18596 leader_election.cc:290] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 13:58:49.218088 18598 raft_consensus.cc:2802] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:58:49.220001 18598 raft_consensus.cc:695] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [term 1 LEADER]: Becoming Leader. State: Replica: 2ca9b618349c4fa6a2bad70473011c13, State: Running, Role: LEADER
I20250622 13:58:49.220813 18598 consensus_queue.cc:237] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } }
I20250622 13:58:49.221249 18596 ts_tablet_manager.cc:1428] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13: Time spent starting tablet: real 0.028s user 0.024s sys 0.005s
I20250622 13:58:49.232214 18122 catalog_manager.cc:5582] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ca9b618349c4fa6a2bad70473011c13 (127.15.25.65). New cstate: current_term: 1 leader_uuid: "2ca9b618349c4fa6a2bad70473011c13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } health_report { overall_health: HEALTHY } } }
I20250622 13:58:49.406687 15461 test_util.cc:276] Using random seed: 2124328131
I20250622 13:58:49.424471 18114 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44200:
name: "TestTable1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250622 13:58:49.449272 18511 tablet_service.cc:1468] Processing CreateTablet for tablet 26064ec815564a6b85d47ef6799f11fa (DEFAULT_TABLE table=TestTable1 [id=7c33359e2f364c5a833382d7f706faed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 13:58:49.450616 18511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26064ec815564a6b85d47ef6799f11fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:58:49.468106 18617 tablet_bootstrap.cc:492] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4: Bootstrap starting.
I20250622 13:58:49.472972 18617 tablet_bootstrap.cc:654] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4: Neither blocks nor log segments found. Creating new log.
I20250622 13:58:49.474571 18617 log.cc:826] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4: Log is configured to *not* fsync() on all Append() calls
I20250622 13:58:49.478278 18617 tablet_bootstrap.cc:492] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4: No bootstrap required, opened a new log
I20250622 13:58:49.478614 18617 ts_tablet_manager.cc:1397] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20250622 13:58:49.493230 18617 raft_consensus.cc:357] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } }
I20250622 13:58:49.493654 18617 raft_consensus.cc:383] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:58:49.493903 18617 raft_consensus.cc:738] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30361482e92a4445867fb9c448c67ea4, State: Initialized, Role: FOLLOWER
I20250622 13:58:49.494536 18617 consensus_queue.cc:260] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } }
I20250622 13:58:49.494992 18617 raft_consensus.cc:397] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 13:58:49.495215 18617 raft_consensus.cc:491] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 13:58:49.495478 18617 raft_consensus.cc:3058] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:58:49.498878 18617 raft_consensus.cc:513] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } }
I20250622 13:58:49.499394 18617 leader_election.cc:304] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30361482e92a4445867fb9c448c67ea4; no voters:
I20250622 13:58:49.500918 18617 leader_election.cc:290] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 13:58:49.501293 18619 raft_consensus.cc:2802] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:58:49.503479 18619 raft_consensus.cc:695] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [term 1 LEADER]: Becoming Leader. State: Replica: 30361482e92a4445867fb9c448c67ea4, State: Running, Role: LEADER
I20250622 13:58:49.504225 18576 heartbeater.cc:499] Master 127.15.25.126:44459 was elected leader, sending a full tablet report...
I20250622 13:58:49.504437 18619 consensus_queue.cc:237] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } }
I20250622 13:58:49.505498 18617 ts_tablet_manager.cc:1428] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4: Time spent starting tablet: real 0.027s user 0.022s sys 0.004s
I20250622 13:58:49.515197 18114 catalog_manager.cc:5582] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30361482e92a4445867fb9c448c67ea4 (127.15.25.67). New cstate: current_term: 1 leader_uuid: "30361482e92a4445867fb9c448c67ea4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } health_report { overall_health: HEALTHY } } }
I20250622 13:58:49.681207 15461 test_util.cc:276] Using random seed: 2124602652
I20250622 13:58:49.704135 18117 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41064:
name: "TestTable2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250622 13:58:49.732944 18378 tablet_service.cc:1468] Processing CreateTablet for tablet 16f4a5fff92e4dc2be69fcfb8e4bbb22 (DEFAULT_TABLE table=TestTable2 [id=84edf2b470974bc58807a81b5258c066]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 13:58:49.734413 18378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16f4a5fff92e4dc2be69fcfb8e4bbb22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:58:49.752902 18638 tablet_bootstrap.cc:492] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd: Bootstrap starting.
I20250622 13:58:49.758502 18638 tablet_bootstrap.cc:654] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd: Neither blocks nor log segments found. Creating new log.
I20250622 13:58:49.760056 18638 log.cc:826] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd: Log is configured to *not* fsync() on all Append() calls
I20250622 13:58:49.763891 18638 tablet_bootstrap.cc:492] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd: No bootstrap required, opened a new log
I20250622 13:58:49.764315 18638 ts_tablet_manager.cc:1397] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20250622 13:58:49.780349 18638 raft_consensus.cc:357] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } }
I20250622 13:58:49.780810 18638 raft_consensus.cc:383] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:58:49.781030 18638 raft_consensus.cc:738] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fda16503a5c54ead9c668695832bd6fd, State: Initialized, Role: FOLLOWER
I20250622 13:58:49.781675 18638 consensus_queue.cc:260] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } }
I20250622 13:58:49.782171 18638 raft_consensus.cc:397] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 13:58:49.782382 18638 raft_consensus.cc:491] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 13:58:49.782606 18638 raft_consensus.cc:3058] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:58:49.786496 18638 raft_consensus.cc:513] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } }
I20250622 13:58:49.787101 18638 leader_election.cc:304] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fda16503a5c54ead9c668695832bd6fd; no voters:
I20250622 13:58:49.788625 18638 leader_election.cc:290] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 13:58:49.788945 18640 raft_consensus.cc:2802] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:58:49.790910 18640 raft_consensus.cc:695] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd [term 1 LEADER]: Becoming Leader. State: Replica: fda16503a5c54ead9c668695832bd6fd, State: Running, Role: LEADER
I20250622 13:58:49.791793 18640 consensus_queue.cc:237] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } }
I20250622 13:58:49.798125 18638 ts_tablet_manager.cc:1428] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd: Time spent starting tablet: real 0.034s user 0.028s sys 0.005s
I20250622 13:58:49.803843 18117 catalog_manager.cc:5582] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd reported cstate change: term changed from 0 to 1, leader changed from <none> to fda16503a5c54ead9c668695832bd6fd (127.15.25.66). New cstate: current_term: 1 leader_uuid: "fda16503a5c54ead9c668695832bd6fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } health_report { overall_health: HEALTHY } } }
I20250622 13:58:49.989665 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 18089
W20250622 13:58:50.245826 18310 heartbeater.cc:646] Failed to heartbeat to 127.15.25.126:44459 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.25.126:44459: connect: Connection refused (error 111)
W20250622 13:58:50.530516 18576 heartbeater.cc:646] Failed to heartbeat to 127.15.25.126:44459 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.25.126:44459: connect: Connection refused (error 111)
W20250622 13:58:50.822191 18443 heartbeater.cc:646] Failed to heartbeat to 127.15.25.126:44459 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.25.126:44459: connect: Connection refused (error 111)
I20250622 13:58:54.505254 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 18181
I20250622 13:58:54.528617 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 18314
I20250622 13:58:54.556546 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 18447
I20250622 13:58:54.584005 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.25.126:44459
--webserver_interface=127.15.25.126
--webserver_port=37457
--builtin_ntp_servers=127.15.25.84:43331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.25.126:44459 with env {}
W20250622 13:58:54.858309 18718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:58:54.858834 18718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 13:58:54.859246 18718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:58:54.887629 18718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250622 13:58:54.887888 18718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 13:58:54.888147 18718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250622 13:58:54.888370 18718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250622 13:58:54.920544 18718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:43331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.25.126:44459
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.25.126:44459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.25.126
--webserver_port=37457
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 13:58:54.921854 18718 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 13:58:54.923453 18718 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:58:54.939059 18724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:58:54.939495 18725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:58:54.939766 18727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:58:56.045400 18726 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250622 13:58:56.045475 18718 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 13:58:56.048806 18718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 13:58:56.051265 18718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 13:58:56.052668 18718 hybrid_clock.cc:648] HybridClock initialized: now 1750600736052628 us; error 55 us; skew 500 ppm
I20250622 13:58:56.053475 18718 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:58:56.064116 18718 webserver.cc:469] Webserver started at http://127.15.25.126:37457/ using document root <none> and password file <none>
I20250622 13:58:56.065021 18718 fs_manager.cc:362] Metadata directory not provided
I20250622 13:58:56.065284 18718 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:58:56.072187 18718 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20250622 13:58:56.076354 18734 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:56.077265 18718 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250622 13:58:56.077540 18718 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
uuid: "a43815bcdf224b538f20db87e79bfe97"
format_stamp: "Formatted at 2025-06-22 13:58:43 on dist-test-slave-6tph"
I20250622 13:58:56.079231 18718 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:58:56.148517 18718 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:58:56.149875 18718 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 13:58:56.150285 18718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:58:56.216941 18718 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.126:44459
I20250622 13:58:56.217012 18785 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.126:44459 every 8 connection(s)
I20250622 13:58:56.219542 18718 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250622 13:58:56.229226 18786 sys_catalog.cc:263] Verifying existing consensus state
I20250622 13:58:56.229460 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 18718
I20250622 13:58:56.230921 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.25.65:35893
--local_ip_for_outbound_sockets=127.15.25.65
--tserver_master_addrs=127.15.25.126:44459
--webserver_port=42715
--webserver_interface=127.15.25.65
--builtin_ntp_servers=127.15.25.84:43331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 13:58:56.234079 18786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97: Bootstrap starting.
I20250622 13:58:56.244541 18786 log.cc:826] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97: Log is configured to *not* fsync() on all Append() calls
I20250622 13:58:56.290326 18786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=13 ignored=0} mutations{seen=10 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:58:56.291111 18786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97: Bootstrap complete.
I20250622 13:58:56.311321 18786 raft_consensus.cc:357] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43815bcdf224b538f20db87e79bfe97" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 44459 } }
I20250622 13:58:56.313349 18786 raft_consensus.cc:738] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a43815bcdf224b538f20db87e79bfe97, State: Initialized, Role: FOLLOWER
I20250622 13:58:56.314100 18786 consensus_queue.cc:260] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43815bcdf224b538f20db87e79bfe97" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 44459 } }
I20250622 13:58:56.314599 18786 raft_consensus.cc:397] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 13:58:56.314862 18786 raft_consensus.cc:491] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 13:58:56.315147 18786 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [term 2 FOLLOWER]: Advancing to term 3
I20250622 13:58:56.320281 18786 raft_consensus.cc:513] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43815bcdf224b538f20db87e79bfe97" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 44459 } }
I20250622 13:58:56.320829 18786 leader_election.cc:304] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a43815bcdf224b538f20db87e79bfe97; no voters:
I20250622 13:58:56.322882 18786 leader_election.cc:290] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250622 13:58:56.323354 18790 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [term 3 FOLLOWER]: Leader election won for term 3
I20250622 13:58:56.326563 18790 raft_consensus.cc:695] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [term 3 LEADER]: Becoming Leader. State: Replica: a43815bcdf224b538f20db87e79bfe97, State: Running, Role: LEADER
I20250622 13:58:56.327346 18790 consensus_queue.cc:237] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43815bcdf224b538f20db87e79bfe97" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 44459 } }
I20250622 13:58:56.327733 18786 sys_catalog.cc:564] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [sys.catalog]: configured and running, proceeding with master startup.
I20250622 13:58:56.337175 18792 sys_catalog.cc:455] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a43815bcdf224b538f20db87e79bfe97. Latest consensus state: current_term: 3 leader_uuid: "a43815bcdf224b538f20db87e79bfe97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43815bcdf224b538f20db87e79bfe97" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 44459 } } }
I20250622 13:58:56.337839 18792 sys_catalog.cc:458] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [sys.catalog]: This master's current role is: LEADER
I20250622 13:58:56.339376 18791 sys_catalog.cc:455] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a43815bcdf224b538f20db87e79bfe97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43815bcdf224b538f20db87e79bfe97" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 44459 } } }
I20250622 13:58:56.340790 18791 sys_catalog.cc:458] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [sys.catalog]: This master's current role is: LEADER
I20250622 13:58:56.345650 18797 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 13:58:56.357987 18797 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=02798003f4eb4716956cd12c400e655f]
I20250622 13:58:56.359450 18797 catalog_manager.cc:671] Loaded metadata for table TestTable [id=691aa902fb6748a5a6b739bd2efbe493]
I20250622 13:58:56.360874 18797 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=84edf2b470974bc58807a81b5258c066]
I20250622 13:58:56.367883 18797 tablet_loader.cc:96] loaded metadata for tablet 16f4a5fff92e4dc2be69fcfb8e4bbb22 (table TestTable2 [id=84edf2b470974bc58807a81b5258c066])
I20250622 13:58:56.369686 18797 tablet_loader.cc:96] loaded metadata for tablet 26064ec815564a6b85d47ef6799f11fa (table TestTable1 [id=02798003f4eb4716956cd12c400e655f])
I20250622 13:58:56.370779 18797 tablet_loader.cc:96] loaded metadata for tablet 596f3292519e417bbd324a7a53c500da (table TestTable [id=691aa902fb6748a5a6b739bd2efbe493])
I20250622 13:58:56.372171 18797 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 13:58:56.376965 18797 catalog_manager.cc:1261] Loaded cluster ID: 2915a2c530594219b4d14e5bb2fb25a5
I20250622 13:58:56.377233 18797 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 13:58:56.384373 18797 catalog_manager.cc:1506] Loading token signing keys...
I20250622 13:58:56.389168 18797 catalog_manager.cc:5966] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97: Loaded TSK: 0
I20250622 13:58:56.390650 18797 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250622 13:58:56.554289 18788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:58:56.554787 18788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 13:58:56.555264 18788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:58:56.585677 18788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 13:58:56.586658 18788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.25.65
I20250622 13:58:56.619024 18788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:43331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.25.65:35893
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.15.25.65
--webserver_port=42715
--tserver_master_addrs=127.15.25.126:44459
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.25.65
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 13:58:56.620283 18788 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 13:58:56.621903 18788 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:58:56.638691 18814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:58:56.641098 18816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:58:56.641566 18788 server_base.cc:1048] running on GCE node
W20250622 13:58:56.641489 18813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:58:57.747349 18788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 13:58:57.750046 18788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 13:58:57.751528 18788 hybrid_clock.cc:648] HybridClock initialized: now 1750600737751494 us; error 40 us; skew 500 ppm
I20250622 13:58:57.752522 18788 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:58:57.759907 18788 webserver.cc:469] Webserver started at http://127.15.25.65:42715/ using document root <none> and password file <none>
I20250622 13:58:57.760983 18788 fs_manager.cc:362] Metadata directory not provided
I20250622 13:58:57.761240 18788 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:58:57.771183 18788 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.004s sys 0.004s
I20250622 13:58:57.778287 18823 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:57.779347 18788 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250622 13:58:57.779742 18788 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "2ca9b618349c4fa6a2bad70473011c13"
format_stamp: "Formatted at 2025-06-22 13:58:45 on dist-test-slave-6tph"
I20250622 13:58:57.782155 18788 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:58:57.841257 18788 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:58:57.843011 18788 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 13:58:57.843482 18788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:58:57.846521 18788 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:58:57.853085 18830 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250622 13:58:57.860033 18788 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250622 13:58:57.860284 18788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.001s sys 0.001s
I20250622 13:58:57.860601 18788 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250622 13:58:57.867427 18788 ts_tablet_manager.cc:610] Registered 1 tablets
I20250622 13:58:57.867655 18788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.007s user 0.006s sys 0.000s
I20250622 13:58:57.867931 18830 tablet_bootstrap.cc:492] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13: Bootstrap starting.
I20250622 13:58:57.931740 18830 log.cc:826] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13: Log is configured to *not* fsync() on all Append() calls
I20250622 13:58:58.037060 18830 tablet_bootstrap.cc:492] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:58:58.038110 18830 tablet_bootstrap.cc:492] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13: Bootstrap complete.
I20250622 13:58:58.040766 18830 ts_tablet_manager.cc:1397] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13: Time spent bootstrapping tablet: real 0.173s user 0.126s sys 0.040s
I20250622 13:58:58.050500 18788 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.65:35893
I20250622 13:58:58.050794 18937 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.65:35893 every 8 connection(s)
I20250622 13:58:58.053869 18788 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250622 13:58:58.061269 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 18788
I20250622 13:58:58.059880 18830 raft_consensus.cc:357] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } }
I20250622 13:58:58.062749 18830 raft_consensus.cc:738] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca9b618349c4fa6a2bad70473011c13, State: Initialized, Role: FOLLOWER
I20250622 13:58:58.062948 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.25.66:39531
--local_ip_for_outbound_sockets=127.15.25.66
--tserver_master_addrs=127.15.25.126:44459
--webserver_port=37569
--webserver_interface=127.15.25.66
--builtin_ntp_servers=127.15.25.84:43331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 13:58:58.063668 18830 consensus_queue.cc:260] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } }
I20250622 13:58:58.064265 18830 raft_consensus.cc:397] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 13:58:58.064621 18830 raft_consensus.cc:491] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 13:58:58.064996 18830 raft_consensus.cc:3058] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:58:58.075994 18830 raft_consensus.cc:513] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } }
I20250622 13:58:58.077047 18830 leader_election.cc:304] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ca9b618349c4fa6a2bad70473011c13; no voters:
I20250622 13:58:58.095007 18830 leader_election.cc:290] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250622 13:58:58.095376 18938 heartbeater.cc:344] Connected to a master server at 127.15.25.126:44459
I20250622 13:58:58.095852 18938 heartbeater.cc:461] Registering TS with master...
I20250622 13:58:58.096071 18943 raft_consensus.cc:2802] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [term 2 FOLLOWER]: Leader election won for term 2
I20250622 13:58:58.097117 18938 heartbeater.cc:507] Master 127.15.25.126:44459 requested a full tablet report, sending...
I20250622 13:58:58.104228 18830 ts_tablet_manager.cc:1428] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13: Time spent starting tablet: real 0.063s user 0.041s sys 0.019s
I20250622 13:58:58.106205 18943 raft_consensus.cc:695] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [term 2 LEADER]: Becoming Leader. State: Replica: 2ca9b618349c4fa6a2bad70473011c13, State: Running, Role: LEADER
I20250622 13:58:58.106952 18943 consensus_queue.cc:237] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } }
I20250622 13:58:58.122774 18751 ts_manager.cc:194] Registered new tserver with Master: 2ca9b618349c4fa6a2bad70473011c13 (127.15.25.65:35893)
I20250622 13:58:58.126649 18751 catalog_manager.cc:5582] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 reported cstate change: term changed from 0 to 2, leader changed from <none> to 2ca9b618349c4fa6a2bad70473011c13 (127.15.25.65), VOTER 2ca9b618349c4fa6a2bad70473011c13 (127.15.25.65) added. New cstate: current_term: 2 leader_uuid: "2ca9b618349c4fa6a2bad70473011c13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } health_report { overall_health: HEALTHY } } }
W20250622 13:58:58.175094 18751 catalog_manager.cc:5260] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 596f3292519e417bbd324a7a53c500da with cas_config_opid_index -1: no extra replica candidate found for tablet 596f3292519e417bbd324a7a53c500da (table TestTable [id=691aa902fb6748a5a6b739bd2efbe493]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (1) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
I20250622 13:58:58.177233 18751 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.25.65:46301
I20250622 13:58:58.181613 18938 heartbeater.cc:499] Master 127.15.25.126:44459 was elected leader, sending a full tablet report...
W20250622 13:58:58.395663 18942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:58:58.396283 18942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 13:58:58.396939 18942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:58:58.428395 18942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 13:58:58.429129 18942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.25.66
I20250622 13:58:58.463816 18942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:43331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.25.66:39531
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.15.25.66
--webserver_port=37569
--tserver_master_addrs=127.15.25.126:44459
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.25.66
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 13:58:58.465070 18942 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 13:58:58.466647 18942 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:58:58.483374 18958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:58:58.483417 18960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:58:58.483441 18957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:58:59.577069 18959 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1091 milliseconds
I20250622 13:58:59.577171 18942 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 13:58:59.578307 18942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 13:58:59.580826 18942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 13:58:59.582214 18942 hybrid_clock.cc:648] HybridClock initialized: now 1750600739582181 us; error 53 us; skew 500 ppm
I20250622 13:58:59.582947 18942 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:58:59.589660 18942 webserver.cc:469] Webserver started at http://127.15.25.66:37569/ using document root <none> and password file <none>
I20250622 13:58:59.590544 18942 fs_manager.cc:362] Metadata directory not provided
I20250622 13:58:59.590734 18942 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:58:59.598062 18942 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.001s
I20250622 13:58:59.602245 18967 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:58:59.603188 18942 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20250622 13:58:59.603492 18942 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "fda16503a5c54ead9c668695832bd6fd"
format_stamp: "Formatted at 2025-06-22 13:58:47 on dist-test-slave-6tph"
I20250622 13:58:59.605343 18942 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:58:59.668043 18942 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:58:59.669661 18942 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 13:58:59.670095 18942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:58:59.672559 18942 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:58:59.678037 18974 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250622 13:58:59.685954 18942 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250622 13:58:59.686206 18942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.002s
I20250622 13:58:59.686487 18942 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250622 13:58:59.691169 18942 ts_tablet_manager.cc:610] Registered 1 tablets
I20250622 13:58:59.691385 18942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.003s sys 0.001s
I20250622 13:58:59.691766 18974 tablet_bootstrap.cc:492] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd: Bootstrap starting.
I20250622 13:58:59.750021 18974 log.cc:826] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd: Log is configured to *not* fsync() on all Append() calls
I20250622 13:58:59.846894 18942 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.66:39531
I20250622 13:58:59.847000 19081 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.66:39531 every 8 connection(s)
I20250622 13:58:59.849993 18942 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250622 13:58:59.859164 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 18942
I20250622 13:58:59.860788 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.25.67:37633
--local_ip_for_outbound_sockets=127.15.25.67
--tserver_master_addrs=127.15.25.126:44459
--webserver_port=43387
--webserver_interface=127.15.25.67
--builtin_ntp_servers=127.15.25.84:43331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 13:58:59.870895 18974 tablet_bootstrap.cc:492] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:58:59.871858 18974 tablet_bootstrap.cc:492] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd: Bootstrap complete.
I20250622 13:58:59.873279 18974 ts_tablet_manager.cc:1397] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd: Time spent bootstrapping tablet: real 0.182s user 0.145s sys 0.032s
I20250622 13:58:59.873788 19082 heartbeater.cc:344] Connected to a master server at 127.15.25.126:44459
I20250622 13:58:59.874290 19082 heartbeater.cc:461] Registering TS with master...
I20250622 13:58:59.875480 19082 heartbeater.cc:507] Master 127.15.25.126:44459 requested a full tablet report, sending...
I20250622 13:58:59.879179 18751 ts_manager.cc:194] Registered new tserver with Master: fda16503a5c54ead9c668695832bd6fd (127.15.25.66:39531)
I20250622 13:58:59.882651 18751 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.25.66:60485
I20250622 13:58:59.886757 18974 raft_consensus.cc:357] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } }
I20250622 13:58:59.888532 18974 raft_consensus.cc:738] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fda16503a5c54ead9c668695832bd6fd, State: Initialized, Role: FOLLOWER
I20250622 13:58:59.889073 18974 consensus_queue.cc:260] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } }
I20250622 13:58:59.889544 18974 raft_consensus.cc:397] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 13:58:59.889786 18974 raft_consensus.cc:491] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 13:58:59.890033 18974 raft_consensus.cc:3058] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:58:59.894840 18974 raft_consensus.cc:513] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } }
I20250622 13:58:59.895418 18974 leader_election.cc:304] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fda16503a5c54ead9c668695832bd6fd; no voters:
I20250622 13:58:59.896953 18974 leader_election.cc:290] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd [CANDIDATE]: Term 2 election: Requested vote from peers
I20250622 13:58:59.897362 19087 raft_consensus.cc:2802] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd [term 2 FOLLOWER]: Leader election won for term 2
I20250622 13:58:59.899792 19082 heartbeater.cc:499] Master 127.15.25.126:44459 was elected leader, sending a full tablet report...
I20250622 13:58:59.900218 18974 ts_tablet_manager.cc:1428] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd: Time spent starting tablet: real 0.027s user 0.017s sys 0.010s
I20250622 13:58:59.900816 19087 raft_consensus.cc:695] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd [term 2 LEADER]: Becoming Leader. State: Replica: fda16503a5c54ead9c668695832bd6fd, State: Running, Role: LEADER
I20250622 13:58:59.901484 19087 consensus_queue.cc:237] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } }
I20250622 13:58:59.912271 18751 catalog_manager.cc:5582] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "fda16503a5c54ead9c668695832bd6fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } health_report { overall_health: HEALTHY } } }
W20250622 13:59:00.154665 19086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:59:00.155133 19086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 13:59:00.155580 19086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:59:00.186623 19086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 13:59:00.187434 19086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.25.67
I20250622 13:59:00.221938 19086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:43331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.25.67:37633
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.15.25.67
--webserver_port=43387
--tserver_master_addrs=127.15.25.126:44459
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.25.67
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 13:59:00.223259 19086 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 13:59:00.224838 19086 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:59:00.240854 19102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:59:00.492548 18893 consensus_queue.cc:237] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: NON_VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: true } }
I20250622 13:59:00.499759 19110 raft_consensus.cc:2953] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index -1 to 8, NON_VOTER fda16503a5c54ead9c668695832bd6fd (127.15.25.66) added. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: NON_VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: true } } }
I20250622 13:59:00.515029 18736 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 596f3292519e417bbd324a7a53c500da with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 8)
I20250622 13:59:00.515242 18751 catalog_manager.cc:5582] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 reported cstate change: config changed from index -1 to 8, NON_VOTER fda16503a5c54ead9c668695832bd6fd (127.15.25.66) added. New cstate: current_term: 2 leader_uuid: "2ca9b618349c4fa6a2bad70473011c13" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: NON_VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250622 13:59:00.528985 18824 consensus_peers.cc:487] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 -> Peer fda16503a5c54ead9c668695832bd6fd (127.15.25.66:39531): Couldn't send request to peer fda16503a5c54ead9c668695832bd6fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 596f3292519e417bbd324a7a53c500da. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:59:01.026017 19118 ts_tablet_manager.cc:927] T 596f3292519e417bbd324a7a53c500da P fda16503a5c54ead9c668695832bd6fd: Initiating tablet copy from peer 2ca9b618349c4fa6a2bad70473011c13 (127.15.25.65:35893)
I20250622 13:59:01.028002 19118 tablet_copy_client.cc:323] T 596f3292519e417bbd324a7a53c500da P fda16503a5c54ead9c668695832bd6fd: tablet copy: Beginning tablet copy session from remote peer at address 127.15.25.65:35893
I20250622 13:59:01.038877 18913 tablet_copy_service.cc:140] P 2ca9b618349c4fa6a2bad70473011c13: Received BeginTabletCopySession request for tablet 596f3292519e417bbd324a7a53c500da from peer fda16503a5c54ead9c668695832bd6fd ({username='slave'} at 127.15.25.66:46577)
I20250622 13:59:01.039593 18913 tablet_copy_service.cc:161] P 2ca9b618349c4fa6a2bad70473011c13: Beginning new tablet copy session on tablet 596f3292519e417bbd324a7a53c500da from peer fda16503a5c54ead9c668695832bd6fd at {username='slave'} at 127.15.25.66:46577: session id = fda16503a5c54ead9c668695832bd6fd-596f3292519e417bbd324a7a53c500da
I20250622 13:59:01.049973 18913 tablet_copy_source_session.cc:215] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 13:59:01.055928 19118 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 596f3292519e417bbd324a7a53c500da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:59:01.077636 19118 tablet_copy_client.cc:806] T 596f3292519e417bbd324a7a53c500da P fda16503a5c54ead9c668695832bd6fd: tablet copy: Starting download of 0 data blocks...
I20250622 13:59:01.078373 19118 tablet_copy_client.cc:670] T 596f3292519e417bbd324a7a53c500da P fda16503a5c54ead9c668695832bd6fd: tablet copy: Starting download of 1 WAL segments...
I20250622 13:59:01.082926 19118 tablet_copy_client.cc:538] T 596f3292519e417bbd324a7a53c500da P fda16503a5c54ead9c668695832bd6fd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 13:59:01.094534 19118 tablet_bootstrap.cc:492] T 596f3292519e417bbd324a7a53c500da P fda16503a5c54ead9c668695832bd6fd: Bootstrap starting.
I20250622 13:59:01.230245 19118 tablet_bootstrap.cc:492] T 596f3292519e417bbd324a7a53c500da P fda16503a5c54ead9c668695832bd6fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:59:01.231117 19118 tablet_bootstrap.cc:492] T 596f3292519e417bbd324a7a53c500da P fda16503a5c54ead9c668695832bd6fd: Bootstrap complete.
I20250622 13:59:01.231662 19118 ts_tablet_manager.cc:1397] T 596f3292519e417bbd324a7a53c500da P fda16503a5c54ead9c668695832bd6fd: Time spent bootstrapping tablet: real 0.137s user 0.111s sys 0.025s
I20250622 13:59:01.233932 19118 raft_consensus.cc:357] T 596f3292519e417bbd324a7a53c500da P fda16503a5c54ead9c668695832bd6fd [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: NON_VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: true } }
I20250622 13:59:01.234499 19118 raft_consensus.cc:738] T 596f3292519e417bbd324a7a53c500da P fda16503a5c54ead9c668695832bd6fd [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: fda16503a5c54ead9c668695832bd6fd, State: Initialized, Role: LEARNER
I20250622 13:59:01.235028 19118 consensus_queue.cc:260] T 596f3292519e417bbd324a7a53c500da P fda16503a5c54ead9c668695832bd6fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: NON_VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: true } }
I20250622 13:59:01.238840 19118 ts_tablet_manager.cc:1428] T 596f3292519e417bbd324a7a53c500da P fda16503a5c54ead9c668695832bd6fd: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250622 13:59:01.243093 18913 tablet_copy_service.cc:342] P 2ca9b618349c4fa6a2bad70473011c13: Request end of tablet copy session fda16503a5c54ead9c668695832bd6fd-596f3292519e417bbd324a7a53c500da received from {username='slave'} at 127.15.25.66:46577
I20250622 13:59:01.243505 18913 tablet_copy_service.cc:434] P 2ca9b618349c4fa6a2bad70473011c13: ending tablet copy session fda16503a5c54ead9c668695832bd6fd-596f3292519e417bbd324a7a53c500da on tablet 596f3292519e417bbd324a7a53c500da with peer fda16503a5c54ead9c668695832bd6fd
W20250622 13:59:00.242086 19101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:59:00.243351 19104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:59:00.243271 19086 server_base.cc:1048] running on GCE node
I20250622 13:59:01.396099 19086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 13:59:01.398541 19086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 13:59:01.399917 19086 hybrid_clock.cc:648] HybridClock initialized: now 1750600741399886 us; error 51 us; skew 500 ppm
I20250622 13:59:01.400605 19086 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:59:01.411540 19086 webserver.cc:469] Webserver started at http://127.15.25.67:43387/ using document root <none> and password file <none>
I20250622 13:59:01.412408 19086 fs_manager.cc:362] Metadata directory not provided
I20250622 13:59:01.412616 19086 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:59:01.419696 19086 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20250622 13:59:01.423804 19128 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:59:01.424741 19086 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250622 13:59:01.425015 19086 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "30361482e92a4445867fb9c448c67ea4"
format_stamp: "Formatted at 2025-06-22 13:58:48 on dist-test-slave-6tph"
I20250622 13:59:01.426875 19086 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:59:01.479105 19037 raft_consensus.cc:1215] T 596f3292519e417bbd324a7a53c500da P fda16503a5c54ead9c668695832bd6fd [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20250622 13:59:01.483688 19086 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:59:01.485421 19086 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 13:59:01.485848 19086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:59:01.488029 19086 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:59:01.493381 19137 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250622 13:59:01.500183 19086 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250622 13:59:01.500401 19086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.002s sys 0.000s
I20250622 13:59:01.500638 19086 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250622 13:59:01.505064 19086 ts_tablet_manager.cc:610] Registered 1 tablets
I20250622 13:59:01.505249 19086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.002s sys 0.000s
I20250622 13:59:01.505595 19137 tablet_bootstrap.cc:492] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4: Bootstrap starting.
I20250622 13:59:01.557066 19137 log.cc:826] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4: Log is configured to *not* fsync() on all Append() calls
I20250622 13:59:01.624876 19137 tablet_bootstrap.cc:492] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:59:01.625955 19137 tablet_bootstrap.cc:492] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4: Bootstrap complete.
I20250622 13:59:01.627692 19137 ts_tablet_manager.cc:1397] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4: Time spent bootstrapping tablet: real 0.122s user 0.076s sys 0.042s
I20250622 13:59:01.645313 19137 raft_consensus.cc:357] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } }
I20250622 13:59:01.647245 19137 raft_consensus.cc:738] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30361482e92a4445867fb9c448c67ea4, State: Initialized, Role: FOLLOWER
I20250622 13:59:01.647985 19137 consensus_queue.cc:260] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } }
I20250622 13:59:01.648408 19137 raft_consensus.cc:397] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 13:59:01.648653 19137 raft_consensus.cc:491] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 13:59:01.649017 19137 raft_consensus.cc:3058] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [term 1 FOLLOWER]: Advancing to term 2
I20250622 13:59:01.655628 19137 raft_consensus.cc:513] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } }
I20250622 13:59:01.656529 19137 leader_election.cc:304] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30361482e92a4445867fb9c448c67ea4; no voters:
I20250622 13:59:01.659174 19137 leader_election.cc:290] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250622 13:59:01.659610 19232 raft_consensus.cc:2802] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [term 2 FOLLOWER]: Leader election won for term 2
I20250622 13:59:01.662467 19232 raft_consensus.cc:695] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [term 2 LEADER]: Becoming Leader. State: Replica: 30361482e92a4445867fb9c448c67ea4, State: Running, Role: LEADER
I20250622 13:59:01.663480 19232 consensus_queue.cc:237] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } }
I20250622 13:59:01.665067 19137 ts_tablet_manager.cc:1428] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4: Time spent starting tablet: real 0.037s user 0.025s sys 0.012s
I20250622 13:59:01.679610 19086 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.67:37633
I20250622 13:59:01.680145 19249 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.67:37633 every 8 connection(s)
I20250622 13:59:01.681977 19086 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250622 13:59:01.687412 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 19086
I20250622 13:59:01.700007 19250 heartbeater.cc:344] Connected to a master server at 127.15.25.126:44459
I20250622 13:59:01.700453 19250 heartbeater.cc:461] Registering TS with master...
I20250622 13:59:01.701301 19250 heartbeater.cc:507] Master 127.15.25.126:44459 requested a full tablet report, sending...
I20250622 13:59:01.704290 18750 ts_manager.cc:194] Registered new tserver with Master: 30361482e92a4445867fb9c448c67ea4 (127.15.25.67:37633)
I20250622 13:59:01.705183 18750 catalog_manager.cc:5582] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 reported cstate change: term changed from 0 to 2, leader changed from <none> to 30361482e92a4445867fb9c448c67ea4 (127.15.25.67), VOTER 30361482e92a4445867fb9c448c67ea4 (127.15.25.67) added. New cstate: current_term: 2 leader_uuid: "30361482e92a4445867fb9c448c67ea4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } health_report { overall_health: HEALTHY } } }
I20250622 13:59:01.717105 15461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250622 13:59:01.720068 18750 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.25.67:52785
I20250622 13:59:01.722213 15461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250622 13:59:01.724140 19250 heartbeater.cc:499] Master 127.15.25.126:44459 was elected leader, sending a full tablet report...
W20250622 13:59:01.725303 15461 ts_itest-base.cc:209] found only 2 out of 3 replicas of tablet 596f3292519e417bbd324a7a53c500da: tablet_id: "596f3292519e417bbd324a7a53c500da" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER } interned_replicas { ts_info_idx: 1 role: LEARNER }
I20250622 13:59:01.734331 19200 consensus_queue.cc:237] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } } peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: NON_VOTER last_known_addr { host: "127.15.25.65" port: 35893 } attrs { promote: true } }
I20250622 13:59:01.737440 19235 raft_consensus.cc:2953] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index -1 to 8, NON_VOTER 2ca9b618349c4fa6a2bad70473011c13 (127.15.25.65) added. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } } peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: NON_VOTER last_known_addr { host: "127.15.25.65" port: 35893 } attrs { promote: true } } }
I20250622 13:59:01.743901 18735 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 26064ec815564a6b85d47ef6799f11fa with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250622 13:59:01.746387 19130 consensus_peers.cc:487] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 -> Peer 2ca9b618349c4fa6a2bad70473011c13 (127.15.25.65:35893): Couldn't send request to peer 2ca9b618349c4fa6a2bad70473011c13. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 26064ec815564a6b85d47ef6799f11fa. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:59:01.746918 18751 catalog_manager.cc:5582] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 reported cstate change: config changed from index -1 to 8, NON_VOTER 2ca9b618349c4fa6a2bad70473011c13 (127.15.25.65) added. New cstate: current_term: 2 leader_uuid: "30361482e92a4445867fb9c448c67ea4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: NON_VOTER last_known_addr { host: "127.15.25.65" port: 35893 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250622 13:59:01.754832 19200 consensus_queue.cc:237] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } } peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: NON_VOTER last_known_addr { host: "127.15.25.65" port: 35893 } attrs { promote: true } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: NON_VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: true } }
I20250622 13:59:01.757779 19236 raft_consensus.cc:2953] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER fda16503a5c54ead9c668695832bd6fd (127.15.25.66) added. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } } peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: NON_VOTER last_known_addr { host: "127.15.25.65" port: 35893 } attrs { promote: true } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: NON_VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: true } } }
W20250622 13:59:01.758997 19130 consensus_peers.cc:487] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 -> Peer 2ca9b618349c4fa6a2bad70473011c13 (127.15.25.65:35893): Couldn't send request to peer 2ca9b618349c4fa6a2bad70473011c13. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 26064ec815564a6b85d47ef6799f11fa. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:59:01.765385 18735 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 26064ec815564a6b85d47ef6799f11fa with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250622 13:59:01.767479 19129 consensus_peers.cc:487] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 -> Peer fda16503a5c54ead9c668695832bd6fd (127.15.25.66:39531): Couldn't send request to peer fda16503a5c54ead9c668695832bd6fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 26064ec815564a6b85d47ef6799f11fa. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:59:01.768289 18751 catalog_manager.cc:5582] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 reported cstate change: config changed from index 8 to 9, NON_VOTER fda16503a5c54ead9c668695832bd6fd (127.15.25.66) added. New cstate: current_term: 2 leader_uuid: "30361482e92a4445867fb9c448c67ea4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: NON_VOTER last_known_addr { host: "127.15.25.65" port: 35893 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: NON_VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250622 13:59:01.997815 19265 raft_consensus.cc:1062] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13: attempting to promote NON_VOTER fda16503a5c54ead9c668695832bd6fd to VOTER
I20250622 13:59:01.998972 19265 consensus_queue.cc:237] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } }
I20250622 13:59:02.002480 19037 raft_consensus.cc:1273] T 596f3292519e417bbd324a7a53c500da P fda16503a5c54ead9c668695832bd6fd [term 2 LEARNER]: Refusing update from remote peer 2ca9b618349c4fa6a2bad70473011c13: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20250622 13:59:02.003759 19266 consensus_queue.cc:1035] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250622 13:59:02.009455 19265 raft_consensus.cc:2953] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, fda16503a5c54ead9c668695832bd6fd (127.15.25.66) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } } }
I20250622 13:59:02.010579 19037 raft_consensus.cc:2953] T 596f3292519e417bbd324a7a53c500da P fda16503a5c54ead9c668695832bd6fd [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, fda16503a5c54ead9c668695832bd6fd (127.15.25.66) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } } }
I20250622 13:59:02.017884 18751 catalog_manager.cc:5582] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 reported cstate change: config changed from index 8 to 9, fda16503a5c54ead9c668695832bd6fd (127.15.25.66) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "2ca9b618349c4fa6a2bad70473011c13" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250622 13:59:02.026656 18893 consensus_queue.cc:237] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } } peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: NON_VOTER last_known_addr { host: "127.15.25.67" port: 37633 } attrs { promote: true } }
I20250622 13:59:02.032451 19037 raft_consensus.cc:1273] T 596f3292519e417bbd324a7a53c500da P fda16503a5c54ead9c668695832bd6fd [term 2 FOLLOWER]: Refusing update from remote peer 2ca9b618349c4fa6a2bad70473011c13: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250622 13:59:02.033809 19269 consensus_queue.cc:1035] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250622 13:59:02.038364 19266 raft_consensus.cc:2953] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, NON_VOTER 30361482e92a4445867fb9c448c67ea4 (127.15.25.67) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } } peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: NON_VOTER last_known_addr { host: "127.15.25.67" port: 37633 } attrs { promote: true } } }
I20250622 13:59:02.039674 19037 raft_consensus.cc:2953] T 596f3292519e417bbd324a7a53c500da P fda16503a5c54ead9c668695832bd6fd [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, NON_VOTER 30361482e92a4445867fb9c448c67ea4 (127.15.25.67) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } } peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: NON_VOTER last_known_addr { host: "127.15.25.67" port: 37633 } attrs { promote: true } } }
I20250622 13:59:02.047626 18736 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 596f3292519e417bbd324a7a53c500da with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250622 13:59:02.047957 18824 consensus_peers.cc:487] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 -> Peer 30361482e92a4445867fb9c448c67ea4 (127.15.25.67:37633): Couldn't send request to peer 30361482e92a4445867fb9c448c67ea4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 596f3292519e417bbd324a7a53c500da. This is attempt 1: this message will repeat every 5th retry.
I20250622 13:59:02.049979 18751 catalog_manager.cc:5582] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 reported cstate change: config changed from index 9 to 10, NON_VOTER 30361482e92a4445867fb9c448c67ea4 (127.15.25.67) added. New cstate: current_term: 2 leader_uuid: "2ca9b618349c4fa6a2bad70473011c13" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: NON_VOTER last_known_addr { host: "127.15.25.67" port: 37633 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250622 13:59:02.158810 19272 ts_tablet_manager.cc:927] T 26064ec815564a6b85d47ef6799f11fa P fda16503a5c54ead9c668695832bd6fd: Initiating tablet copy from peer 30361482e92a4445867fb9c448c67ea4 (127.15.25.67:37633)
I20250622 13:59:02.159883 19272 tablet_copy_client.cc:323] T 26064ec815564a6b85d47ef6799f11fa P fda16503a5c54ead9c668695832bd6fd: tablet copy: Beginning tablet copy session from remote peer at address 127.15.25.67:37633
I20250622 13:59:02.169512 19220 tablet_copy_service.cc:140] P 30361482e92a4445867fb9c448c67ea4: Received BeginTabletCopySession request for tablet 26064ec815564a6b85d47ef6799f11fa from peer fda16503a5c54ead9c668695832bd6fd ({username='slave'} at 127.15.25.66:48119)
I20250622 13:59:02.170008 19220 tablet_copy_service.cc:161] P 30361482e92a4445867fb9c448c67ea4: Beginning new tablet copy session on tablet 26064ec815564a6b85d47ef6799f11fa from peer fda16503a5c54ead9c668695832bd6fd at {username='slave'} at 127.15.25.66:48119: session id = fda16503a5c54ead9c668695832bd6fd-26064ec815564a6b85d47ef6799f11fa
I20250622 13:59:02.174680 19220 tablet_copy_source_session.cc:215] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 13:59:02.177469 19272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26064ec815564a6b85d47ef6799f11fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:59:02.186527 19272 tablet_copy_client.cc:806] T 26064ec815564a6b85d47ef6799f11fa P fda16503a5c54ead9c668695832bd6fd: tablet copy: Starting download of 0 data blocks...
I20250622 13:59:02.186947 19272 tablet_copy_client.cc:670] T 26064ec815564a6b85d47ef6799f11fa P fda16503a5c54ead9c668695832bd6fd: tablet copy: Starting download of 1 WAL segments...
I20250622 13:59:02.190085 19272 tablet_copy_client.cc:538] T 26064ec815564a6b85d47ef6799f11fa P fda16503a5c54ead9c668695832bd6fd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 13:59:02.194645 19272 tablet_bootstrap.cc:492] T 26064ec815564a6b85d47ef6799f11fa P fda16503a5c54ead9c668695832bd6fd: Bootstrap starting.
I20250622 13:59:02.284646 19272 tablet_bootstrap.cc:492] T 26064ec815564a6b85d47ef6799f11fa P fda16503a5c54ead9c668695832bd6fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:59:02.285182 19272 tablet_bootstrap.cc:492] T 26064ec815564a6b85d47ef6799f11fa P fda16503a5c54ead9c668695832bd6fd: Bootstrap complete.
I20250622 13:59:02.285560 19272 ts_tablet_manager.cc:1397] T 26064ec815564a6b85d47ef6799f11fa P fda16503a5c54ead9c668695832bd6fd: Time spent bootstrapping tablet: real 0.091s user 0.079s sys 0.008s
I20250622 13:59:02.286908 19272 raft_consensus.cc:357] T 26064ec815564a6b85d47ef6799f11fa P fda16503a5c54ead9c668695832bd6fd [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } } peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: NON_VOTER last_known_addr { host: "127.15.25.65" port: 35893 } attrs { promote: true } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: NON_VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: true } }
I20250622 13:59:02.287305 19272 raft_consensus.cc:738] T 26064ec815564a6b85d47ef6799f11fa P fda16503a5c54ead9c668695832bd6fd [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: fda16503a5c54ead9c668695832bd6fd, State: Initialized, Role: LEARNER
I20250622 13:59:02.287624 19272 consensus_queue.cc:260] T 26064ec815564a6b85d47ef6799f11fa P fda16503a5c54ead9c668695832bd6fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } } peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: NON_VOTER last_known_addr { host: "127.15.25.65" port: 35893 } attrs { promote: true } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: NON_VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: true } }
I20250622 13:59:02.288731 19272 ts_tablet_manager.cc:1428] T 26064ec815564a6b85d47ef6799f11fa P fda16503a5c54ead9c668695832bd6fd: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20250622 13:59:02.290163 19220 tablet_copy_service.cc:342] P 30361482e92a4445867fb9c448c67ea4: Request end of tablet copy session fda16503a5c54ead9c668695832bd6fd-26064ec815564a6b85d47ef6799f11fa received from {username='slave'} at 127.15.25.66:48119
I20250622 13:59:02.290562 19220 tablet_copy_service.cc:434] P 30361482e92a4445867fb9c448c67ea4: ending tablet copy session fda16503a5c54ead9c668695832bd6fd-26064ec815564a6b85d47ef6799f11fa on tablet 26064ec815564a6b85d47ef6799f11fa with peer fda16503a5c54ead9c668695832bd6fd
I20250622 13:59:02.303997 19276 ts_tablet_manager.cc:927] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13: Initiating tablet copy from peer 30361482e92a4445867fb9c448c67ea4 (127.15.25.67:37633)
I20250622 13:59:02.305953 19276 tablet_copy_client.cc:323] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13: tablet copy: Beginning tablet copy session from remote peer at address 127.15.25.67:37633
I20250622 13:59:02.307128 19220 tablet_copy_service.cc:140] P 30361482e92a4445867fb9c448c67ea4: Received BeginTabletCopySession request for tablet 26064ec815564a6b85d47ef6799f11fa from peer 2ca9b618349c4fa6a2bad70473011c13 ({username='slave'} at 127.15.25.65:51601)
I20250622 13:59:02.307440 19220 tablet_copy_service.cc:161] P 30361482e92a4445867fb9c448c67ea4: Beginning new tablet copy session on tablet 26064ec815564a6b85d47ef6799f11fa from peer 2ca9b618349c4fa6a2bad70473011c13 at {username='slave'} at 127.15.25.65:51601: session id = 2ca9b618349c4fa6a2bad70473011c13-26064ec815564a6b85d47ef6799f11fa
I20250622 13:59:02.311021 19220 tablet_copy_source_session.cc:215] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 13:59:02.313246 19276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26064ec815564a6b85d47ef6799f11fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:59:02.322144 19276 tablet_copy_client.cc:806] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13: tablet copy: Starting download of 0 data blocks...
I20250622 13:59:02.322563 19276 tablet_copy_client.cc:670] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13: tablet copy: Starting download of 1 WAL segments...
I20250622 13:59:02.325541 19276 tablet_copy_client.cc:538] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 13:59:02.329975 19276 tablet_bootstrap.cc:492] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13: Bootstrap starting.
I20250622 13:59:02.395956 19276 tablet_bootstrap.cc:492] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:59:02.396529 19276 tablet_bootstrap.cc:492] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13: Bootstrap complete.
I20250622 13:59:02.396950 19276 ts_tablet_manager.cc:1397] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13: Time spent bootstrapping tablet: real 0.067s user 0.056s sys 0.008s
I20250622 13:59:02.398449 19276 raft_consensus.cc:357] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } } peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: NON_VOTER last_known_addr { host: "127.15.25.65" port: 35893 } attrs { promote: true } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: NON_VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: true } }
I20250622 13:59:02.398882 19276 raft_consensus.cc:738] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 2ca9b618349c4fa6a2bad70473011c13, State: Initialized, Role: LEARNER
I20250622 13:59:02.399250 19276 consensus_queue.cc:260] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } } peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: NON_VOTER last_known_addr { host: "127.15.25.65" port: 35893 } attrs { promote: true } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: NON_VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: true } }
I20250622 13:59:02.400403 19276 ts_tablet_manager.cc:1428] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20250622 13:59:02.401594 19220 tablet_copy_service.cc:342] P 30361482e92a4445867fb9c448c67ea4: Request end of tablet copy session 2ca9b618349c4fa6a2bad70473011c13-26064ec815564a6b85d47ef6799f11fa received from {username='slave'} at 127.15.25.65:51601
I20250622 13:59:02.401891 19220 tablet_copy_service.cc:434] P 30361482e92a4445867fb9c448c67ea4: ending tablet copy session 2ca9b618349c4fa6a2bad70473011c13-26064ec815564a6b85d47ef6799f11fa on tablet 26064ec815564a6b85d47ef6799f11fa with peer 2ca9b618349c4fa6a2bad70473011c13
I20250622 13:59:02.582036 19279 ts_tablet_manager.cc:927] T 596f3292519e417bbd324a7a53c500da P 30361482e92a4445867fb9c448c67ea4: Initiating tablet copy from peer 2ca9b618349c4fa6a2bad70473011c13 (127.15.25.65:35893)
I20250622 13:59:02.583758 19279 tablet_copy_client.cc:323] T 596f3292519e417bbd324a7a53c500da P 30361482e92a4445867fb9c448c67ea4: tablet copy: Beginning tablet copy session from remote peer at address 127.15.25.65:35893
I20250622 13:59:02.585036 18913 tablet_copy_service.cc:140] P 2ca9b618349c4fa6a2bad70473011c13: Received BeginTabletCopySession request for tablet 596f3292519e417bbd324a7a53c500da from peer 30361482e92a4445867fb9c448c67ea4 ({username='slave'} at 127.15.25.67:57905)
I20250622 13:59:02.585387 18913 tablet_copy_service.cc:161] P 2ca9b618349c4fa6a2bad70473011c13: Beginning new tablet copy session on tablet 596f3292519e417bbd324a7a53c500da from peer 30361482e92a4445867fb9c448c67ea4 at {username='slave'} at 127.15.25.67:57905: session id = 30361482e92a4445867fb9c448c67ea4-596f3292519e417bbd324a7a53c500da
I20250622 13:59:02.588907 18913 tablet_copy_source_session.cc:215] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 13:59:02.591089 19279 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 596f3292519e417bbd324a7a53c500da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:59:02.601533 19279 tablet_copy_client.cc:806] T 596f3292519e417bbd324a7a53c500da P 30361482e92a4445867fb9c448c67ea4: tablet copy: Starting download of 0 data blocks...
I20250622 13:59:02.601950 19279 tablet_copy_client.cc:670] T 596f3292519e417bbd324a7a53c500da P 30361482e92a4445867fb9c448c67ea4: tablet copy: Starting download of 1 WAL segments...
I20250622 13:59:02.604888 19279 tablet_copy_client.cc:538] T 596f3292519e417bbd324a7a53c500da P 30361482e92a4445867fb9c448c67ea4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 13:59:02.609551 19279 tablet_bootstrap.cc:492] T 596f3292519e417bbd324a7a53c500da P 30361482e92a4445867fb9c448c67ea4: Bootstrap starting.
I20250622 13:59:02.635113 19037 raft_consensus.cc:1215] T 26064ec815564a6b85d47ef6799f11fa P fda16503a5c54ead9c668695832bd6fd [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20250622 13:59:02.678928 19279 tablet_bootstrap.cc:492] T 596f3292519e417bbd324a7a53c500da P 30361482e92a4445867fb9c448c67ea4: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:59:02.679533 19279 tablet_bootstrap.cc:492] T 596f3292519e417bbd324a7a53c500da P 30361482e92a4445867fb9c448c67ea4: Bootstrap complete.
I20250622 13:59:02.679927 19279 ts_tablet_manager.cc:1397] T 596f3292519e417bbd324a7a53c500da P 30361482e92a4445867fb9c448c67ea4: Time spent bootstrapping tablet: real 0.070s user 0.052s sys 0.019s
I20250622 13:59:02.681483 19279 raft_consensus.cc:357] T 596f3292519e417bbd324a7a53c500da P 30361482e92a4445867fb9c448c67ea4 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } } peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: NON_VOTER last_known_addr { host: "127.15.25.67" port: 37633 } attrs { promote: true } }
I20250622 13:59:02.681988 19279 raft_consensus.cc:738] T 596f3292519e417bbd324a7a53c500da P 30361482e92a4445867fb9c448c67ea4 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 30361482e92a4445867fb9c448c67ea4, State: Initialized, Role: LEARNER
I20250622 13:59:02.682440 19279 consensus_queue.cc:260] T 596f3292519e417bbd324a7a53c500da P 30361482e92a4445867fb9c448c67ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } } peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: NON_VOTER last_known_addr { host: "127.15.25.67" port: 37633 } attrs { promote: true } }
I20250622 13:59:02.683960 19279 ts_tablet_manager.cc:1428] T 596f3292519e417bbd324a7a53c500da P 30361482e92a4445867fb9c448c67ea4: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250622 13:59:02.685510 18913 tablet_copy_service.cc:342] P 2ca9b618349c4fa6a2bad70473011c13: Request end of tablet copy session 30361482e92a4445867fb9c448c67ea4-596f3292519e417bbd324a7a53c500da received from {username='slave'} at 127.15.25.67:57905
I20250622 13:59:02.685977 18913 tablet_copy_service.cc:434] P 2ca9b618349c4fa6a2bad70473011c13: ending tablet copy session 30361482e92a4445867fb9c448c67ea4-596f3292519e417bbd324a7a53c500da on tablet 596f3292519e417bbd324a7a53c500da with peer 30361482e92a4445867fb9c448c67ea4
I20250622 13:59:02.730033 15461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 2ca9b618349c4fa6a2bad70473011c13 to finish bootstrapping
I20250622 13:59:02.742861 15461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver fda16503a5c54ead9c668695832bd6fd to finish bootstrapping
I20250622 13:59:02.747659 18737 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 596f3292519e417bbd324a7a53c500da with cas_config_opid_index 8: aborting the task: latest config opid_index 10; task opid_index 8
I20250622 13:59:02.753095 15461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 30361482e92a4445867fb9c448c67ea4 to finish bootstrapping
I20250622 13:59:02.776970 18893 raft_consensus.cc:1215] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20250622 13:59:03.008772 18873 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250622 13:59:03.016502 19180 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250622 13:59:03.017697 19017 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250622 13:59:03.030635 19200 raft_consensus.cc:1215] T 596f3292519e417bbd324a7a53c500da P 30361482e92a4445867fb9c448c67ea4 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20250622 13:59:03.171190 19282 raft_consensus.cc:1062] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4: attempting to promote NON_VOTER fda16503a5c54ead9c668695832bd6fd to VOTER
I20250622 13:59:03.173214 19282 consensus_queue.cc:237] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } } peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: NON_VOTER last_known_addr { host: "127.15.25.65" port: 35893 } attrs { promote: true } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } }
I20250622 13:59:03.191167 19037 raft_consensus.cc:1273] T 26064ec815564a6b85d47ef6799f11fa P fda16503a5c54ead9c668695832bd6fd [term 2 LEARNER]: Refusing update from remote peer 30361482e92a4445867fb9c448c67ea4: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250622 13:59:03.192670 19235 consensus_queue.cc:1035] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250622 13:59:03.201570 18893 raft_consensus.cc:1273] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13 [term 2 LEARNER]: Refusing update from remote peer 30361482e92a4445867fb9c448c67ea4: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250622 13:59:03.207760 19235 consensus_queue.cc:1035] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: NON_VOTER last_known_addr { host: "127.15.25.65" port: 35893 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250622 13:59:03.209779 19235 raft_consensus.cc:2953] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, fda16503a5c54ead9c668695832bd6fd (127.15.25.66) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } } peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: NON_VOTER last_known_addr { host: "127.15.25.65" port: 35893 } attrs { promote: true } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } } }
I20250622 13:59:03.215317 19036 raft_consensus.cc:2953] T 26064ec815564a6b85d47ef6799f11fa P fda16503a5c54ead9c668695832bd6fd [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, fda16503a5c54ead9c668695832bd6fd (127.15.25.66) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } } peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: NON_VOTER last_known_addr { host: "127.15.25.65" port: 35893 } attrs { promote: true } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } } }
I20250622 13:59:03.235397 18749 catalog_manager.cc:5582] T 26064ec815564a6b85d47ef6799f11fa P fda16503a5c54ead9c668695832bd6fd reported cstate change: config changed from index 9 to 10, fda16503a5c54ead9c668695832bd6fd (127.15.25.66) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "30361482e92a4445867fb9c448c67ea4" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } } peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: NON_VOTER last_known_addr { host: "127.15.25.65" port: 35893 } attrs { promote: true } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } } }
I20250622 13:59:03.243014 19282 raft_consensus.cc:1062] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4: attempting to promote NON_VOTER 2ca9b618349c4fa6a2bad70473011c13 to VOTER
I20250622 13:59:03.244629 19282 consensus_queue.cc:237] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } } peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } }
I20250622 13:59:03.246455 18892 raft_consensus.cc:2953] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13 [term 2 LEARNER]: Committing config change with OpId 2.10: config changed from index 9 to 10, fda16503a5c54ead9c668695832bd6fd (127.15.25.66) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } } peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: NON_VOTER last_known_addr { host: "127.15.25.65" port: 35893 } attrs { promote: true } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } } }
I20250622 13:59:03.251029 19036 raft_consensus.cc:1273] T 26064ec815564a6b85d47ef6799f11fa P fda16503a5c54ead9c668695832bd6fd [term 2 FOLLOWER]: Refusing update from remote peer 30361482e92a4445867fb9c448c67ea4: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250622 13:59:03.252802 19235 consensus_queue.cc:1035] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250622 13:59:03.256052 18893 raft_consensus.cc:1273] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13 [term 2 LEARNER]: Refusing update from remote peer 30361482e92a4445867fb9c448c67ea4: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250622 13:59:03.259284 19283 consensus_queue.cc:1035] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250622 13:59:03.269810 19235 raft_consensus.cc:2953] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 2ca9b618349c4fa6a2bad70473011c13 (127.15.25.65) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } } peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } } }
I20250622 13:59:03.274777 18893 raft_consensus.cc:2953] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 2ca9b618349c4fa6a2bad70473011c13 (127.15.25.65) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } } peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } } }
I20250622 13:59:03.279304 19036 raft_consensus.cc:2953] T 26064ec815564a6b85d47ef6799f11fa P fda16503a5c54ead9c668695832bd6fd [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 2ca9b618349c4fa6a2bad70473011c13 (127.15.25.65) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } } peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } } }
I20250622 13:59:03.287382 18751 catalog_manager.cc:5582] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 reported cstate change: config changed from index 10 to 11, 2ca9b618349c4fa6a2bad70473011c13 (127.15.25.65) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "30361482e92a4445867fb9c448c67ea4" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
a43815bcdf224b538f20db87e79bfe97 | 127.15.25.126:44459 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.15.25.84:43331 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
2ca9b618349c4fa6a2bad70473011c13 | 127.15.25.65:35893 | HEALTHY | <none> | 1 | 0
30361482e92a4445867fb9c448c67ea4 | 127.15.25.67:37633 | HEALTHY | <none> | 1 | 0
fda16503a5c54ead9c668695832bd6fd | 127.15.25.66:39531 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.15.25.65 | experimental | 127.15.25.65:35893
local_ip_for_outbound_sockets | 127.15.25.66 | experimental | 127.15.25.66:39531
local_ip_for_outbound_sockets | 127.15.25.67 | experimental | 127.15.25.67:37633
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.15.25.65:35893
server_dump_info_path | /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.15.25.66:39531
server_dump_info_path | /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.15.25.67:37633
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.15.25.84:43331 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 596f3292519e417bbd324a7a53c500da of table 'TestTable' is conflicted: 1 replicas' active configs disagree with the leader master's
2ca9b618349c4fa6a2bad70473011c13 (127.15.25.65:35893): RUNNING [LEADER]
fda16503a5c54ead9c668695832bd6fd (127.15.25.66:39531): RUNNING
30361482e92a4445867fb9c448c67ea4 (127.15.25.67:37633): RUNNING [NONVOTER]
All reported replicas are:
A = 2ca9b618349c4fa6a2bad70473011c13
B = fda16503a5c54ead9c668695832bd6fd
C = 30361482e92a4445867fb9c448c67ea4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B C~ | | | Yes
A | A* B C~ | 2 | 10 | Yes
B | A* B C~ | 2 | 10 | Yes
C | A B C~ | 2 | 10 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------+----+--------------------+---------------+---------+------------+------------------+-------------
TestTable1 | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable2 | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 3
Tablets | 3
Replicas | 7
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 3 table(s) are not healthy
FAILED
I20250622 13:59:03.318004 15461 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20250622 13:59:03.579409 19313 raft_consensus.cc:1062] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13: attempting to promote NON_VOTER 30361482e92a4445867fb9c448c67ea4 to VOTER
I20250622 13:59:03.580940 19313 consensus_queue.cc:237] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } } peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } attrs { promote: false } }
I20250622 13:59:03.585505 19200 raft_consensus.cc:1273] T 596f3292519e417bbd324a7a53c500da P 30361482e92a4445867fb9c448c67ea4 [term 2 LEARNER]: Refusing update from remote peer 2ca9b618349c4fa6a2bad70473011c13: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250622 13:59:03.586803 19330 consensus_queue.cc:1035] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250622 13:59:03.588058 19036 raft_consensus.cc:1273] T 596f3292519e417bbd324a7a53c500da P fda16503a5c54ead9c668695832bd6fd [term 2 FOLLOWER]: Refusing update from remote peer 2ca9b618349c4fa6a2bad70473011c13: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250622 13:59:03.589077 19330 consensus_queue.cc:1035] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250622 13:59:03.593673 19313 raft_consensus.cc:2953] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 30361482e92a4445867fb9c448c67ea4 (127.15.25.67) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } } peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } attrs { promote: false } } }
I20250622 13:59:03.597633 19200 raft_consensus.cc:2953] T 596f3292519e417bbd324a7a53c500da P 30361482e92a4445867fb9c448c67ea4 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 30361482e92a4445867fb9c448c67ea4 (127.15.25.67) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } } peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } attrs { promote: false } } }
I20250622 13:59:03.600178 19036 raft_consensus.cc:2953] T 596f3292519e417bbd324a7a53c500da P fda16503a5c54ead9c668695832bd6fd [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 30361482e92a4445867fb9c448c67ea4 (127.15.25.67) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } } peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } attrs { promote: false } } }
I20250622 13:59:03.605039 18748 catalog_manager.cc:5582] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 reported cstate change: config changed from index 10 to 11, 30361482e92a4445867fb9c448c67ea4 (127.15.25.67) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "2ca9b618349c4fa6a2bad70473011c13" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250622 13:59:03.705942 18873 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250622 13:59:03.721832 19180 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250622 13:59:03.724561 19017 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
a43815bcdf224b538f20db87e79bfe97 | 127.15.25.126:44459 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.15.25.84:43331 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
2ca9b618349c4fa6a2bad70473011c13 | 127.15.25.65:35893 | HEALTHY | <none> | 1 | 0
30361482e92a4445867fb9c448c67ea4 | 127.15.25.67:37633 | HEALTHY | <none> | 1 | 0
fda16503a5c54ead9c668695832bd6fd | 127.15.25.66:39531 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.15.25.65 | experimental | 127.15.25.65:35893
local_ip_for_outbound_sockets | 127.15.25.66 | experimental | 127.15.25.66:39531
local_ip_for_outbound_sockets | 127.15.25.67 | experimental | 127.15.25.67:37633
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.15.25.65:35893
server_dump_info_path | /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.15.25.66:39531
server_dump_info_path | /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.15.25.67:37633
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.15.25.84:43331 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable1 | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable2 | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 3
Tablets | 3
Replicas | 7
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250622 13:59:03.908340 15461 log_verifier.cc:126] Checking tablet 16f4a5fff92e4dc2be69fcfb8e4bbb22
I20250622 13:59:03.938282 15461 log_verifier.cc:177] Verified matching terms for 8 ops in tablet 16f4a5fff92e4dc2be69fcfb8e4bbb22
I20250622 13:59:03.938575 15461 log_verifier.cc:126] Checking tablet 26064ec815564a6b85d47ef6799f11fa
I20250622 13:59:04.011827 15461 log_verifier.cc:177] Verified matching terms for 11 ops in tablet 26064ec815564a6b85d47ef6799f11fa
I20250622 13:59:04.012058 15461 log_verifier.cc:126] Checking tablet 596f3292519e417bbd324a7a53c500da
I20250622 13:59:04.085397 15461 log_verifier.cc:177] Verified matching terms for 11 ops in tablet 596f3292519e417bbd324a7a53c500da
I20250622 13:59:04.085850 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 18718
I20250622 13:59:04.113718 15461 minidump.cc:252] Setting minidump size limit to 20M
I20250622 13:59:04.115001 15461 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 13:59:04.115958 15461 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:59:04.125710 19358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:59:04.126650 15461 server_base.cc:1048] running on GCE node
W20250622 13:59:04.126390 19360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:59:04.125751 19357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:59:04.217094 15461 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 13:59:04.217348 15461 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 13:59:04.217525 15461 hybrid_clock.cc:648] HybridClock initialized: now 1750600744217506 us; error 0 us; skew 500 ppm
I20250622 13:59:04.218176 15461 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:59:04.220957 15461 webserver.cc:469] Webserver started at http://0.0.0.0:33309/ using document root <none> and password file <none>
I20250622 13:59:04.221843 15461 fs_manager.cc:362] Metadata directory not provided
I20250622 13:59:04.222054 15461 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:59:04.226795 15461 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20250622 13:59:04.230016 19365 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:59:04.230908 15461 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20250622 13:59:04.231173 15461 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
uuid: "a43815bcdf224b538f20db87e79bfe97"
format_stamp: "Formatted at 2025-06-22 13:58:43 on dist-test-slave-6tph"
I20250622 13:59:04.232911 15461 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:59:04.252696 15461 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:59:04.254032 15461 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 13:59:04.254434 15461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:59:04.262993 15461 sys_catalog.cc:263] Verifying existing consensus state
W20250622 13:59:04.266204 15461 sys_catalog.cc:243] For a single master config, on-disk Raft master: 127.15.25.126:44459 exists but no master address supplied!
I20250622 13:59:04.267928 15461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97: Bootstrap starting.
I20250622 13:59:04.306744 15461 log.cc:826] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97: Log is configured to *not* fsync() on all Append() calls
I20250622 13:59:04.364270 15461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=13 ignored=0} mutations{seen=21 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:59:04.365026 15461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97: Bootstrap complete.
I20250622 13:59:04.377313 15461 raft_consensus.cc:357] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43815bcdf224b538f20db87e79bfe97" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 44459 } }
I20250622 13:59:04.377949 15461 raft_consensus.cc:738] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: a43815bcdf224b538f20db87e79bfe97, State: Initialized, Role: FOLLOWER
I20250622 13:59:04.378700 15461 consensus_queue.cc:260] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43815bcdf224b538f20db87e79bfe97" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 44459 } }
I20250622 13:59:04.379104 15461 raft_consensus.cc:397] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [term 3 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 13:59:04.379335 15461 raft_consensus.cc:491] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [term 3 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 13:59:04.379623 15461 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [term 3 FOLLOWER]: Advancing to term 4
I20250622 13:59:04.384397 15461 raft_consensus.cc:513] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43815bcdf224b538f20db87e79bfe97" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 44459 } }
I20250622 13:59:04.385082 15461 leader_election.cc:304] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a43815bcdf224b538f20db87e79bfe97; no voters:
I20250622 13:59:04.386189 15461 leader_election.cc:290] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [CANDIDATE]: Term 4 election: Requested vote from peers
I20250622 13:59:04.386408 19372 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [term 4 FOLLOWER]: Leader election won for term 4
I20250622 13:59:04.387683 19372 raft_consensus.cc:695] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [term 4 LEADER]: Becoming Leader. State: Replica: a43815bcdf224b538f20db87e79bfe97, State: Running, Role: LEADER
I20250622 13:59:04.388305 19372 consensus_queue.cc:237] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43815bcdf224b538f20db87e79bfe97" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 44459 } }
I20250622 13:59:04.394608 19374 sys_catalog.cc:455] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a43815bcdf224b538f20db87e79bfe97. Latest consensus state: current_term: 4 leader_uuid: "a43815bcdf224b538f20db87e79bfe97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43815bcdf224b538f20db87e79bfe97" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 44459 } } }
I20250622 13:59:04.395081 19374 sys_catalog.cc:458] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [sys.catalog]: This master's current role is: LEADER
I20250622 13:59:04.395535 19373 sys_catalog.cc:455] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 leader_uuid: "a43815bcdf224b538f20db87e79bfe97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43815bcdf224b538f20db87e79bfe97" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 44459 } } }
I20250622 13:59:04.396035 19373 sys_catalog.cc:458] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [sys.catalog]: This master's current role is: LEADER
I20250622 13:59:04.419481 15461 tablet_replica.cc:331] stopping tablet replica
I20250622 13:59:04.419984 15461 raft_consensus.cc:2241] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [term 4 LEADER]: Raft consensus shutting down.
I20250622 13:59:04.420363 15461 raft_consensus.cc:2270] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250622 13:59:04.422698 15461 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250622 13:59:04.423182 15461 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250622 13:59:04.446364 15461 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
W20250622 13:59:04.636096 19250 heartbeater.cc:646] Failed to heartbeat to 127.15.25.126:44459 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.25.126:44459: connect: Connection refused (error 111)
W20250622 13:59:04.638477 19082 heartbeater.cc:646] Failed to heartbeat to 127.15.25.126:44459 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.25.126:44459: connect: Connection refused (error 111)
W20250622 13:59:04.640416 18938 heartbeater.cc:646] Failed to heartbeat to 127.15.25.126:44459 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.25.126:44459: connect: Connection refused (error 111)
I20250622 13:59:09.371321 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 18788
I20250622 13:59:09.397091 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 18942
I20250622 13:59:09.422857 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 19086
I20250622 13:59:09.451258 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.25.126:44459
--webserver_interface=127.15.25.126
--webserver_port=37457
--builtin_ntp_servers=127.15.25.84:43331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.25.126:44459 with env {}
W20250622 13:59:09.741374 19446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:59:09.742008 19446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 13:59:09.742465 19446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:59:09.771241 19446 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250622 13:59:09.771494 19446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 13:59:09.771735 19446 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250622 13:59:09.771963 19446 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250622 13:59:09.803160 19446 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:43331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.25.126:44459
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.25.126:44459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.25.126
--webserver_port=37457
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 13:59:09.804330 19446 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 13:59:09.805903 19446 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:59:09.818854 19453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:59:09.818982 19452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:59:09.821010 19446 server_base.cc:1048] running on GCE node
W20250622 13:59:09.819586 19455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:59:10.922056 19446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 13:59:10.924870 19446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 13:59:10.926393 19446 hybrid_clock.cc:648] HybridClock initialized: now 1750600750926316 us; error 59 us; skew 500 ppm
I20250622 13:59:10.927603 19446 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:59:10.935277 19446 webserver.cc:469] Webserver started at http://127.15.25.126:37457/ using document root <none> and password file <none>
I20250622 13:59:10.936214 19446 fs_manager.cc:362] Metadata directory not provided
I20250622 13:59:10.936460 19446 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:59:10.943820 19446 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20250622 13:59:10.948091 19462 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:59:10.949020 19446 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250622 13:59:10.949324 19446 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
uuid: "a43815bcdf224b538f20db87e79bfe97"
format_stamp: "Formatted at 2025-06-22 13:58:43 on dist-test-slave-6tph"
I20250622 13:59:10.951218 19446 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:59:11.008126 19446 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:59:11.009531 19446 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 13:59:11.009968 19446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:59:11.078409 19446 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.126:44459
I20250622 13:59:11.078465 19513 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.126:44459 every 8 connection(s)
I20250622 13:59:11.080886 19446 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250622 13:59:11.089902 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 19446
I20250622 13:59:11.090248 19514 sys_catalog.cc:263] Verifying existing consensus state
I20250622 13:59:11.091187 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.25.65:35893
--local_ip_for_outbound_sockets=127.15.25.65
--tserver_master_addrs=127.15.25.126:44459
--webserver_port=42715
--webserver_interface=127.15.25.65
--builtin_ntp_servers=127.15.25.84:43331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 13:59:11.094689 19514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97: Bootstrap starting.
I20250622 13:59:11.103627 19514 log.cc:826] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97: Log is configured to *not* fsync() on all Append() calls
I20250622 13:59:11.172890 19514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=15 ignored=0} mutations{seen=23 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:59:11.173645 19514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97: Bootstrap complete.
I20250622 13:59:11.190349 19514 raft_consensus.cc:357] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43815bcdf224b538f20db87e79bfe97" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 44459 } }
I20250622 13:59:11.192086 19514 raft_consensus.cc:738] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: a43815bcdf224b538f20db87e79bfe97, State: Initialized, Role: FOLLOWER
I20250622 13:59:11.192922 19514 consensus_queue.cc:260] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43815bcdf224b538f20db87e79bfe97" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 44459 } }
I20250622 13:59:11.193353 19514 raft_consensus.cc:397] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [term 5 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 13:59:11.193594 19514 raft_consensus.cc:491] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [term 5 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 13:59:11.193869 19514 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [term 5 FOLLOWER]: Advancing to term 6
I20250622 13:59:11.198477 19514 raft_consensus.cc:513] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43815bcdf224b538f20db87e79bfe97" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 44459 } }
I20250622 13:59:11.198990 19514 leader_election.cc:304] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a43815bcdf224b538f20db87e79bfe97; no voters:
I20250622 13:59:11.200873 19514 leader_election.cc:290] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [CANDIDATE]: Term 6 election: Requested vote from peers
I20250622 13:59:11.201298 19518 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [term 6 FOLLOWER]: Leader election won for term 6
I20250622 13:59:11.204032 19518 raft_consensus.cc:695] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [term 6 LEADER]: Becoming Leader. State: Replica: a43815bcdf224b538f20db87e79bfe97, State: Running, Role: LEADER
I20250622 13:59:11.204788 19518 consensus_queue.cc:237] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 6, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43815bcdf224b538f20db87e79bfe97" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 44459 } }
I20250622 13:59:11.205336 19514 sys_catalog.cc:564] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [sys.catalog]: configured and running, proceeding with master startup.
I20250622 13:59:11.216828 19519 sys_catalog.cc:455] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 leader_uuid: "a43815bcdf224b538f20db87e79bfe97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43815bcdf224b538f20db87e79bfe97" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 44459 } } }
I20250622 13:59:11.218433 19519 sys_catalog.cc:458] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [sys.catalog]: This master's current role is: LEADER
I20250622 13:59:11.217586 19520 sys_catalog.cc:455] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a43815bcdf224b538f20db87e79bfe97. Latest consensus state: current_term: 6 leader_uuid: "a43815bcdf224b538f20db87e79bfe97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43815bcdf224b538f20db87e79bfe97" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 44459 } } }
I20250622 13:59:11.219760 19520 sys_catalog.cc:458] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97 [sys.catalog]: This master's current role is: LEADER
I20250622 13:59:11.222944 19526 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 13:59:11.242326 19526 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=02798003f4eb4716956cd12c400e655f]
I20250622 13:59:11.244155 19526 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=84edf2b470974bc58807a81b5258c066]
I20250622 13:59:11.245898 19526 catalog_manager.cc:671] Loaded metadata for table TestTable [id=d0785f6b91894d16b505c88258dacf4b]
I20250622 13:59:11.254395 19526 tablet_loader.cc:96] loaded metadata for tablet 16f4a5fff92e4dc2be69fcfb8e4bbb22 (table TestTable2 [id=84edf2b470974bc58807a81b5258c066])
I20250622 13:59:11.255825 19526 tablet_loader.cc:96] loaded metadata for tablet 26064ec815564a6b85d47ef6799f11fa (table TestTable1 [id=02798003f4eb4716956cd12c400e655f])
I20250622 13:59:11.257076 19526 tablet_loader.cc:96] loaded metadata for tablet 596f3292519e417bbd324a7a53c500da (table TestTable [id=d0785f6b91894d16b505c88258dacf4b])
I20250622 13:59:11.258517 19526 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 13:59:11.263927 19526 catalog_manager.cc:1261] Loaded cluster ID: 2915a2c530594219b4d14e5bb2fb25a5
I20250622 13:59:11.264250 19526 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 13:59:11.266243 19535 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250622 13:59:11.273595 19526 catalog_manager.cc:1506] Loading token signing keys...
I20250622 13:59:11.279371 19526 catalog_manager.cc:5966] T 00000000000000000000000000000000 P a43815bcdf224b538f20db87e79bfe97: Loaded TSK: 0
I20250622 13:59:11.280879 19526 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250622 13:59:11.405481 19516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:59:11.406028 19516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 13:59:11.406576 19516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:59:11.436339 19516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 13:59:11.437201 19516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.25.65
I20250622 13:59:11.470748 19516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:43331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.25.65:35893
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.15.25.65
--webserver_port=42715
--tserver_master_addrs=127.15.25.126:44459
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.25.65
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 13:59:11.472025 19516 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 13:59:11.473630 19516 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:59:11.490366 19542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:59:11.491832 19541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:59:11.491904 19516 server_base.cc:1048] running on GCE node
W20250622 13:59:11.491623 19544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:59:12.578910 19516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 13:59:12.581696 19516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 13:59:12.583166 19516 hybrid_clock.cc:648] HybridClock initialized: now 1750600752583090 us; error 76 us; skew 500 ppm
I20250622 13:59:12.584339 19516 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:59:12.598299 19516 webserver.cc:469] Webserver started at http://127.15.25.65:42715/ using document root <none> and password file <none>
I20250622 13:59:12.599478 19516 fs_manager.cc:362] Metadata directory not provided
I20250622 13:59:12.599737 19516 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:59:12.609757 19516 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.006s sys 0.001s
I20250622 13:59:12.615491 19552 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:59:12.616565 19516 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250622 13:59:12.616858 19516 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "2ca9b618349c4fa6a2bad70473011c13"
format_stamp: "Formatted at 2025-06-22 13:58:45 on dist-test-slave-6tph"
I20250622 13:59:12.618695 19516 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:59:12.671509 19516 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:59:12.673015 19516 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 13:59:12.673434 19516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:59:12.676043 19516 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:59:12.682003 19559 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250622 13:59:12.699442 19516 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250622 13:59:12.699718 19516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.019s user 0.002s sys 0.000s
I20250622 13:59:12.700098 19516 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250622 13:59:12.706204 19559 tablet_bootstrap.cc:492] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13: Bootstrap starting.
I20250622 13:59:12.708724 19516 ts_tablet_manager.cc:610] Registered 2 tablets
I20250622 13:59:12.708925 19516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.009s user 0.006s sys 0.000s
I20250622 13:59:12.785851 19559 log.cc:826] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13: Log is configured to *not* fsync() on all Append() calls
I20250622 13:59:12.910676 19516 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.65:35893
I20250622 13:59:12.910836 19666 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.65:35893 every 8 connection(s)
I20250622 13:59:12.913352 19516 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250622 13:59:12.921126 19559 tablet_bootstrap.cc:492] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:59:12.922086 19559 tablet_bootstrap.cc:492] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13: Bootstrap complete.
I20250622 13:59:12.924227 19559 ts_tablet_manager.cc:1397] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13: Time spent bootstrapping tablet: real 0.219s user 0.147s sys 0.053s
I20250622 13:59:12.924252 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 19516
I20250622 13:59:12.925969 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.25.66:39531
--local_ip_for_outbound_sockets=127.15.25.66
--tserver_master_addrs=127.15.25.126:44459
--webserver_port=37569
--webserver_interface=127.15.25.66
--builtin_ntp_servers=127.15.25.84:43331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 13:59:12.953020 19559 raft_consensus.cc:357] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } } peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } }
I20250622 13:59:12.956403 19559 raft_consensus.cc:738] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca9b618349c4fa6a2bad70473011c13, State: Initialized, Role: FOLLOWER
I20250622 13:59:12.957549 19559 consensus_queue.cc:260] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } } peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } }
I20250622 13:59:12.972504 19559 ts_tablet_manager.cc:1428] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13: Time spent starting tablet: real 0.048s user 0.033s sys 0.008s
I20250622 13:59:12.972795 19667 heartbeater.cc:344] Connected to a master server at 127.15.25.126:44459
I20250622 13:59:12.973327 19667 heartbeater.cc:461] Registering TS with master...
I20250622 13:59:12.973341 19559 tablet_bootstrap.cc:492] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13: Bootstrap starting.
I20250622 13:59:12.974507 19667 heartbeater.cc:507] Master 127.15.25.126:44459 requested a full tablet report, sending...
I20250622 13:59:12.979326 19479 ts_manager.cc:194] Registered new tserver with Master: 2ca9b618349c4fa6a2bad70473011c13 (127.15.25.65:35893)
I20250622 13:59:12.983491 19479 catalog_manager.cc:5582] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 reported cstate change: config changed from index -1 to 11, term changed from 0 to 2, VOTER 2ca9b618349c4fa6a2bad70473011c13 (127.15.25.65) added, VOTER 30361482e92a4445867fb9c448c67ea4 (127.15.25.67) added, VOTER fda16503a5c54ead9c668695832bd6fd (127.15.25.66) added. New cstate: current_term: 2 committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } } peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } attrs { promote: false } } }
I20250622 13:59:13.034834 19479 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.25.65:37153
I20250622 13:59:13.038986 19667 heartbeater.cc:499] Master 127.15.25.126:44459 was elected leader, sending a full tablet report...
I20250622 13:59:13.077308 19559 tablet_bootstrap.cc:492] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:59:13.078130 19559 tablet_bootstrap.cc:492] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13: Bootstrap complete.
I20250622 13:59:13.079541 19559 ts_tablet_manager.cc:1397] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13: Time spent bootstrapping tablet: real 0.106s user 0.087s sys 0.016s
I20250622 13:59:13.081498 19559 raft_consensus.cc:357] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } } peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } attrs { promote: false } }
I20250622 13:59:13.082185 19559 raft_consensus.cc:738] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca9b618349c4fa6a2bad70473011c13, State: Initialized, Role: FOLLOWER
I20250622 13:59:13.082733 19559 consensus_queue.cc:260] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } } peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } attrs { promote: false } }
I20250622 13:59:13.084471 19559 ts_tablet_manager.cc:1428] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20250622 13:59:13.258205 19671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:59:13.258690 19671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 13:59:13.259138 19671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:59:13.287783 19671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 13:59:13.288528 19671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.25.66
I20250622 13:59:13.320317 19671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:43331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.25.66:39531
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.15.25.66
--webserver_port=37569
--tserver_master_addrs=127.15.25.126:44459
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.25.66
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 13:59:13.321554 19671 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 13:59:13.323088 19671 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:59:13.338469 19684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:59:14.245304 19689 raft_consensus.cc:491] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:59:14.245781 19689 raft_consensus.cc:513] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } } peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } }
I20250622 13:59:14.247912 19689 leader_election.cc:290] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30361482e92a4445867fb9c448c67ea4 (127.15.25.67:37633), fda16503a5c54ead9c668695832bd6fd (127.15.25.66:39531)
W20250622 13:59:14.251809 19553 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.25.67:37633: connect: Connection refused (error 111)
W20250622 13:59:14.255498 19553 leader_election.cc:336] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 30361482e92a4445867fb9c448c67ea4 (127.15.25.67:37633): Network error: Client connection negotiation failed: client connection to 127.15.25.67:37633: connect: Connection refused (error 111)
W20250622 13:59:14.256115 19553 leader_election.cc:336] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer fda16503a5c54ead9c668695832bd6fd (127.15.25.66:39531): Network error: Client connection negotiation failed: client connection to 127.15.25.66:39531: connect: Connection refused (error 111)
I20250622 13:59:14.256472 19553 leader_election.cc:304] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ca9b618349c4fa6a2bad70473011c13; no voters: 30361482e92a4445867fb9c448c67ea4, fda16503a5c54ead9c668695832bd6fd
I20250622 13:59:14.257187 19689 raft_consensus.cc:2747] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250622 13:59:13.338791 19686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:59:13.338472 19683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:59:14.423609 19685 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250622 13:59:14.423728 19671 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 13:59:14.427376 19671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 13:59:14.429550 19671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 13:59:14.430879 19671 hybrid_clock.cc:648] HybridClock initialized: now 1750600754430859 us; error 43 us; skew 500 ppm
I20250622 13:59:14.431612 19671 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:59:14.437922 19671 webserver.cc:469] Webserver started at http://127.15.25.66:37569/ using document root <none> and password file <none>
I20250622 13:59:14.438805 19671 fs_manager.cc:362] Metadata directory not provided
I20250622 13:59:14.439026 19671 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:59:14.446508 19671 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20250622 13:59:14.450976 19697 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:59:14.451908 19671 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20250622 13:59:14.452209 19671 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "fda16503a5c54ead9c668695832bd6fd"
format_stamp: "Formatted at 2025-06-22 13:58:47 on dist-test-slave-6tph"
I20250622 13:59:14.454255 19671 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:59:14.508787 19671 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:59:14.510288 19671 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 13:59:14.510705 19671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:59:14.513127 19671 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:59:14.518582 19704 ts_tablet_manager.cc:542] Loading tablet metadata (0/3 complete)
I20250622 13:59:14.533493 19671 ts_tablet_manager.cc:579] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20250622 13:59:14.533763 19671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.017s user 0.000s sys 0.002s
I20250622 13:59:14.534034 19671 ts_tablet_manager.cc:594] Registering tablets (0/3 complete)
I20250622 13:59:14.539047 19704 tablet_bootstrap.cc:492] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd: Bootstrap starting.
I20250622 13:59:14.544754 19671 ts_tablet_manager.cc:610] Registered 3 tablets
I20250622 13:59:14.544977 19671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.011s user 0.008s sys 0.000s
I20250622 13:59:14.586871 19704 log.cc:826] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd: Log is configured to *not* fsync() on all Append() calls
I20250622 13:59:14.673923 19704 tablet_bootstrap.cc:492] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:59:14.674664 19704 tablet_bootstrap.cc:492] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd: Bootstrap complete.
I20250622 13:59:14.675940 19704 ts_tablet_manager.cc:1397] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd: Time spent bootstrapping tablet: real 0.137s user 0.088s sys 0.045s
I20250622 13:59:14.693240 19704 raft_consensus.cc:357] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } }
I20250622 13:59:14.695235 19704 raft_consensus.cc:738] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fda16503a5c54ead9c668695832bd6fd, State: Initialized, Role: FOLLOWER
I20250622 13:59:14.696131 19704 consensus_queue.cc:260] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } }
I20250622 13:59:14.696765 19704 raft_consensus.cc:397] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 13:59:14.697072 19704 raft_consensus.cc:491] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 13:59:14.697472 19704 raft_consensus.cc:3058] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd [term 2 FOLLOWER]: Advancing to term 3
I20250622 13:59:14.704020 19704 raft_consensus.cc:513] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } }
I20250622 13:59:14.704592 19704 leader_election.cc:304] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fda16503a5c54ead9c668695832bd6fd; no voters:
I20250622 13:59:14.707252 19704 leader_election.cc:290] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd [CANDIDATE]: Term 3 election: Requested vote from peers
I20250622 13:59:14.707556 19809 raft_consensus.cc:2802] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd [term 3 FOLLOWER]: Leader election won for term 3
I20250622 13:59:14.710760 19809 raft_consensus.cc:695] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd [term 3 LEADER]: Becoming Leader. State: Replica: fda16503a5c54ead9c668695832bd6fd, State: Running, Role: LEADER
I20250622 13:59:14.711833 19809 consensus_queue.cc:237] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } }
I20250622 13:59:14.713663 19704 ts_tablet_manager.cc:1428] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd: Time spent starting tablet: real 0.037s user 0.034s sys 0.004s
I20250622 13:59:14.714268 19671 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.66:39531
I20250622 13:59:14.714452 19814 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.66:39531 every 8 connection(s)
I20250622 13:59:14.714601 19704 tablet_bootstrap.cc:492] T 26064ec815564a6b85d47ef6799f11fa P fda16503a5c54ead9c668695832bd6fd: Bootstrap starting.
I20250622 13:59:14.718641 19671 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250622 13:59:14.724889 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 19671
I20250622 13:59:14.727336 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.25.67:37633
--local_ip_for_outbound_sockets=127.15.25.67
--tserver_master_addrs=127.15.25.126:44459
--webserver_port=43387
--webserver_interface=127.15.25.67
--builtin_ntp_servers=127.15.25.84:43331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 13:59:14.756268 19816 heartbeater.cc:344] Connected to a master server at 127.15.25.126:44459
I20250622 13:59:14.756655 19816 heartbeater.cc:461] Registering TS with master...
I20250622 13:59:14.757532 19816 heartbeater.cc:507] Master 127.15.25.126:44459 requested a full tablet report, sending...
I20250622 13:59:14.761950 19479 ts_manager.cc:194] Registered new tserver with Master: fda16503a5c54ead9c668695832bd6fd (127.15.25.66:39531)
I20250622 13:59:14.764604 19479 catalog_manager.cc:5582] T 16f4a5fff92e4dc2be69fcfb8e4bbb22 P fda16503a5c54ead9c668695832bd6fd reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "fda16503a5c54ead9c668695832bd6fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } health_report { overall_health: HEALTHY } } }
I20250622 13:59:14.784926 19479 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.25.66:55947
I20250622 13:59:14.788724 19816 heartbeater.cc:499] Master 127.15.25.126:44459 was elected leader, sending a full tablet report...
I20250622 13:59:14.829494 19704 tablet_bootstrap.cc:492] T 26064ec815564a6b85d47ef6799f11fa P fda16503a5c54ead9c668695832bd6fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:59:14.830165 19704 tablet_bootstrap.cc:492] T 26064ec815564a6b85d47ef6799f11fa P fda16503a5c54ead9c668695832bd6fd: Bootstrap complete.
I20250622 13:59:14.831645 19704 ts_tablet_manager.cc:1397] T 26064ec815564a6b85d47ef6799f11fa P fda16503a5c54ead9c668695832bd6fd: Time spent bootstrapping tablet: real 0.117s user 0.086s sys 0.020s
I20250622 13:59:14.833185 19704 raft_consensus.cc:357] T 26064ec815564a6b85d47ef6799f11fa P fda16503a5c54ead9c668695832bd6fd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } } peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } }
I20250622 13:59:14.833647 19704 raft_consensus.cc:738] T 26064ec815564a6b85d47ef6799f11fa P fda16503a5c54ead9c668695832bd6fd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fda16503a5c54ead9c668695832bd6fd, State: Initialized, Role: FOLLOWER
I20250622 13:59:14.834157 19704 consensus_queue.cc:260] T 26064ec815564a6b85d47ef6799f11fa P fda16503a5c54ead9c668695832bd6fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } } peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } }
I20250622 13:59:14.835969 19704 ts_tablet_manager.cc:1428] T 26064ec815564a6b85d47ef6799f11fa P fda16503a5c54ead9c668695832bd6fd: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250622 13:59:14.836568 19704 tablet_bootstrap.cc:492] T 596f3292519e417bbd324a7a53c500da P fda16503a5c54ead9c668695832bd6fd: Bootstrap starting.
I20250622 13:59:14.898880 19828 raft_consensus.cc:491] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:59:14.899379 19828 raft_consensus.cc:513] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } } peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } attrs { promote: false } }
I20250622 13:59:14.902284 19828 leader_election.cc:290] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fda16503a5c54ead9c668695832bd6fd (127.15.25.66:39531), 30361482e92a4445867fb9c448c67ea4 (127.15.25.67:37633)
I20250622 13:59:14.941134 19704 tablet_bootstrap.cc:492] T 596f3292519e417bbd324a7a53c500da P fda16503a5c54ead9c668695832bd6fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:59:14.942058 19704 tablet_bootstrap.cc:492] T 596f3292519e417bbd324a7a53c500da P fda16503a5c54ead9c668695832bd6fd: Bootstrap complete.
I20250622 13:59:14.943636 19704 ts_tablet_manager.cc:1397] T 596f3292519e417bbd324a7a53c500da P fda16503a5c54ead9c668695832bd6fd: Time spent bootstrapping tablet: real 0.107s user 0.085s sys 0.019s
I20250622 13:59:14.945761 19704 raft_consensus.cc:357] T 596f3292519e417bbd324a7a53c500da P fda16503a5c54ead9c668695832bd6fd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } } peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } attrs { promote: false } }
I20250622 13:59:14.946439 19704 raft_consensus.cc:738] T 596f3292519e417bbd324a7a53c500da P fda16503a5c54ead9c668695832bd6fd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fda16503a5c54ead9c668695832bd6fd, State: Initialized, Role: FOLLOWER
I20250622 13:59:14.947160 19704 consensus_queue.cc:260] T 596f3292519e417bbd324a7a53c500da P fda16503a5c54ead9c668695832bd6fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } } peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } attrs { promote: false } }
W20250622 13:59:14.947999 19553 leader_election.cc:336] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 30361482e92a4445867fb9c448c67ea4 (127.15.25.67:37633): Network error: Client connection negotiation failed: client connection to 127.15.25.67:37633: connect: Connection refused (error 111)
I20250622 13:59:14.950507 19767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "596f3292519e417bbd324a7a53c500da" candidate_uuid: "2ca9b618349c4fa6a2bad70473011c13" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "fda16503a5c54ead9c668695832bd6fd" is_pre_election: true
I20250622 13:59:14.951323 19767 raft_consensus.cc:2466] T 596f3292519e417bbd324a7a53c500da P fda16503a5c54ead9c668695832bd6fd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ca9b618349c4fa6a2bad70473011c13 in term 2.
I20250622 13:59:14.953778 19704 ts_tablet_manager.cc:1428] T 596f3292519e417bbd324a7a53c500da P fda16503a5c54ead9c668695832bd6fd: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20250622 13:59:14.953673 19553 leader_election.cc:304] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ca9b618349c4fa6a2bad70473011c13, fda16503a5c54ead9c668695832bd6fd; no voters: 30361482e92a4445867fb9c448c67ea4
I20250622 13:59:14.954655 19828 raft_consensus.cc:2802] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250622 13:59:14.955021 19828 raft_consensus.cc:491] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:59:14.955358 19828 raft_consensus.cc:3058] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [term 2 FOLLOWER]: Advancing to term 3
I20250622 13:59:14.962234 19828 raft_consensus.cc:513] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } } peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } attrs { promote: false } }
I20250622 13:59:14.963605 19828 leader_election.cc:290] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [CANDIDATE]: Term 3 election: Requested vote from peers fda16503a5c54ead9c668695832bd6fd (127.15.25.66:39531), 30361482e92a4445867fb9c448c67ea4 (127.15.25.67:37633)
I20250622 13:59:14.964435 19767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "596f3292519e417bbd324a7a53c500da" candidate_uuid: "2ca9b618349c4fa6a2bad70473011c13" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "fda16503a5c54ead9c668695832bd6fd"
I20250622 13:59:14.965021 19767 raft_consensus.cc:3058] T 596f3292519e417bbd324a7a53c500da P fda16503a5c54ead9c668695832bd6fd [term 2 FOLLOWER]: Advancing to term 3
W20250622 13:59:14.967636 19553 leader_election.cc:336] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 30361482e92a4445867fb9c448c67ea4 (127.15.25.67:37633): Network error: Client connection negotiation failed: client connection to 127.15.25.67:37633: connect: Connection refused (error 111)
I20250622 13:59:14.970381 19767 raft_consensus.cc:2466] T 596f3292519e417bbd324a7a53c500da P fda16503a5c54ead9c668695832bd6fd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ca9b618349c4fa6a2bad70473011c13 in term 3.
I20250622 13:59:14.971240 19553 leader_election.cc:304] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ca9b618349c4fa6a2bad70473011c13, fda16503a5c54ead9c668695832bd6fd; no voters: 30361482e92a4445867fb9c448c67ea4
I20250622 13:59:14.971969 19828 raft_consensus.cc:2802] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [term 3 FOLLOWER]: Leader election won for term 3
I20250622 13:59:14.973452 19828 raft_consensus.cc:695] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [term 3 LEADER]: Becoming Leader. State: Replica: 2ca9b618349c4fa6a2bad70473011c13, State: Running, Role: LEADER
I20250622 13:59:14.974203 19828 consensus_queue.cc:237] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } } peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } attrs { promote: false } }
I20250622 13:59:14.985637 19479 catalog_manager.cc:5582] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 reported cstate change: term changed from 2 to 3, leader changed from <none> to 2ca9b618349c4fa6a2bad70473011c13 (127.15.25.65). New cstate: current_term: 3 leader_uuid: "2ca9b618349c4fa6a2bad70473011c13" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250622 13:59:15.080816 19821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:59:15.081317 19821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 13:59:15.081859 19821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:59:15.112891 19821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 13:59:15.113811 19821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.25.67
I20250622 13:59:15.147270 19821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:43331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.25.67:37633
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.15.25.67
--webserver_port=43387
--tserver_master_addrs=127.15.25.126:44459
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.25.67
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 13:59:15.148532 19821 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 13:59:15.150133 19821 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:59:15.166355 19840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:59:15.583660 19553 consensus_peers.cc:487] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 -> Peer 30361482e92a4445867fb9c448c67ea4 (127.15.25.67:37633): Couldn't send request to peer 30361482e92a4445867fb9c448c67ea4. Status: Network error: Client connection negotiation failed: client connection to 127.15.25.67:37633: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250622 13:59:15.584444 19767 raft_consensus.cc:1273] T 596f3292519e417bbd324a7a53c500da P fda16503a5c54ead9c668695832bd6fd [term 3 FOLLOWER]: Refusing update from remote peer 2ca9b618349c4fa6a2bad70473011c13: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20250622 13:59:15.586338 19846 consensus_queue.cc:1035] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20250622 13:59:15.630198 19622 consensus_queue.cc:237] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } }
I20250622 13:59:15.635401 19767 raft_consensus.cc:1273] T 596f3292519e417bbd324a7a53c500da P fda16503a5c54ead9c668695832bd6fd [term 3 FOLLOWER]: Refusing update from remote peer 2ca9b618349c4fa6a2bad70473011c13: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20250622 13:59:15.636574 19854 consensus_queue.cc:1035] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250622 13:59:15.641058 19846 raft_consensus.cc:2953] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [term 3 LEADER]: Committing config change with OpId 3.13: config changed from index 11 to 13, VOTER 30361482e92a4445867fb9c448c67ea4 (127.15.25.67) evicted. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } } }
I20250622 13:59:15.642158 19767 raft_consensus.cc:2953] T 596f3292519e417bbd324a7a53c500da P fda16503a5c54ead9c668695832bd6fd [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 11 to 13, VOTER 30361482e92a4445867fb9c448c67ea4 (127.15.25.67) evicted. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } } }
I20250622 13:59:15.650750 19479 catalog_manager.cc:5582] T 596f3292519e417bbd324a7a53c500da P fda16503a5c54ead9c668695832bd6fd reported cstate change: config changed from index 11 to 13, VOTER 30361482e92a4445867fb9c448c67ea4 (127.15.25.67) evicted. New cstate: current_term: 3 leader_uuid: "2ca9b618349c4fa6a2bad70473011c13" committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } } }
I20250622 13:59:15.653854 19464 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 596f3292519e417bbd324a7a53c500da with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20250622 13:59:15.661979 19479 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet 596f3292519e417bbd324a7a53c500da on TS 30361482e92a4445867fb9c448c67ea4: Not found: failed to reset TS proxy: Could not find TS for UUID 30361482e92a4445867fb9c448c67ea4
I20250622 13:59:15.665027 19622 consensus_queue.cc:237] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 11, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } }
I20250622 13:59:15.666772 19854 raft_consensus.cc:2953] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 13 to 14, VOTER fda16503a5c54ead9c668695832bd6fd (127.15.25.66) evicted. New config: { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } } }
I20250622 13:59:15.675532 19464 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 596f3292519e417bbd324a7a53c500da with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 13:59:15.677904 19478 catalog_manager.cc:5582] T 596f3292519e417bbd324a7a53c500da P 2ca9b618349c4fa6a2bad70473011c13 reported cstate change: config changed from index 13 to 14, VOTER fda16503a5c54ead9c668695832bd6fd (127.15.25.66) evicted. New cstate: current_term: 3 leader_uuid: "2ca9b618349c4fa6a2bad70473011c13" committed_config { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } health_report { overall_health: HEALTHY } } }
I20250622 13:59:15.694160 19747 tablet_service.cc:1515] Processing DeleteTablet for tablet 596f3292519e417bbd324a7a53c500da with delete_type TABLET_DATA_TOMBSTONED (TS fda16503a5c54ead9c668695832bd6fd not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:35528
I20250622 13:59:15.699034 19861 tablet_replica.cc:331] stopping tablet replica
I20250622 13:59:15.699698 19861 raft_consensus.cc:2241] T 596f3292519e417bbd324a7a53c500da P fda16503a5c54ead9c668695832bd6fd [term 3 FOLLOWER]: Raft consensus shutting down.
I20250622 13:59:15.700166 19861 raft_consensus.cc:2270] T 596f3292519e417bbd324a7a53c500da P fda16503a5c54ead9c668695832bd6fd [term 3 FOLLOWER]: Raft consensus is shut down!
I20250622 13:59:15.702647 19861 ts_tablet_manager.cc:1905] T 596f3292519e417bbd324a7a53c500da P fda16503a5c54ead9c668695832bd6fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20250622 13:59:15.715001 19464 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 596f3292519e417bbd324a7a53c500da on TS 30361482e92a4445867fb9c448c67ea4 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 30361482e92a4445867fb9c448c67ea4
I20250622 13:59:15.716750 19861 ts_tablet_manager.cc:1918] T 596f3292519e417bbd324a7a53c500da P fda16503a5c54ead9c668695832bd6fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.13
I20250622 13:59:15.717142 19861 log.cc:1199] T 596f3292519e417bbd324a7a53c500da P fda16503a5c54ead9c668695832bd6fd: Deleting WAL directory at /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal/wals/596f3292519e417bbd324a7a53c500da
I20250622 13:59:15.718751 19463 catalog_manager.cc:4928] TS fda16503a5c54ead9c668695832bd6fd (127.15.25.66:39531): tablet 596f3292519e417bbd324a7a53c500da (table TestTable [id=d0785f6b91894d16b505c88258dacf4b]) successfully deleted
I20250622 13:59:16.298928 19862 raft_consensus.cc:491] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:59:16.299290 19862 raft_consensus.cc:513] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } } peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } }
I20250622 13:59:16.300671 19862 leader_election.cc:290] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30361482e92a4445867fb9c448c67ea4 (127.15.25.67:37633), fda16503a5c54ead9c668695832bd6fd (127.15.25.66:39531)
I20250622 13:59:16.303543 19767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26064ec815564a6b85d47ef6799f11fa" candidate_uuid: "2ca9b618349c4fa6a2bad70473011c13" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "fda16503a5c54ead9c668695832bd6fd" is_pre_election: true
I20250622 13:59:16.304067 19767 raft_consensus.cc:2466] T 26064ec815564a6b85d47ef6799f11fa P fda16503a5c54ead9c668695832bd6fd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ca9b618349c4fa6a2bad70473011c13 in term 2.
I20250622 13:59:16.305759 19553 leader_election.cc:304] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca9b618349c4fa6a2bad70473011c13, fda16503a5c54ead9c668695832bd6fd; no voters:
I20250622 13:59:16.306526 19862 raft_consensus.cc:2802] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250622 13:59:16.306797 19862 raft_consensus.cc:491] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:59:16.307024 19862 raft_consensus.cc:3058] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13 [term 2 FOLLOWER]: Advancing to term 3
W20250622 13:59:16.307236 19553 leader_election.cc:336] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 30361482e92a4445867fb9c448c67ea4 (127.15.25.67:37633): Network error: Client connection negotiation failed: client connection to 127.15.25.67:37633: connect: Connection refused (error 111)
I20250622 13:59:16.310938 19862 raft_consensus.cc:513] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } } peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } }
I20250622 13:59:16.312187 19862 leader_election.cc:290] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13 [CANDIDATE]: Term 3 election: Requested vote from peers 30361482e92a4445867fb9c448c67ea4 (127.15.25.67:37633), fda16503a5c54ead9c668695832bd6fd (127.15.25.66:39531)
I20250622 13:59:16.313822 19767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26064ec815564a6b85d47ef6799f11fa" candidate_uuid: "2ca9b618349c4fa6a2bad70473011c13" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "fda16503a5c54ead9c668695832bd6fd"
I20250622 13:59:16.314231 19767 raft_consensus.cc:3058] T 26064ec815564a6b85d47ef6799f11fa P fda16503a5c54ead9c668695832bd6fd [term 2 FOLLOWER]: Advancing to term 3
W20250622 13:59:16.315729 19553 leader_election.cc:336] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 30361482e92a4445867fb9c448c67ea4 (127.15.25.67:37633): Network error: Client connection negotiation failed: client connection to 127.15.25.67:37633: connect: Connection refused (error 111)
W20250622 13:59:15.167840 19843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:59:15.168602 19821 server_base.cc:1048] running on GCE node
W20250622 13:59:15.166404 19841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:59:16.319275 19767 raft_consensus.cc:2466] T 26064ec815564a6b85d47ef6799f11fa P fda16503a5c54ead9c668695832bd6fd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ca9b618349c4fa6a2bad70473011c13 in term 3.
I20250622 13:59:16.320214 19553 leader_election.cc:304] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ca9b618349c4fa6a2bad70473011c13, fda16503a5c54ead9c668695832bd6fd; no voters: 30361482e92a4445867fb9c448c67ea4
I20250622 13:59:16.320782 19862 raft_consensus.cc:2802] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13 [term 3 FOLLOWER]: Leader election won for term 3
I20250622 13:59:16.321632 19862 raft_consensus.cc:695] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13 [term 3 LEADER]: Becoming Leader. State: Replica: 2ca9b618349c4fa6a2bad70473011c13, State: Running, Role: LEADER
I20250622 13:59:16.322477 19821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 13:59:16.322566 19862 consensus_queue.cc:237] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } } peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } }
I20250622 13:59:16.325294 19821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 13:59:16.326859 19821 hybrid_clock.cc:648] HybridClock initialized: now 1750600756326804 us; error 31 us; skew 500 ppm
I20250622 13:59:16.327761 19821 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:59:16.329439 19478 catalog_manager.cc:5582] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13 reported cstate change: term changed from 2 to 3, leader changed from 30361482e92a4445867fb9c448c67ea4 (127.15.25.67) to 2ca9b618349c4fa6a2bad70473011c13 (127.15.25.65). New cstate: current_term: 3 leader_uuid: "2ca9b618349c4fa6a2bad70473011c13" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250622 13:59:16.334568 19821 webserver.cc:469] Webserver started at http://127.15.25.67:43387/ using document root <none> and password file <none>
I20250622 13:59:16.335443 19821 fs_manager.cc:362] Metadata directory not provided
I20250622 13:59:16.335675 19821 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:59:16.343740 19821 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20250622 13:59:16.347962 19874 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:59:16.348857 19821 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20250622 13:59:16.349134 19821 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "30361482e92a4445867fb9c448c67ea4"
format_stamp: "Formatted at 2025-06-22 13:58:48 on dist-test-slave-6tph"
I20250622 13:59:16.350885 19821 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:59:16.395174 19821 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:59:16.396562 19821 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 13:59:16.397023 19821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:59:16.399390 19821 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:59:16.405515 19881 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250622 13:59:16.416587 19821 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250622 13:59:16.416805 19821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.002s sys 0.001s
I20250622 13:59:16.417037 19821 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250622 13:59:16.421876 19881 tablet_bootstrap.cc:492] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4: Bootstrap starting.
I20250622 13:59:16.424129 19821 ts_tablet_manager.cc:610] Registered 2 tablets
I20250622 13:59:16.424317 19821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.007s user 0.002s sys 0.003s
I20250622 13:59:16.473367 19881 log.cc:826] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4: Log is configured to *not* fsync() on all Append() calls
I20250622 13:59:16.561244 19881 tablet_bootstrap.cc:492] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:59:16.562202 19881 tablet_bootstrap.cc:492] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4: Bootstrap complete.
I20250622 13:59:16.563381 19881 ts_tablet_manager.cc:1397] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4: Time spent bootstrapping tablet: real 0.142s user 0.097s sys 0.043s
I20250622 13:59:16.579701 19881 raft_consensus.cc:357] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } } peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } }
I20250622 13:59:16.581705 19881 raft_consensus.cc:738] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30361482e92a4445867fb9c448c67ea4, State: Initialized, Role: FOLLOWER
I20250622 13:59:16.582759 19881 consensus_queue.cc:260] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } } peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } attrs { promote: false } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } }
I20250622 13:59:16.587066 19881 ts_tablet_manager.cc:1428] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4: Time spent starting tablet: real 0.023s user 0.019s sys 0.005s
I20250622 13:59:16.587728 19881 tablet_bootstrap.cc:492] T 596f3292519e417bbd324a7a53c500da P 30361482e92a4445867fb9c448c67ea4: Bootstrap starting.
I20250622 13:59:16.594936 19821 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.67:37633
I20250622 13:59:16.595263 19989 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.67:37633 every 8 connection(s)
I20250622 13:59:16.597465 19821 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250622 13:59:16.599093 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 19821
I20250622 13:59:16.641302 19990 heartbeater.cc:344] Connected to a master server at 127.15.25.126:44459
I20250622 13:59:16.641693 19990 heartbeater.cc:461] Registering TS with master...
I20250622 13:59:16.642688 19990 heartbeater.cc:507] Master 127.15.25.126:44459 requested a full tablet report, sending...
I20250622 13:59:16.646643 19478 ts_manager.cc:194] Registered new tserver with Master: 30361482e92a4445867fb9c448c67ea4 (127.15.25.67:37633)
I20250622 13:59:16.651038 19478 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.25.67:41005
I20250622 13:59:16.651398 15461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250622 13:59:16.657253 15461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
W20250622 13:59:16.660648 15461 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 596f3292519e417bbd324a7a53c500da: tablet_id: "596f3292519e417bbd324a7a53c500da" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
I20250622 13:59:16.701812 19881 tablet_bootstrap.cc:492] T 596f3292519e417bbd324a7a53c500da P 30361482e92a4445867fb9c448c67ea4: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 13:59:16.702473 19881 tablet_bootstrap.cc:492] T 596f3292519e417bbd324a7a53c500da P 30361482e92a4445867fb9c448c67ea4: Bootstrap complete.
I20250622 13:59:16.703627 19881 ts_tablet_manager.cc:1397] T 596f3292519e417bbd324a7a53c500da P 30361482e92a4445867fb9c448c67ea4: Time spent bootstrapping tablet: real 0.116s user 0.091s sys 0.014s
I20250622 13:59:16.705147 19881 raft_consensus.cc:357] T 596f3292519e417bbd324a7a53c500da P 30361482e92a4445867fb9c448c67ea4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } } peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } attrs { promote: false } }
I20250622 13:59:16.705613 19881 raft_consensus.cc:738] T 596f3292519e417bbd324a7a53c500da P 30361482e92a4445867fb9c448c67ea4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30361482e92a4445867fb9c448c67ea4, State: Initialized, Role: FOLLOWER
I20250622 13:59:16.706197 19881 consensus_queue.cc:260] T 596f3292519e417bbd324a7a53c500da P 30361482e92a4445867fb9c448c67ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2ca9b618349c4fa6a2bad70473011c13" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 35893 } } peers { permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false } } peers { permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 } attrs { promote: false } }
I20250622 13:59:16.707197 19990 heartbeater.cc:499] Master 127.15.25.126:44459 was elected leader, sending a full tablet report...
I20250622 13:59:16.707650 19881 ts_tablet_manager.cc:1428] T 596f3292519e417bbd324a7a53c500da P 30361482e92a4445867fb9c448c67ea4: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250622 13:59:16.726016 19767 raft_consensus.cc:1273] T 26064ec815564a6b85d47ef6799f11fa P fda16503a5c54ead9c668695832bd6fd [term 3 FOLLOWER]: Refusing update from remote peer 2ca9b618349c4fa6a2bad70473011c13: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20250622 13:59:16.727278 19862 consensus_queue.cc:1035] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fda16503a5c54ead9c668695832bd6fd" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 39531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250622 13:59:16.756475 19944 raft_consensus.cc:3058] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [term 2 FOLLOWER]: Advancing to term 3
I20250622 13:59:16.762610 19944 raft_consensus.cc:1273] T 26064ec815564a6b85d47ef6799f11fa P 30361482e92a4445867fb9c448c67ea4 [term 3 FOLLOWER]: Refusing update from remote peer 2ca9b618349c4fa6a2bad70473011c13: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20250622 13:59:16.763921 20001 consensus_queue.cc:1035] T 26064ec815564a6b85d47ef6799f11fa P 2ca9b618349c4fa6a2bad70473011c13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30361482e92a4445867fb9c448c67ea4" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 37633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250622 13:59:16.890206 19924 tablet_service.cc:1515] Processing DeleteTablet for tablet 596f3292519e417bbd324a7a53c500da with delete_type TABLET_DATA_TOMBSTONED (TS 30361482e92a4445867fb9c448c67ea4 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:44394
I20250622 13:59:16.894516 20006 tablet_replica.cc:331] stopping tablet replica
I20250622 13:59:16.895135 20006 raft_consensus.cc:2241] T 596f3292519e417bbd324a7a53c500da P 30361482e92a4445867fb9c448c67ea4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 13:59:16.895545 20006 raft_consensus.cc:2270] T 596f3292519e417bbd324a7a53c500da P 30361482e92a4445867fb9c448c67ea4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 13:59:16.897704 20006 ts_tablet_manager.cc:1905] T 596f3292519e417bbd324a7a53c500da P 30361482e92a4445867fb9c448c67ea4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250622 13:59:16.907069 20006 ts_tablet_manager.cc:1918] T 596f3292519e417bbd324a7a53c500da P 30361482e92a4445867fb9c448c67ea4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.11
I20250622 13:59:16.907377 20006 log.cc:1199] T 596f3292519e417bbd324a7a53c500da P 30361482e92a4445867fb9c448c67ea4: Deleting WAL directory at /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal/wals/596f3292519e417bbd324a7a53c500da
I20250622 13:59:16.908577 19463 catalog_manager.cc:4928] TS 30361482e92a4445867fb9c448c67ea4 (127.15.25.67:37633): tablet 596f3292519e417bbd324a7a53c500da (table TestTable [id=d0785f6b91894d16b505c88258dacf4b]) successfully deleted
W20250622 13:59:17.664120 15461 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 596f3292519e417bbd324a7a53c500da: tablet_id: "596f3292519e417bbd324a7a53c500da" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 13:59:18.667395 15461 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 596f3292519e417bbd324a7a53c500da: tablet_id: "596f3292519e417bbd324a7a53c500da" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 13:59:19.671027 15461 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 596f3292519e417bbd324a7a53c500da: tablet_id: "596f3292519e417bbd324a7a53c500da" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 13:59:20.674559 15461 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 596f3292519e417bbd324a7a53c500da: tablet_id: "596f3292519e417bbd324a7a53c500da" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 13:59:21.677764 15461 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 596f3292519e417bbd324a7a53c500da: tablet_id: "596f3292519e417bbd324a7a53c500da" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 13:59:22.681908 15461 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 596f3292519e417bbd324a7a53c500da: tablet_id: "596f3292519e417bbd324a7a53c500da" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 13:59:23.685405 15461 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 596f3292519e417bbd324a7a53c500da: tablet_id: "596f3292519e417bbd324a7a53c500da" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 13:59:24.688575 15461 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 596f3292519e417bbd324a7a53c500da: tablet_id: "596f3292519e417bbd324a7a53c500da" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 13:59:25.691876 15461 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 596f3292519e417bbd324a7a53c500da: tablet_id: "596f3292519e417bbd324a7a53c500da" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 13:59:26.695421 15461 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 596f3292519e417bbd324a7a53c500da: tablet_id: "596f3292519e417bbd324a7a53c500da" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 13:59:27.698823 15461 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 596f3292519e417bbd324a7a53c500da: tablet_id: "596f3292519e417bbd324a7a53c500da" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 13:59:28.039907 19986 debug-util.cc:398] Leaking SignalData structure 0x7b08000cc320 after lost signal to thread 19835
W20250622 13:59:28.040664 19986 debug-util.cc:398] Leaking SignalData structure 0x7b08000bd760 after lost signal to thread 19989
W20250622 13:59:28.702498 15461 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 596f3292519e417bbd324a7a53c500da: tablet_id: "596f3292519e417bbd324a7a53c500da" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 13:59:29.706056 15461 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 596f3292519e417bbd324a7a53c500da: tablet_id: "596f3292519e417bbd324a7a53c500da" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 13:59:30.709671 15461 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 596f3292519e417bbd324a7a53c500da: tablet_id: "596f3292519e417bbd324a7a53c500da" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 13:59:31.712934 15461 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 596f3292519e417bbd324a7a53c500da: tablet_id: "596f3292519e417bbd324a7a53c500da" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 13:59:32.717195 15461 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 596f3292519e417bbd324a7a53c500da: tablet_id: "596f3292519e417bbd324a7a53c500da" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 13:59:33.720659 15461 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 596f3292519e417bbd324a7a53c500da: tablet_id: "596f3292519e417bbd324a7a53c500da" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 13:59:34.723847 15461 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 596f3292519e417bbd324a7a53c500da: tablet_id: "596f3292519e417bbd324a7a53c500da" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 13:59:35.727269 15461 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 596f3292519e417bbd324a7a53c500da: tablet_id: "596f3292519e417bbd324a7a53c500da" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/kudu-admin-test.cc:3914: Failure
Failed
Bad status: Not found: not all replicas of tablets comprising table TestTable are registered yet
I20250622 13:59:36.732190 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 19516
I20250622 13:59:36.757139 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 19671
I20250622 13:59:36.784410 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 19821
I20250622 13:59:36.811272 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 19446
2025-06-22T13:59:36Z chronyd exiting
I20250622 13:59:36.859930 15461 test_util.cc:183] -----------------------------------------------
I20250622 13:59:36.860146 15461 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750600661728289-15461-0
[ FAILED ] AdminCliTest.TestRebuildTables (54733 ms)
[----------] 5 tests from AdminCliTest (115069 ms total)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest
[ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4
I20250622 13:59:36.863662 15461 test_util.cc:276] Using random seed: -2123182184
I20250622 13:59:36.867681 15461 ts_itest-base.cc:115] Starting cluster with:
I20250622 13:59:36.867830 15461 ts_itest-base.cc:116] --------------
I20250622 13:59:36.867970 15461 ts_itest-base.cc:117] 5 tablet servers
I20250622 13:59:36.868106 15461 ts_itest-base.cc:118] 3 replicas per TS
I20250622 13:59:36.868238 15461 ts_itest-base.cc:119] --------------
2025-06-22T13:59:36Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-22T13:59:36Z Disabled control of system clock
I20250622 13:59:36.909379 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.25.126:35699
--webserver_interface=127.15.25.126
--webserver_port=0
--builtin_ntp_servers=127.15.25.84:35651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.25.126:35699
--raft_prepare_replacement_before_eviction=true with env {}
W20250622 13:59:37.198851 20028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:59:37.199368 20028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 13:59:37.199729 20028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:59:37.229512 20028 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250622 13:59:37.229908 20028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250622 13:59:37.230126 20028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 13:59:37.230319 20028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250622 13:59:37.230501 20028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250622 13:59:37.264060 20028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:35651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.25.126:35699
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.25.126:35699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.25.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 13:59:37.265370 20028 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 13:59:37.267246 20028 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:59:37.280483 20035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:59:37.280478 20034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:59:37.287878 20028 server_base.cc:1048] running on GCE node
W20250622 13:59:37.288261 20037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:59:38.415705 20028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 13:59:38.418255 20028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 13:59:38.419592 20028 hybrid_clock.cc:648] HybridClock initialized: now 1750600778419572 us; error 43 us; skew 500 ppm
I20250622 13:59:38.420399 20028 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:59:38.426797 20028 webserver.cc:469] Webserver started at http://127.15.25.126:46813/ using document root <none> and password file <none>
I20250622 13:59:38.427681 20028 fs_manager.cc:362] Metadata directory not provided
I20250622 13:59:38.427896 20028 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:59:38.428341 20028 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:59:38.432596 20028 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "3f0772393de14278ad163ab8ffe5b616"
format_stamp: "Formatted at 2025-06-22 13:59:38 on dist-test-slave-6tph"
I20250622 13:59:38.433668 20028 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "3f0772393de14278ad163ab8ffe5b616"
format_stamp: "Formatted at 2025-06-22 13:59:38 on dist-test-slave-6tph"
I20250622 13:59:38.440511 20028 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20250622 13:59:38.445991 20044 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:59:38.446920 20028 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250622 13:59:38.447221 20028 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
uuid: "3f0772393de14278ad163ab8ffe5b616"
format_stamp: "Formatted at 2025-06-22 13:59:38 on dist-test-slave-6tph"
I20250622 13:59:38.447520 20028 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:59:38.494462 20028 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:59:38.495962 20028 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 13:59:38.496384 20028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:59:38.564467 20028 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.126:35699
I20250622 13:59:38.564540 20095 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.126:35699 every 8 connection(s)
I20250622 13:59:38.567142 20028 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250622 13:59:38.572007 20096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:59:38.577324 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 20028
I20250622 13:59:38.577760 15461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250622 13:59:38.589741 20096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f0772393de14278ad163ab8ffe5b616: Bootstrap starting.
I20250622 13:59:38.596295 20096 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f0772393de14278ad163ab8ffe5b616: Neither blocks nor log segments found. Creating new log.
I20250622 13:59:38.598153 20096 log.cc:826] T 00000000000000000000000000000000 P 3f0772393de14278ad163ab8ffe5b616: Log is configured to *not* fsync() on all Append() calls
I20250622 13:59:38.603092 20096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f0772393de14278ad163ab8ffe5b616: No bootstrap required, opened a new log
I20250622 13:59:38.619805 20096 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3f0772393de14278ad163ab8ffe5b616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f0772393de14278ad163ab8ffe5b616" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 35699 } }
I20250622 13:59:38.620584 20096 raft_consensus.cc:383] T 00000000000000000000000000000000 P 3f0772393de14278ad163ab8ffe5b616 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:59:38.620817 20096 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3f0772393de14278ad163ab8ffe5b616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f0772393de14278ad163ab8ffe5b616, State: Initialized, Role: FOLLOWER
I20250622 13:59:38.621536 20096 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f0772393de14278ad163ab8ffe5b616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f0772393de14278ad163ab8ffe5b616" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 35699 } }
I20250622 13:59:38.622079 20096 raft_consensus.cc:397] T 00000000000000000000000000000000 P 3f0772393de14278ad163ab8ffe5b616 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 13:59:38.622301 20096 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3f0772393de14278ad163ab8ffe5b616 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 13:59:38.622531 20096 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 3f0772393de14278ad163ab8ffe5b616 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:59:38.626459 20096 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3f0772393de14278ad163ab8ffe5b616 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f0772393de14278ad163ab8ffe5b616" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 35699 } }
I20250622 13:59:38.627195 20096 leader_election.cc:304] T 00000000000000000000000000000000 P 3f0772393de14278ad163ab8ffe5b616 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f0772393de14278ad163ab8ffe5b616; no voters:
I20250622 13:59:38.628799 20096 leader_election.cc:290] T 00000000000000000000000000000000 P 3f0772393de14278ad163ab8ffe5b616 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 13:59:38.629511 20101 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 3f0772393de14278ad163ab8ffe5b616 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:59:38.631565 20101 raft_consensus.cc:695] T 00000000000000000000000000000000 P 3f0772393de14278ad163ab8ffe5b616 [term 1 LEADER]: Becoming Leader. State: Replica: 3f0772393de14278ad163ab8ffe5b616, State: Running, Role: LEADER
I20250622 13:59:38.632308 20101 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f0772393de14278ad163ab8ffe5b616 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f0772393de14278ad163ab8ffe5b616" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 35699 } }
I20250622 13:59:38.633253 20096 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f0772393de14278ad163ab8ffe5b616 [sys.catalog]: configured and running, proceeding with master startup.
I20250622 13:59:38.642190 20103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f0772393de14278ad163ab8ffe5b616 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f0772393de14278ad163ab8ffe5b616. Latest consensus state: current_term: 1 leader_uuid: "3f0772393de14278ad163ab8ffe5b616" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f0772393de14278ad163ab8ffe5b616" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 35699 } } }
I20250622 13:59:38.642920 20103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f0772393de14278ad163ab8ffe5b616 [sys.catalog]: This master's current role is: LEADER
I20250622 13:59:38.644381 20102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f0772393de14278ad163ab8ffe5b616 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f0772393de14278ad163ab8ffe5b616" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f0772393de14278ad163ab8ffe5b616" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 35699 } } }
I20250622 13:59:38.645114 20102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f0772393de14278ad163ab8ffe5b616 [sys.catalog]: This master's current role is: LEADER
I20250622 13:59:38.648067 20110 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 13:59:38.658794 20110 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 13:59:38.674465 20110 catalog_manager.cc:1349] Generated new cluster ID: 497dfe5f1b4146c4b636bac6f1dab7bf
I20250622 13:59:38.674736 20110 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 13:59:38.712706 20110 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 13:59:38.714231 20110 catalog_manager.cc:1506] Loading token signing keys...
I20250622 13:59:38.725780 20110 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 3f0772393de14278ad163ab8ffe5b616: Generated new TSK 0
I20250622 13:59:38.726692 20110 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 13:59:38.752537 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.25.65:0
--local_ip_for_outbound_sockets=127.15.25.65
--webserver_interface=127.15.25.65
--webserver_port=0
--tserver_master_addrs=127.15.25.126:35699
--builtin_ntp_servers=127.15.25.84:35651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250622 13:59:39.069494 20120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:59:39.070008 20120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 13:59:39.070490 20120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:59:39.101337 20120 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250622 13:59:39.101861 20120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 13:59:39.102607 20120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.25.65
I20250622 13:59:39.135972 20120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:35651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.25.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.15.25.65
--webserver_port=0
--tserver_master_addrs=127.15.25.126:35699
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.25.65
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 13:59:39.137306 20120 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 13:59:39.138914 20120 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:59:39.156352 20127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:59:39.866220 20092 debug-util.cc:398] Leaking SignalData structure 0x7b080006f060 after lost signal to thread 20029
W20250622 13:59:39.867185 20092 debug-util.cc:398] Leaking SignalData structure 0x7b080006f7c0 after lost signal to thread 20095
W20250622 13:59:39.158407 20129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:59:40.538303 20128 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1382 milliseconds
I20250622 13:59:40.538677 20120 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250622 13:59:39.156399 20126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:59:40.544651 20120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 13:59:40.548609 20120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 13:59:40.550135 20120 hybrid_clock.cc:648] HybridClock initialized: now 1750600780550062 us; error 81 us; skew 500 ppm
I20250622 13:59:40.551290 20120 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:59:40.562472 20120 webserver.cc:469] Webserver started at http://127.15.25.65:38585/ using document root <none> and password file <none>
I20250622 13:59:40.563894 20120 fs_manager.cc:362] Metadata directory not provided
I20250622 13:59:40.564239 20120 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:59:40.564889 20120 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:59:40.571879 20120 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b7523a32950c4d57b21211710ab08ccf"
format_stamp: "Formatted at 2025-06-22 13:59:40 on dist-test-slave-6tph"
I20250622 13:59:40.573442 20120 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b7523a32950c4d57b21211710ab08ccf"
format_stamp: "Formatted at 2025-06-22 13:59:40 on dist-test-slave-6tph"
I20250622 13:59:40.583003 20120 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.007s sys 0.001s
I20250622 13:59:40.590446 20136 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:59:40.591696 20120 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20250622 13:59:40.592137 20120 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b7523a32950c4d57b21211710ab08ccf"
format_stamp: "Formatted at 2025-06-22 13:59:40 on dist-test-slave-6tph"
I20250622 13:59:40.592624 20120 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:59:40.664901 20120 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:59:40.666965 20120 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 13:59:40.667582 20120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:59:40.671115 20120 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:59:40.677074 20120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:59:40.677397 20120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:59:40.677812 20120 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:59:40.678129 20120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:59:40.877743 20120 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.65:37005
I20250622 13:59:40.877876 20248 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.65:37005 every 8 connection(s)
I20250622 13:59:40.881184 20120 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250622 13:59:40.881978 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 20120
I20250622 13:59:40.882553 15461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250622 13:59:40.890937 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.25.66:0
--local_ip_for_outbound_sockets=127.15.25.66
--webserver_interface=127.15.25.66
--webserver_port=0
--tserver_master_addrs=127.15.25.126:35699
--builtin_ntp_servers=127.15.25.84:35651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250622 13:59:40.941499 20249 heartbeater.cc:344] Connected to a master server at 127.15.25.126:35699
I20250622 13:59:40.942008 20249 heartbeater.cc:461] Registering TS with master...
I20250622 13:59:40.943354 20249 heartbeater.cc:507] Master 127.15.25.126:35699 requested a full tablet report, sending...
I20250622 13:59:40.946789 20061 ts_manager.cc:194] Registered new tserver with Master: b7523a32950c4d57b21211710ab08ccf (127.15.25.65:37005)
I20250622 13:59:40.949709 20061 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.25.65:50695
W20250622 13:59:41.404764 20253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:59:41.405369 20253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 13:59:41.406077 20253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:59:41.460471 20253 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250622 13:59:41.461037 20253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 13:59:41.462420 20253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.25.66
I20250622 13:59:41.524648 20253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:35651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.25.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.15.25.66
--webserver_port=0
--tserver_master_addrs=127.15.25.126:35699
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.25.66
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 13:59:41.526165 20253 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 13:59:41.528225 20253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:59:41.547174 20260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:59:41.954751 20249 heartbeater.cc:499] Master 127.15.25.126:35699 was elected leader, sending a full tablet report...
W20250622 13:59:42.343391 20245 debug-util.cc:398] Leaking SignalData structure 0x7b08000ac060 after lost signal to thread 20121
W20250622 13:59:42.344799 20245 debug-util.cc:398] Leaking SignalData structure 0x7b08000acf40 after lost signal to thread 20248
W20250622 13:59:41.548305 20259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:59:42.832751 20253 thread.cc:640] GCE (cloud detector) Time spent creating pthread: real 1.288s user 0.465s sys 0.819s
W20250622 13:59:42.833154 20253 thread.cc:606] GCE (cloud detector) Time spent starting thread: real 1.288s user 0.466s sys 0.821s
I20250622 13:59:42.840487 20253 server_base.cc:1048] running on GCE node
W20250622 13:59:42.841364 20265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:59:42.843022 20253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 13:59:42.845752 20253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 13:59:42.847262 20253 hybrid_clock.cc:648] HybridClock initialized: now 1750600782847187 us; error 72 us; skew 500 ppm
I20250622 13:59:42.848534 20253 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:59:42.862926 20253 webserver.cc:469] Webserver started at http://127.15.25.66:34419/ using document root <none> and password file <none>
I20250622 13:59:42.864362 20253 fs_manager.cc:362] Metadata directory not provided
I20250622 13:59:42.864691 20253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:59:42.865314 20253 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:59:42.872702 20253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "001a58be4a4241b9a2a254811eec39e1"
format_stamp: "Formatted at 2025-06-22 13:59:42 on dist-test-slave-6tph"
I20250622 13:59:42.874248 20253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "001a58be4a4241b9a2a254811eec39e1"
format_stamp: "Formatted at 2025-06-22 13:59:42 on dist-test-slave-6tph"
I20250622 13:59:42.884189 20253 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s
I20250622 13:59:42.892226 20270 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:59:42.893548 20253 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20250622 13:59:42.893959 20253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "001a58be4a4241b9a2a254811eec39e1"
format_stamp: "Formatted at 2025-06-22 13:59:42 on dist-test-slave-6tph"
I20250622 13:59:42.894438 20253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:59:42.990249 20253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:59:42.991724 20253 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 13:59:42.992157 20253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:59:42.994812 20253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:59:42.999526 20253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:59:42.999722 20253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:59:42.999984 20253 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:59:43.000130 20253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:59:43.167907 20253 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.66:44705
I20250622 13:59:43.168013 20382 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.66:44705 every 8 connection(s)
I20250622 13:59:43.170591 20253 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250622 13:59:43.172245 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 20253
I20250622 13:59:43.172991 15461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250622 13:59:43.183092 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.25.67:0
--local_ip_for_outbound_sockets=127.15.25.67
--webserver_interface=127.15.25.67
--webserver_port=0
--tserver_master_addrs=127.15.25.126:35699
--builtin_ntp_servers=127.15.25.84:35651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250622 13:59:43.198385 20383 heartbeater.cc:344] Connected to a master server at 127.15.25.126:35699
I20250622 13:59:43.198776 20383 heartbeater.cc:461] Registering TS with master...
I20250622 13:59:43.199715 20383 heartbeater.cc:507] Master 127.15.25.126:35699 requested a full tablet report, sending...
I20250622 13:59:43.201647 20061 ts_manager.cc:194] Registered new tserver with Master: 001a58be4a4241b9a2a254811eec39e1 (127.15.25.66:44705)
I20250622 13:59:43.202800 20061 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.25.66:57531
W20250622 13:59:43.528867 20387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:59:43.529394 20387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 13:59:43.529937 20387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:59:43.558177 20387 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250622 13:59:43.558499 20387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 13:59:43.559173 20387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.25.67
I20250622 13:59:43.590334 20387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:35651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.25.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.15.25.67
--webserver_port=0
--tserver_master_addrs=127.15.25.126:35699
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.25.67
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 13:59:43.591670 20387 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 13:59:43.593250 20387 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:59:43.609670 20393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:59:44.205317 20383 heartbeater.cc:499] Master 127.15.25.126:35699 was elected leader, sending a full tablet report...
W20250622 13:59:43.610260 20394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:59:43.611076 20396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:59:43.612424 20387 server_base.cc:1048] running on GCE node
I20250622 13:59:44.713519 20387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 13:59:44.715629 20387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 13:59:44.716953 20387 hybrid_clock.cc:648] HybridClock initialized: now 1750600784716930 us; error 64 us; skew 500 ppm
I20250622 13:59:44.717779 20387 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:59:44.723937 20387 webserver.cc:469] Webserver started at http://127.15.25.67:37621/ using document root <none> and password file <none>
I20250622 13:59:44.724712 20387 fs_manager.cc:362] Metadata directory not provided
I20250622 13:59:44.724893 20387 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:59:44.725276 20387 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:59:44.729271 20387 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "21b6b7bc00f34a68a32baa5424118ec2"
format_stamp: "Formatted at 2025-06-22 13:59:44 on dist-test-slave-6tph"
I20250622 13:59:44.730293 20387 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "21b6b7bc00f34a68a32baa5424118ec2"
format_stamp: "Formatted at 2025-06-22 13:59:44 on dist-test-slave-6tph"
I20250622 13:59:44.737149 20387 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20250622 13:59:44.742498 20403 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:59:44.743454 20387 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250622 13:59:44.743724 20387 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "21b6b7bc00f34a68a32baa5424118ec2"
format_stamp: "Formatted at 2025-06-22 13:59:44 on dist-test-slave-6tph"
I20250622 13:59:44.743988 20387 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:59:44.795792 20387 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:59:44.797133 20387 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 13:59:44.797524 20387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:59:44.799845 20387 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:59:44.803794 20387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:59:44.803998 20387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:59:44.804190 20387 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:59:44.804318 20387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:59:44.930042 20387 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.67:39195
I20250622 13:59:44.930148 20515 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.67:39195 every 8 connection(s)
I20250622 13:59:44.932581 20387 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250622 13:59:44.935248 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 20387
I20250622 13:59:44.935850 15461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250622 13:59:44.943975 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.25.68:0
--local_ip_for_outbound_sockets=127.15.25.68
--webserver_interface=127.15.25.68
--webserver_port=0
--tserver_master_addrs=127.15.25.126:35699
--builtin_ntp_servers=127.15.25.84:35651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250622 13:59:44.957288 20516 heartbeater.cc:344] Connected to a master server at 127.15.25.126:35699
I20250622 13:59:44.957743 20516 heartbeater.cc:461] Registering TS with master...
I20250622 13:59:44.958745 20516 heartbeater.cc:507] Master 127.15.25.126:35699 requested a full tablet report, sending...
I20250622 13:59:44.960784 20061 ts_manager.cc:194] Registered new tserver with Master: 21b6b7bc00f34a68a32baa5424118ec2 (127.15.25.67:39195)
I20250622 13:59:44.961990 20061 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.25.67:49421
W20250622 13:59:45.244443 20520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:59:45.244940 20520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 13:59:45.245419 20520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:59:45.274506 20520 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250622 13:59:45.274859 20520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 13:59:45.275527 20520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.25.68
I20250622 13:59:45.306866 20520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:35651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.25.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.15.25.68
--webserver_port=0
--tserver_master_addrs=127.15.25.126:35699
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.25.68
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 13:59:45.308195 20520 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 13:59:45.309700 20520 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:59:45.323383 20527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:59:45.964896 20516 heartbeater.cc:499] Master 127.15.25.126:35699 was elected leader, sending a full tablet report...
W20250622 13:59:45.323837 20526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:59:45.325001 20529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:59:45.325904 20520 server_base.cc:1048] running on GCE node
I20250622 13:59:46.427088 20520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 13:59:46.429627 20520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 13:59:46.431221 20520 hybrid_clock.cc:648] HybridClock initialized: now 1750600786431186 us; error 38 us; skew 500 ppm
I20250622 13:59:46.432061 20520 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:59:46.438997 20520 webserver.cc:469] Webserver started at http://127.15.25.68:36269/ using document root <none> and password file <none>
I20250622 13:59:46.439896 20520 fs_manager.cc:362] Metadata directory not provided
I20250622 13:59:46.440116 20520 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:59:46.440570 20520 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:59:46.445022 20520 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "9f14cdeb6a5641e788d4ce37be697439"
format_stamp: "Formatted at 2025-06-22 13:59:46 on dist-test-slave-6tph"
I20250622 13:59:46.446131 20520 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "9f14cdeb6a5641e788d4ce37be697439"
format_stamp: "Formatted at 2025-06-22 13:59:46 on dist-test-slave-6tph"
I20250622 13:59:46.453150 20520 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250622 13:59:46.458652 20536 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:59:46.459604 20520 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250622 13:59:46.459914 20520 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "9f14cdeb6a5641e788d4ce37be697439"
format_stamp: "Formatted at 2025-06-22 13:59:46 on dist-test-slave-6tph"
I20250622 13:59:46.460235 20520 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:59:46.508622 20520 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:59:46.510071 20520 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 13:59:46.510488 20520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:59:46.512969 20520 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:59:46.516930 20520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:59:46.517145 20520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:59:46.517396 20520 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:59:46.517556 20520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:59:46.645241 20520 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.68:46517
I20250622 13:59:46.645340 20648 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.68:46517 every 8 connection(s)
I20250622 13:59:46.647737 20520 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250622 13:59:46.656248 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 20520
I20250622 13:59:46.656662 15461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250622 13:59:46.662359 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.25.69:0
--local_ip_for_outbound_sockets=127.15.25.69
--webserver_interface=127.15.25.69
--webserver_port=0
--tserver_master_addrs=127.15.25.126:35699
--builtin_ntp_servers=127.15.25.84:35651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250622 13:59:46.669116 20649 heartbeater.cc:344] Connected to a master server at 127.15.25.126:35699
I20250622 13:59:46.669626 20649 heartbeater.cc:461] Registering TS with master...
I20250622 13:59:46.670913 20649 heartbeater.cc:507] Master 127.15.25.126:35699 requested a full tablet report, sending...
I20250622 13:59:46.673074 20061 ts_manager.cc:194] Registered new tserver with Master: 9f14cdeb6a5641e788d4ce37be697439 (127.15.25.68:46517)
I20250622 13:59:46.674870 20061 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.25.68:43307
W20250622 13:59:46.950927 20653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:59:46.951368 20653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 13:59:46.951850 20653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:59:46.984401 20653 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250622 13:59:46.984764 20653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 13:59:46.985483 20653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.25.69
I20250622 13:59:47.019616 20653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:35651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.25.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.15.25.69
--webserver_port=0
--tserver_master_addrs=127.15.25.126:35699
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.25.69
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 13:59:47.020790 20653 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 13:59:47.022230 20653 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:59:47.039340 20660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:59:47.678184 20649 heartbeater.cc:499] Master 127.15.25.126:35699 was elected leader, sending a full tablet report...
W20250622 13:59:47.039994 20659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:59:47.041129 20662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:59:48.157433 20661 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250622 13:59:48.157572 20653 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 13:59:48.161602 20653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 13:59:48.163748 20653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 13:59:48.165117 20653 hybrid_clock.cc:648] HybridClock initialized: now 1750600788165087 us; error 41 us; skew 500 ppm
I20250622 13:59:48.166147 20653 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:59:48.176976 20653 webserver.cc:469] Webserver started at http://127.15.25.69:44097/ using document root <none> and password file <none>
I20250622 13:59:48.177946 20653 fs_manager.cc:362] Metadata directory not provided
I20250622 13:59:48.178330 20653 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:59:48.178855 20653 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:59:48.183472 20653 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "69146f2ec8b44d898791204ce591b9eb"
format_stamp: "Formatted at 2025-06-22 13:59:48 on dist-test-slave-6tph"
I20250622 13:59:48.184600 20653 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "69146f2ec8b44d898791204ce591b9eb"
format_stamp: "Formatted at 2025-06-22 13:59:48 on dist-test-slave-6tph"
I20250622 13:59:48.191785 20653 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20250622 13:59:48.197819 20670 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:59:48.198787 20653 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20250622 13:59:48.199074 20653 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "69146f2ec8b44d898791204ce591b9eb"
format_stamp: "Formatted at 2025-06-22 13:59:48 on dist-test-slave-6tph"
I20250622 13:59:48.199364 20653 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:59:48.255280 20653 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:59:48.256683 20653 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 13:59:48.257097 20653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:59:48.259604 20653 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:59:48.263469 20653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:59:48.263682 20653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:59:48.263919 20653 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:59:48.264070 20653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:59:48.392789 20653 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.69:32887
I20250622 13:59:48.393052 20782 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.69:32887 every 8 connection(s)
I20250622 13:59:48.395270 20653 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250622 13:59:48.402606 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 20653
I20250622 13:59:48.403012 15461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250622 13:59:48.416179 20783 heartbeater.cc:344] Connected to a master server at 127.15.25.126:35699
I20250622 13:59:48.416567 20783 heartbeater.cc:461] Registering TS with master...
I20250622 13:59:48.417541 20783 heartbeater.cc:507] Master 127.15.25.126:35699 requested a full tablet report, sending...
I20250622 13:59:48.419481 20061 ts_manager.cc:194] Registered new tserver with Master: 69146f2ec8b44d898791204ce591b9eb (127.15.25.69:32887)
I20250622 13:59:48.421504 20061 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.25.69:42381
I20250622 13:59:48.422940 15461 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250622 13:59:48.459859 20061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46940:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250622 13:59:48.542541 20718 tablet_service.cc:1468] Processing CreateTablet for tablet 471ace9ca3be41b8bef17fc6da185986 (DEFAULT_TABLE table=TestTable [id=610388d0a1f24cba956b368409db8b7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 13:59:48.545053 20718 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 471ace9ca3be41b8bef17fc6da185986. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:59:48.545648 20584 tablet_service.cc:1468] Processing CreateTablet for tablet 471ace9ca3be41b8bef17fc6da185986 (DEFAULT_TABLE table=TestTable [id=610388d0a1f24cba956b368409db8b7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 13:59:48.546599 20184 tablet_service.cc:1468] Processing CreateTablet for tablet 471ace9ca3be41b8bef17fc6da185986 (DEFAULT_TABLE table=TestTable [id=610388d0a1f24cba956b368409db8b7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 13:59:48.547434 20584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 471ace9ca3be41b8bef17fc6da185986. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:59:48.548360 20184 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 471ace9ca3be41b8bef17fc6da185986. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:59:48.569978 20802 tablet_bootstrap.cc:492] T 471ace9ca3be41b8bef17fc6da185986 P 69146f2ec8b44d898791204ce591b9eb: Bootstrap starting.
I20250622 13:59:48.576959 20802 tablet_bootstrap.cc:654] T 471ace9ca3be41b8bef17fc6da185986 P 69146f2ec8b44d898791204ce591b9eb: Neither blocks nor log segments found. Creating new log.
I20250622 13:59:48.577900 20803 tablet_bootstrap.cc:492] T 471ace9ca3be41b8bef17fc6da185986 P 9f14cdeb6a5641e788d4ce37be697439: Bootstrap starting.
I20250622 13:59:48.579722 20802 log.cc:826] T 471ace9ca3be41b8bef17fc6da185986 P 69146f2ec8b44d898791204ce591b9eb: Log is configured to *not* fsync() on all Append() calls
I20250622 13:59:48.583173 20804 tablet_bootstrap.cc:492] T 471ace9ca3be41b8bef17fc6da185986 P b7523a32950c4d57b21211710ab08ccf: Bootstrap starting.
I20250622 13:59:48.585512 20803 tablet_bootstrap.cc:654] T 471ace9ca3be41b8bef17fc6da185986 P 9f14cdeb6a5641e788d4ce37be697439: Neither blocks nor log segments found. Creating new log.
I20250622 13:59:48.585711 20802 tablet_bootstrap.cc:492] T 471ace9ca3be41b8bef17fc6da185986 P 69146f2ec8b44d898791204ce591b9eb: No bootstrap required, opened a new log
I20250622 13:59:48.586230 20802 ts_tablet_manager.cc:1397] T 471ace9ca3be41b8bef17fc6da185986 P 69146f2ec8b44d898791204ce591b9eb: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.011s
I20250622 13:59:48.587724 20803 log.cc:826] T 471ace9ca3be41b8bef17fc6da185986 P 9f14cdeb6a5641e788d4ce37be697439: Log is configured to *not* fsync() on all Append() calls
I20250622 13:59:48.590979 20804 tablet_bootstrap.cc:654] T 471ace9ca3be41b8bef17fc6da185986 P b7523a32950c4d57b21211710ab08ccf: Neither blocks nor log segments found. Creating new log.
I20250622 13:59:48.592993 20804 log.cc:826] T 471ace9ca3be41b8bef17fc6da185986 P b7523a32950c4d57b21211710ab08ccf: Log is configured to *not* fsync() on all Append() calls
I20250622 13:59:48.593328 20803 tablet_bootstrap.cc:492] T 471ace9ca3be41b8bef17fc6da185986 P 9f14cdeb6a5641e788d4ce37be697439: No bootstrap required, opened a new log
I20250622 13:59:48.593760 20803 ts_tablet_manager.cc:1397] T 471ace9ca3be41b8bef17fc6da185986 P 9f14cdeb6a5641e788d4ce37be697439: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.007s
I20250622 13:59:48.598980 20804 tablet_bootstrap.cc:492] T 471ace9ca3be41b8bef17fc6da185986 P b7523a32950c4d57b21211710ab08ccf: No bootstrap required, opened a new log
I20250622 13:59:48.599385 20804 ts_tablet_manager.cc:1397] T 471ace9ca3be41b8bef17fc6da185986 P b7523a32950c4d57b21211710ab08ccf: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.005s
I20250622 13:59:48.611886 20803 raft_consensus.cc:357] T 471ace9ca3be41b8bef17fc6da185986 P 9f14cdeb6a5641e788d4ce37be697439 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7523a32950c4d57b21211710ab08ccf" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 37005 } } peers { permanent_uuid: "69146f2ec8b44d898791204ce591b9eb" member_type: VOTER last_known_addr { host: "127.15.25.69" port: 32887 } } peers { permanent_uuid: "9f14cdeb6a5641e788d4ce37be697439" member_type: VOTER last_known_addr { host: "127.15.25.68" port: 46517 } }
I20250622 13:59:48.612541 20803 raft_consensus.cc:383] T 471ace9ca3be41b8bef17fc6da185986 P 9f14cdeb6a5641e788d4ce37be697439 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:59:48.612807 20803 raft_consensus.cc:738] T 471ace9ca3be41b8bef17fc6da185986 P 9f14cdeb6a5641e788d4ce37be697439 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f14cdeb6a5641e788d4ce37be697439, State: Initialized, Role: FOLLOWER
I20250622 13:59:48.612619 20802 raft_consensus.cc:357] T 471ace9ca3be41b8bef17fc6da185986 P 69146f2ec8b44d898791204ce591b9eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7523a32950c4d57b21211710ab08ccf" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 37005 } } peers { permanent_uuid: "69146f2ec8b44d898791204ce591b9eb" member_type: VOTER last_known_addr { host: "127.15.25.69" port: 32887 } } peers { permanent_uuid: "9f14cdeb6a5641e788d4ce37be697439" member_type: VOTER last_known_addr { host: "127.15.25.68" port: 46517 } }
I20250622 13:59:48.613399 20802 raft_consensus.cc:383] T 471ace9ca3be41b8bef17fc6da185986 P 69146f2ec8b44d898791204ce591b9eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:59:48.613663 20802 raft_consensus.cc:738] T 471ace9ca3be41b8bef17fc6da185986 P 69146f2ec8b44d898791204ce591b9eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69146f2ec8b44d898791204ce591b9eb, State: Initialized, Role: FOLLOWER
I20250622 13:59:48.613507 20803 consensus_queue.cc:260] T 471ace9ca3be41b8bef17fc6da185986 P 9f14cdeb6a5641e788d4ce37be697439 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7523a32950c4d57b21211710ab08ccf" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 37005 } } peers { permanent_uuid: "69146f2ec8b44d898791204ce591b9eb" member_type: VOTER last_known_addr { host: "127.15.25.69" port: 32887 } } peers { permanent_uuid: "9f14cdeb6a5641e788d4ce37be697439" member_type: VOTER last_known_addr { host: "127.15.25.68" port: 46517 } }
I20250622 13:59:48.614414 20802 consensus_queue.cc:260] T 471ace9ca3be41b8bef17fc6da185986 P 69146f2ec8b44d898791204ce591b9eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7523a32950c4d57b21211710ab08ccf" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 37005 } } peers { permanent_uuid: "69146f2ec8b44d898791204ce591b9eb" member_type: VOTER last_known_addr { host: "127.15.25.69" port: 32887 } } peers { permanent_uuid: "9f14cdeb6a5641e788d4ce37be697439" member_type: VOTER last_known_addr { host: "127.15.25.68" port: 46517 } }
I20250622 13:59:48.618183 20803 ts_tablet_manager.cc:1428] T 471ace9ca3be41b8bef17fc6da185986 P 9f14cdeb6a5641e788d4ce37be697439: Time spent starting tablet: real 0.024s user 0.022s sys 0.002s
I20250622 13:59:48.622470 20783 heartbeater.cc:499] Master 127.15.25.126:35699 was elected leader, sending a full tablet report...
I20250622 13:59:48.624416 20804 raft_consensus.cc:357] T 471ace9ca3be41b8bef17fc6da185986 P b7523a32950c4d57b21211710ab08ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7523a32950c4d57b21211710ab08ccf" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 37005 } } peers { permanent_uuid: "69146f2ec8b44d898791204ce591b9eb" member_type: VOTER last_known_addr { host: "127.15.25.69" port: 32887 } } peers { permanent_uuid: "9f14cdeb6a5641e788d4ce37be697439" member_type: VOTER last_known_addr { host: "127.15.25.68" port: 46517 } }
I20250622 13:59:48.625350 20804 raft_consensus.cc:383] T 471ace9ca3be41b8bef17fc6da185986 P b7523a32950c4d57b21211710ab08ccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:59:48.625784 20802 ts_tablet_manager.cc:1428] T 471ace9ca3be41b8bef17fc6da185986 P 69146f2ec8b44d898791204ce591b9eb: Time spent starting tablet: real 0.039s user 0.031s sys 0.005s
I20250622 13:59:48.625687 20804 raft_consensus.cc:738] T 471ace9ca3be41b8bef17fc6da185986 P b7523a32950c4d57b21211710ab08ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7523a32950c4d57b21211710ab08ccf, State: Initialized, Role: FOLLOWER
I20250622 13:59:48.627013 20804 consensus_queue.cc:260] T 471ace9ca3be41b8bef17fc6da185986 P b7523a32950c4d57b21211710ab08ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7523a32950c4d57b21211710ab08ccf" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 37005 } } peers { permanent_uuid: "69146f2ec8b44d898791204ce591b9eb" member_type: VOTER last_known_addr { host: "127.15.25.69" port: 32887 } } peers { permanent_uuid: "9f14cdeb6a5641e788d4ce37be697439" member_type: VOTER last_known_addr { host: "127.15.25.68" port: 46517 } }
I20250622 13:59:48.632313 20804 ts_tablet_manager.cc:1428] T 471ace9ca3be41b8bef17fc6da185986 P b7523a32950c4d57b21211710ab08ccf: Time spent starting tablet: real 0.033s user 0.029s sys 0.003s
W20250622 13:59:48.649915 20784 tablet.cc:2378] T 471ace9ca3be41b8bef17fc6da185986 P 69146f2ec8b44d898791204ce591b9eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250622 13:59:48.654328 20650 tablet.cc:2378] T 471ace9ca3be41b8bef17fc6da185986 P 9f14cdeb6a5641e788d4ce37be697439: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250622 13:59:48.672746 20250 tablet.cc:2378] T 471ace9ca3be41b8bef17fc6da185986 P b7523a32950c4d57b21211710ab08ccf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 13:59:48.705638 20810 raft_consensus.cc:491] T 471ace9ca3be41b8bef17fc6da185986 P b7523a32950c4d57b21211710ab08ccf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:59:48.706113 20810 raft_consensus.cc:513] T 471ace9ca3be41b8bef17fc6da185986 P b7523a32950c4d57b21211710ab08ccf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7523a32950c4d57b21211710ab08ccf" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 37005 } } peers { permanent_uuid: "69146f2ec8b44d898791204ce591b9eb" member_type: VOTER last_known_addr { host: "127.15.25.69" port: 32887 } } peers { permanent_uuid: "9f14cdeb6a5641e788d4ce37be697439" member_type: VOTER last_known_addr { host: "127.15.25.68" port: 46517 } }
I20250622 13:59:48.708307 20810 leader_election.cc:290] T 471ace9ca3be41b8bef17fc6da185986 P b7523a32950c4d57b21211710ab08ccf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69146f2ec8b44d898791204ce591b9eb (127.15.25.69:32887), 9f14cdeb6a5641e788d4ce37be697439 (127.15.25.68:46517)
I20250622 13:59:48.721421 20738 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "471ace9ca3be41b8bef17fc6da185986" candidate_uuid: "b7523a32950c4d57b21211710ab08ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69146f2ec8b44d898791204ce591b9eb" is_pre_election: true
I20250622 13:59:48.721477 20604 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "471ace9ca3be41b8bef17fc6da185986" candidate_uuid: "b7523a32950c4d57b21211710ab08ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f14cdeb6a5641e788d4ce37be697439" is_pre_election: true
I20250622 13:59:48.722093 20738 raft_consensus.cc:2466] T 471ace9ca3be41b8bef17fc6da185986 P 69146f2ec8b44d898791204ce591b9eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7523a32950c4d57b21211710ab08ccf in term 0.
I20250622 13:59:48.722491 20604 raft_consensus.cc:2466] T 471ace9ca3be41b8bef17fc6da185986 P 9f14cdeb6a5641e788d4ce37be697439 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7523a32950c4d57b21211710ab08ccf in term 0.
I20250622 13:59:48.723171 20138 leader_election.cc:304] T 471ace9ca3be41b8bef17fc6da185986 P b7523a32950c4d57b21211710ab08ccf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69146f2ec8b44d898791204ce591b9eb, b7523a32950c4d57b21211710ab08ccf; no voters:
I20250622 13:59:48.723838 20810 raft_consensus.cc:2802] T 471ace9ca3be41b8bef17fc6da185986 P b7523a32950c4d57b21211710ab08ccf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 13:59:48.724100 20810 raft_consensus.cc:491] T 471ace9ca3be41b8bef17fc6da185986 P b7523a32950c4d57b21211710ab08ccf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:59:48.724352 20810 raft_consensus.cc:3058] T 471ace9ca3be41b8bef17fc6da185986 P b7523a32950c4d57b21211710ab08ccf [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:59:48.728812 20810 raft_consensus.cc:513] T 471ace9ca3be41b8bef17fc6da185986 P b7523a32950c4d57b21211710ab08ccf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7523a32950c4d57b21211710ab08ccf" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 37005 } } peers { permanent_uuid: "69146f2ec8b44d898791204ce591b9eb" member_type: VOTER last_known_addr { host: "127.15.25.69" port: 32887 } } peers { permanent_uuid: "9f14cdeb6a5641e788d4ce37be697439" member_type: VOTER last_known_addr { host: "127.15.25.68" port: 46517 } }
I20250622 13:59:48.730190 20810 leader_election.cc:290] T 471ace9ca3be41b8bef17fc6da185986 P b7523a32950c4d57b21211710ab08ccf [CANDIDATE]: Term 1 election: Requested vote from peers 69146f2ec8b44d898791204ce591b9eb (127.15.25.69:32887), 9f14cdeb6a5641e788d4ce37be697439 (127.15.25.68:46517)
I20250622 13:59:48.730792 20738 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "471ace9ca3be41b8bef17fc6da185986" candidate_uuid: "b7523a32950c4d57b21211710ab08ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69146f2ec8b44d898791204ce591b9eb"
I20250622 13:59:48.731083 20604 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "471ace9ca3be41b8bef17fc6da185986" candidate_uuid: "b7523a32950c4d57b21211710ab08ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f14cdeb6a5641e788d4ce37be697439"
I20250622 13:59:48.731211 20738 raft_consensus.cc:3058] T 471ace9ca3be41b8bef17fc6da185986 P 69146f2ec8b44d898791204ce591b9eb [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:59:48.731544 20604 raft_consensus.cc:3058] T 471ace9ca3be41b8bef17fc6da185986 P 9f14cdeb6a5641e788d4ce37be697439 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:59:48.737183 20604 raft_consensus.cc:2466] T 471ace9ca3be41b8bef17fc6da185986 P 9f14cdeb6a5641e788d4ce37be697439 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7523a32950c4d57b21211710ab08ccf in term 1.
I20250622 13:59:48.737296 20738 raft_consensus.cc:2466] T 471ace9ca3be41b8bef17fc6da185986 P 69146f2ec8b44d898791204ce591b9eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7523a32950c4d57b21211710ab08ccf in term 1.
I20250622 13:59:48.738134 20137 leader_election.cc:304] T 471ace9ca3be41b8bef17fc6da185986 P b7523a32950c4d57b21211710ab08ccf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f14cdeb6a5641e788d4ce37be697439, b7523a32950c4d57b21211710ab08ccf; no voters:
I20250622 13:59:48.738744 20810 raft_consensus.cc:2802] T 471ace9ca3be41b8bef17fc6da185986 P b7523a32950c4d57b21211710ab08ccf [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:59:48.740407 20810 raft_consensus.cc:695] T 471ace9ca3be41b8bef17fc6da185986 P b7523a32950c4d57b21211710ab08ccf [term 1 LEADER]: Becoming Leader. State: Replica: b7523a32950c4d57b21211710ab08ccf, State: Running, Role: LEADER
I20250622 13:59:48.741042 20810 consensus_queue.cc:237] T 471ace9ca3be41b8bef17fc6da185986 P b7523a32950c4d57b21211710ab08ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7523a32950c4d57b21211710ab08ccf" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 37005 } } peers { permanent_uuid: "69146f2ec8b44d898791204ce591b9eb" member_type: VOTER last_known_addr { host: "127.15.25.69" port: 32887 } } peers { permanent_uuid: "9f14cdeb6a5641e788d4ce37be697439" member_type: VOTER last_known_addr { host: "127.15.25.68" port: 46517 } }
I20250622 13:59:48.749260 20061 catalog_manager.cc:5582] T 471ace9ca3be41b8bef17fc6da185986 P b7523a32950c4d57b21211710ab08ccf reported cstate change: term changed from 0 to 1, leader changed from <none> to b7523a32950c4d57b21211710ab08ccf (127.15.25.65). New cstate: current_term: 1 leader_uuid: "b7523a32950c4d57b21211710ab08ccf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7523a32950c4d57b21211710ab08ccf" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 37005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69146f2ec8b44d898791204ce591b9eb" member_type: VOTER last_known_addr { host: "127.15.25.69" port: 32887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f14cdeb6a5641e788d4ce37be697439" member_type: VOTER last_known_addr { host: "127.15.25.68" port: 46517 } health_report { overall_health: UNKNOWN } } }
I20250622 13:59:48.790369 15461 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250622 13:59:48.793378 15461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver b7523a32950c4d57b21211710ab08ccf to finish bootstrapping
I20250622 13:59:48.805481 15461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 9f14cdeb6a5641e788d4ce37be697439 to finish bootstrapping
I20250622 13:59:48.815732 15461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 69146f2ec8b44d898791204ce591b9eb to finish bootstrapping
I20250622 13:59:48.825663 15461 test_util.cc:276] Using random seed: -2111220180
I20250622 13:59:48.848646 15461 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20250622 13:59:48.849476 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 20120
W20250622 13:59:48.877346 20820 negotiation.cc:337] Failed RPC negotiation. Trace:
0622 13:59:48.860463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.25.65:37005 (local address 127.0.0.1:42036)
0622 13:59:48.861325 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0622 13:59:48.861387 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0622 13:59:48.861647 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0622 13:59:48.876064 (+ 14417us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.25.65:37005: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":77}
W20250622 13:59:48.888195 20818 meta_cache.cc:302] tablet 471ace9ca3be41b8bef17fc6da185986: replica b7523a32950c4d57b21211710ab08ccf (127.15.25.65:37005) has failed: Network error: Client connection negotiation failed: client connection to 127.15.25.65:37005: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250622 13:59:48.905298 20698 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:58186: Illegal state: replica 69146f2ec8b44d898791204ce591b9eb is not leader of this config: current role FOLLOWER
W20250622 13:59:48.925319 20564 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38658: Illegal state: replica 9f14cdeb6a5641e788d4ce37be697439 is not leader of this config: current role FOLLOWER
W20250622 13:59:48.948014 20698 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:58186: Illegal state: replica 69146f2ec8b44d898791204ce591b9eb is not leader of this config: current role FOLLOWER
W20250622 13:59:48.956836 20564 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38658: Illegal state: replica 9f14cdeb6a5641e788d4ce37be697439 is not leader of this config: current role FOLLOWER
W20250622 13:59:48.981447 20698 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:58186: Illegal state: replica 69146f2ec8b44d898791204ce591b9eb is not leader of this config: current role FOLLOWER
W20250622 13:59:48.994028 20564 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38658: Illegal state: replica 9f14cdeb6a5641e788d4ce37be697439 is not leader of this config: current role FOLLOWER
W20250622 13:59:49.026558 20698 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:58186: Illegal state: replica 69146f2ec8b44d898791204ce591b9eb is not leader of this config: current role FOLLOWER
W20250622 13:59:49.041981 20564 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38658: Illegal state: replica 9f14cdeb6a5641e788d4ce37be697439 is not leader of this config: current role FOLLOWER
W20250622 13:59:49.079150 20698 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:58186: Illegal state: replica 69146f2ec8b44d898791204ce591b9eb is not leader of this config: current role FOLLOWER
W20250622 13:59:49.098537 20564 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38658: Illegal state: replica 9f14cdeb6a5641e788d4ce37be697439 is not leader of this config: current role FOLLOWER
W20250622 13:59:49.139770 20698 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:58186: Illegal state: replica 69146f2ec8b44d898791204ce591b9eb is not leader of this config: current role FOLLOWER
W20250622 13:59:49.160538 20564 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38658: Illegal state: replica 9f14cdeb6a5641e788d4ce37be697439 is not leader of this config: current role FOLLOWER
W20250622 13:59:49.208694 20698 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:58186: Illegal state: replica 69146f2ec8b44d898791204ce591b9eb is not leader of this config: current role FOLLOWER
W20250622 13:59:49.235584 20564 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38658: Illegal state: replica 9f14cdeb6a5641e788d4ce37be697439 is not leader of this config: current role FOLLOWER
W20250622 13:59:49.293092 20698 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:58186: Illegal state: replica 69146f2ec8b44d898791204ce591b9eb is not leader of this config: current role FOLLOWER
W20250622 13:59:49.322351 20564 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38658: Illegal state: replica 9f14cdeb6a5641e788d4ce37be697439 is not leader of this config: current role FOLLOWER
W20250622 13:59:49.388295 20698 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:58186: Illegal state: replica 69146f2ec8b44d898791204ce591b9eb is not leader of this config: current role FOLLOWER
W20250622 13:59:49.423516 20564 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38658: Illegal state: replica 9f14cdeb6a5641e788d4ce37be697439 is not leader of this config: current role FOLLOWER
W20250622 13:59:49.495057 20698 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:58186: Illegal state: replica 69146f2ec8b44d898791204ce591b9eb is not leader of this config: current role FOLLOWER
W20250622 13:59:49.533869 20564 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38658: Illegal state: replica 9f14cdeb6a5641e788d4ce37be697439 is not leader of this config: current role FOLLOWER
W20250622 13:59:49.613155 20698 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:58186: Illegal state: replica 69146f2ec8b44d898791204ce591b9eb is not leader of this config: current role FOLLOWER
W20250622 13:59:49.653052 20564 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38658: Illegal state: replica 9f14cdeb6a5641e788d4ce37be697439 is not leader of this config: current role FOLLOWER
W20250622 13:59:49.742452 20698 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:58186: Illegal state: replica 69146f2ec8b44d898791204ce591b9eb is not leader of this config: current role FOLLOWER
W20250622 13:59:49.784443 20564 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38658: Illegal state: replica 9f14cdeb6a5641e788d4ce37be697439 is not leader of this config: current role FOLLOWER
W20250622 13:59:49.876004 20698 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:58186: Illegal state: replica 69146f2ec8b44d898791204ce591b9eb is not leader of this config: current role FOLLOWER
W20250622 13:59:49.921280 20564 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38658: Illegal state: replica 9f14cdeb6a5641e788d4ce37be697439 is not leader of this config: current role FOLLOWER
W20250622 13:59:49.974254 20818 meta_cache.cc:302] tablet 471ace9ca3be41b8bef17fc6da185986: replica b7523a32950c4d57b21211710ab08ccf (127.15.25.65:37005) has failed: Network error: Client connection negotiation failed: client connection to 127.15.25.65:37005: connect: Connection refused (error 111)
W20250622 13:59:50.018877 20698 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:58186: Illegal state: replica 69146f2ec8b44d898791204ce591b9eb is not leader of this config: current role FOLLOWER
W20250622 13:59:50.069640 20564 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38658: Illegal state: replica 9f14cdeb6a5641e788d4ce37be697439 is not leader of this config: current role FOLLOWER
W20250622 13:59:50.176733 20698 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:58186: Illegal state: replica 69146f2ec8b44d898791204ce591b9eb is not leader of this config: current role FOLLOWER
W20250622 13:59:50.229864 20564 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38658: Illegal state: replica 9f14cdeb6a5641e788d4ce37be697439 is not leader of this config: current role FOLLOWER
I20250622 13:59:50.334666 20833 raft_consensus.cc:491] T 471ace9ca3be41b8bef17fc6da185986 P 69146f2ec8b44d898791204ce591b9eb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 13:59:50.335142 20833 raft_consensus.cc:513] T 471ace9ca3be41b8bef17fc6da185986 P 69146f2ec8b44d898791204ce591b9eb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7523a32950c4d57b21211710ab08ccf" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 37005 } } peers { permanent_uuid: "69146f2ec8b44d898791204ce591b9eb" member_type: VOTER last_known_addr { host: "127.15.25.69" port: 32887 } } peers { permanent_uuid: "9f14cdeb6a5641e788d4ce37be697439" member_type: VOTER last_known_addr { host: "127.15.25.68" port: 46517 } }
I20250622 13:59:50.337922 20833 leader_election.cc:290] T 471ace9ca3be41b8bef17fc6da185986 P 69146f2ec8b44d898791204ce591b9eb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7523a32950c4d57b21211710ab08ccf (127.15.25.65:37005), 9f14cdeb6a5641e788d4ce37be697439 (127.15.25.68:46517)
W20250622 13:59:50.342711 20698 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:58186: Illegal state: replica 69146f2ec8b44d898791204ce591b9eb is not leader of this config: current role FOLLOWER
W20250622 13:59:50.354439 20673 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.25.65:37005: connect: Connection refused (error 111)
I20250622 13:59:50.356274 20604 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "471ace9ca3be41b8bef17fc6da185986" candidate_uuid: "69146f2ec8b44d898791204ce591b9eb" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f14cdeb6a5641e788d4ce37be697439" is_pre_election: true
I20250622 13:59:50.357045 20604 raft_consensus.cc:2466] T 471ace9ca3be41b8bef17fc6da185986 P 9f14cdeb6a5641e788d4ce37be697439 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69146f2ec8b44d898791204ce591b9eb in term 1.
I20250622 13:59:50.358911 20671 leader_election.cc:304] T 471ace9ca3be41b8bef17fc6da185986 P 69146f2ec8b44d898791204ce591b9eb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69146f2ec8b44d898791204ce591b9eb, 9f14cdeb6a5641e788d4ce37be697439; no voters:
I20250622 13:59:50.359972 20833 raft_consensus.cc:2802] T 471ace9ca3be41b8bef17fc6da185986 P 69146f2ec8b44d898791204ce591b9eb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250622 13:59:50.360360 20833 raft_consensus.cc:491] T 471ace9ca3be41b8bef17fc6da185986 P 69146f2ec8b44d898791204ce591b9eb [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 13:59:50.360744 20833 raft_consensus.cc:3058] T 471ace9ca3be41b8bef17fc6da185986 P 69146f2ec8b44d898791204ce591b9eb [term 1 FOLLOWER]: Advancing to term 2
W20250622 13:59:50.364238 20673 leader_election.cc:336] T 471ace9ca3be41b8bef17fc6da185986 P 69146f2ec8b44d898791204ce591b9eb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7523a32950c4d57b21211710ab08ccf (127.15.25.65:37005): Network error: Client connection negotiation failed: client connection to 127.15.25.65:37005: connect: Connection refused (error 111)
I20250622 13:59:50.366533 20833 raft_consensus.cc:513] T 471ace9ca3be41b8bef17fc6da185986 P 69146f2ec8b44d898791204ce591b9eb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7523a32950c4d57b21211710ab08ccf" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 37005 } } peers { permanent_uuid: "69146f2ec8b44d898791204ce591b9eb" member_type: VOTER last_known_addr { host: "127.15.25.69" port: 32887 } } peers { permanent_uuid: "9f14cdeb6a5641e788d4ce37be697439" member_type: VOTER last_known_addr { host: "127.15.25.68" port: 46517 } }
I20250622 13:59:50.368969 20604 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "471ace9ca3be41b8bef17fc6da185986" candidate_uuid: "69146f2ec8b44d898791204ce591b9eb" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f14cdeb6a5641e788d4ce37be697439"
I20250622 13:59:50.369444 20604 raft_consensus.cc:3058] T 471ace9ca3be41b8bef17fc6da185986 P 9f14cdeb6a5641e788d4ce37be697439 [term 1 FOLLOWER]: Advancing to term 2
W20250622 13:59:50.371646 20673 leader_election.cc:336] T 471ace9ca3be41b8bef17fc6da185986 P 69146f2ec8b44d898791204ce591b9eb [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b7523a32950c4d57b21211710ab08ccf (127.15.25.65:37005): Network error: Client connection negotiation failed: client connection to 127.15.25.65:37005: connect: Connection refused (error 111)
I20250622 13:59:50.372366 20833 leader_election.cc:290] T 471ace9ca3be41b8bef17fc6da185986 P 69146f2ec8b44d898791204ce591b9eb [CANDIDATE]: Term 2 election: Requested vote from peers b7523a32950c4d57b21211710ab08ccf (127.15.25.65:37005), 9f14cdeb6a5641e788d4ce37be697439 (127.15.25.68:46517)
I20250622 13:59:50.374826 20604 raft_consensus.cc:2466] T 471ace9ca3be41b8bef17fc6da185986 P 9f14cdeb6a5641e788d4ce37be697439 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69146f2ec8b44d898791204ce591b9eb in term 2.
I20250622 13:59:50.375579 20671 leader_election.cc:304] T 471ace9ca3be41b8bef17fc6da185986 P 69146f2ec8b44d898791204ce591b9eb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 69146f2ec8b44d898791204ce591b9eb, 9f14cdeb6a5641e788d4ce37be697439; no voters: b7523a32950c4d57b21211710ab08ccf
I20250622 13:59:50.376127 20833 raft_consensus.cc:2802] T 471ace9ca3be41b8bef17fc6da185986 P 69146f2ec8b44d898791204ce591b9eb [term 2 FOLLOWER]: Leader election won for term 2
I20250622 13:59:50.378113 20833 raft_consensus.cc:695] T 471ace9ca3be41b8bef17fc6da185986 P 69146f2ec8b44d898791204ce591b9eb [term 2 LEADER]: Becoming Leader. State: Replica: 69146f2ec8b44d898791204ce591b9eb, State: Running, Role: LEADER
I20250622 13:59:50.378906 20833 consensus_queue.cc:237] T 471ace9ca3be41b8bef17fc6da185986 P 69146f2ec8b44d898791204ce591b9eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7523a32950c4d57b21211710ab08ccf" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 37005 } } peers { permanent_uuid: "69146f2ec8b44d898791204ce591b9eb" member_type: VOTER last_known_addr { host: "127.15.25.69" port: 32887 } } peers { permanent_uuid: "9f14cdeb6a5641e788d4ce37be697439" member_type: VOTER last_known_addr { host: "127.15.25.68" port: 46517 } }
I20250622 13:59:50.388918 20060 catalog_manager.cc:5582] T 471ace9ca3be41b8bef17fc6da185986 P 69146f2ec8b44d898791204ce591b9eb reported cstate change: term changed from 1 to 2, leader changed from b7523a32950c4d57b21211710ab08ccf (127.15.25.65) to 69146f2ec8b44d898791204ce591b9eb (127.15.25.69). New cstate: current_term: 2 leader_uuid: "69146f2ec8b44d898791204ce591b9eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7523a32950c4d57b21211710ab08ccf" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 37005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69146f2ec8b44d898791204ce591b9eb" member_type: VOTER last_known_addr { host: "127.15.25.69" port: 32887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f14cdeb6a5641e788d4ce37be697439" member_type: VOTER last_known_addr { host: "127.15.25.68" port: 46517 } health_report { overall_health: UNKNOWN } } }
W20250622 13:59:50.401630 20564 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38658: Illegal state: replica 9f14cdeb6a5641e788d4ce37be697439 is not leader of this config: current role FOLLOWER
I20250622 13:59:50.475325 20604 raft_consensus.cc:1273] T 471ace9ca3be41b8bef17fc6da185986 P 9f14cdeb6a5641e788d4ce37be697439 [term 2 FOLLOWER]: Refusing update from remote peer 69146f2ec8b44d898791204ce591b9eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20250622 13:59:50.477308 20673 consensus_peers.cc:487] T 471ace9ca3be41b8bef17fc6da185986 P 69146f2ec8b44d898791204ce591b9eb -> Peer b7523a32950c4d57b21211710ab08ccf (127.15.25.65:37005): Couldn't send request to peer b7523a32950c4d57b21211710ab08ccf. Status: Network error: Client connection negotiation failed: client connection to 127.15.25.65:37005: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250622 13:59:50.477960 20833 consensus_queue.cc:1035] T 471ace9ca3be41b8bef17fc6da185986 P 69146f2ec8b44d898791204ce591b9eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f14cdeb6a5641e788d4ce37be697439" member_type: VOTER last_known_addr { host: "127.15.25.68" port: 46517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250622 13:59:50.498299 20844 mvcc.cc:204] Tried to move back new op lower bound from 7170460837758545920 to 7170460837398515712. Current Snapshot: MvccSnapshot[applied={T|T < 7170460837758545920}]
I20250622 13:59:50.958494 20451 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 13:59:50.984099 20318 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 13:59:50.999696 20584 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 13:59:51.014524 20718 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
W20250622 13:59:52.775516 20779 debug-util.cc:398] Leaking SignalData structure 0x7b08000b6de0 after lost signal to thread 20654
W20250622 13:59:52.776549 20779 debug-util.cc:398] Leaking SignalData structure 0x7b08000c8ba0 after lost signal to thread 20782
I20250622 13:59:52.837380 20451 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 13:59:52.854805 20718 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250622 13:59:52.863294 20318 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 13:59:52.874125 20584 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250622 13:59:53.059098 20673 consensus_peers.cc:487] T 471ace9ca3be41b8bef17fc6da185986 P 69146f2ec8b44d898791204ce591b9eb -> Peer b7523a32950c4d57b21211710ab08ccf (127.15.25.65:37005): Couldn't send request to peer b7523a32950c4d57b21211710ab08ccf. Status: Network error: Client connection negotiation failed: client connection to 127.15.25.65:37005: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250622 13:59:55.397403 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 20253
I20250622 13:59:55.421494 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 20387
I20250622 13:59:55.445945 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 20520
I20250622 13:59:55.475857 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 20653
I20250622 13:59:55.517742 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 20028
2025-06-22T13:59:55Z chronyd exiting
[ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4 (18711 ms)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest (18712 ms total)
[----------] 1 test from ListTableCliSimpleParamTest
[ RUN ] ListTableCliSimpleParamTest.TestListTables/2
I20250622 13:59:55.575615 15461 test_util.cc:276] Using random seed: -2104470232
I20250622 13:59:55.579383 15461 ts_itest-base.cc:115] Starting cluster with:
I20250622 13:59:55.579524 15461 ts_itest-base.cc:116] --------------
I20250622 13:59:55.579656 15461 ts_itest-base.cc:117] 1 tablet servers
I20250622 13:59:55.579769 15461 ts_itest-base.cc:118] 1 replicas per TS
I20250622 13:59:55.579893 15461 ts_itest-base.cc:119] --------------
2025-06-22T13:59:55Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-22T13:59:55Z Disabled control of system clock
I20250622 13:59:55.619861 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.25.126:38419
--webserver_interface=127.15.25.126
--webserver_port=0
--builtin_ntp_servers=127.15.25.84:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.25.126:38419 with env {}
W20250622 13:59:55.907415 20935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:59:55.908087 20935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 13:59:55.908778 20935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:59:55.937997 20935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250622 13:59:55.938334 20935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 13:59:55.938627 20935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250622 13:59:55.938853 20935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250622 13:59:55.971701 20935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.25.126:38419
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.25.126:38419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.25.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 13:59:55.972999 20935 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 13:59:55.974507 20935 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:59:55.987155 20942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:59:55.989070 20935 server_base.cc:1048] running on GCE node
W20250622 13:59:55.987269 20941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:59:55.989272 20944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:59:57.111590 20935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 13:59:57.114220 20935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 13:59:57.115594 20935 hybrid_clock.cc:648] HybridClock initialized: now 1750600797115539 us; error 65 us; skew 500 ppm
I20250622 13:59:57.116355 20935 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:59:57.125787 20935 webserver.cc:469] Webserver started at http://127.15.25.126:41113/ using document root <none> and password file <none>
I20250622 13:59:57.126600 20935 fs_manager.cc:362] Metadata directory not provided
I20250622 13:59:57.126799 20935 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:59:57.127205 20935 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:59:57.131353 20935 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "5e44ab0c40384556ac4f8db234fc6227"
format_stamp: "Formatted at 2025-06-22 13:59:57 on dist-test-slave-6tph"
I20250622 13:59:57.132407 20935 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "5e44ab0c40384556ac4f8db234fc6227"
format_stamp: "Formatted at 2025-06-22 13:59:57 on dist-test-slave-6tph"
I20250622 13:59:57.138831 20935 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20250622 13:59:57.143698 20951 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:59:57.144574 20935 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250622 13:59:57.144857 20935 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
uuid: "5e44ab0c40384556ac4f8db234fc6227"
format_stamp: "Formatted at 2025-06-22 13:59:57 on dist-test-slave-6tph"
I20250622 13:59:57.145131 20935 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:59:57.190352 20935 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:59:57.191723 20935 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 13:59:57.192139 20935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:59:57.256443 20935 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.126:38419
I20250622 13:59:57.256526 21002 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.126:38419 every 8 connection(s)
I20250622 13:59:57.259011 20935 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250622 13:59:57.263797 21003 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:59:57.266993 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 20935
I20250622 13:59:57.267486 15461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750600661728289-15461-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250622 13:59:57.284952 21003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e44ab0c40384556ac4f8db234fc6227: Bootstrap starting.
I20250622 13:59:57.290709 21003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e44ab0c40384556ac4f8db234fc6227: Neither blocks nor log segments found. Creating new log.
I20250622 13:59:57.292356 21003 log.cc:826] T 00000000000000000000000000000000 P 5e44ab0c40384556ac4f8db234fc6227: Log is configured to *not* fsync() on all Append() calls
I20250622 13:59:57.296257 21003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e44ab0c40384556ac4f8db234fc6227: No bootstrap required, opened a new log
I20250622 13:59:57.311571 21003 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5e44ab0c40384556ac4f8db234fc6227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e44ab0c40384556ac4f8db234fc6227" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 38419 } }
I20250622 13:59:57.312155 21003 raft_consensus.cc:383] T 00000000000000000000000000000000 P 5e44ab0c40384556ac4f8db234fc6227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:59:57.312383 21003 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5e44ab0c40384556ac4f8db234fc6227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e44ab0c40384556ac4f8db234fc6227, State: Initialized, Role: FOLLOWER
I20250622 13:59:57.313066 21003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e44ab0c40384556ac4f8db234fc6227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e44ab0c40384556ac4f8db234fc6227" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 38419 } }
I20250622 13:59:57.313602 21003 raft_consensus.cc:397] T 00000000000000000000000000000000 P 5e44ab0c40384556ac4f8db234fc6227 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 13:59:57.313885 21003 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5e44ab0c40384556ac4f8db234fc6227 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 13:59:57.314204 21003 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5e44ab0c40384556ac4f8db234fc6227 [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:59:57.317696 21003 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5e44ab0c40384556ac4f8db234fc6227 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e44ab0c40384556ac4f8db234fc6227" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 38419 } }
I20250622 13:59:57.318305 21003 leader_election.cc:304] T 00000000000000000000000000000000 P 5e44ab0c40384556ac4f8db234fc6227 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e44ab0c40384556ac4f8db234fc6227; no voters:
I20250622 13:59:57.319774 21003 leader_election.cc:290] T 00000000000000000000000000000000 P 5e44ab0c40384556ac4f8db234fc6227 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 13:59:57.320530 21008 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5e44ab0c40384556ac4f8db234fc6227 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:59:57.322721 21008 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5e44ab0c40384556ac4f8db234fc6227 [term 1 LEADER]: Becoming Leader. State: Replica: 5e44ab0c40384556ac4f8db234fc6227, State: Running, Role: LEADER
I20250622 13:59:57.323462 21003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e44ab0c40384556ac4f8db234fc6227 [sys.catalog]: configured and running, proceeding with master startup.
I20250622 13:59:57.323386 21008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e44ab0c40384556ac4f8db234fc6227 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e44ab0c40384556ac4f8db234fc6227" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 38419 } }
I20250622 13:59:57.333670 21009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e44ab0c40384556ac4f8db234fc6227 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e44ab0c40384556ac4f8db234fc6227" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e44ab0c40384556ac4f8db234fc6227" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 38419 } } }
I20250622 13:59:57.334393 21009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e44ab0c40384556ac4f8db234fc6227 [sys.catalog]: This master's current role is: LEADER
I20250622 13:59:57.334153 21010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e44ab0c40384556ac4f8db234fc6227 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e44ab0c40384556ac4f8db234fc6227. Latest consensus state: current_term: 1 leader_uuid: "5e44ab0c40384556ac4f8db234fc6227" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e44ab0c40384556ac4f8db234fc6227" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 38419 } } }
I20250622 13:59:57.334789 21010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e44ab0c40384556ac4f8db234fc6227 [sys.catalog]: This master's current role is: LEADER
I20250622 13:59:57.336871 21018 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 13:59:57.348186 21018 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 13:59:57.363114 21018 catalog_manager.cc:1349] Generated new cluster ID: b82d40fb221041d0b5bb41070fc0768b
I20250622 13:59:57.363411 21018 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 13:59:57.402297 21018 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 13:59:57.403692 21018 catalog_manager.cc:1506] Loading token signing keys...
I20250622 13:59:57.419273 21018 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 5e44ab0c40384556ac4f8db234fc6227: Generated new TSK 0
I20250622 13:59:57.420050 21018 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 13:59:57.432548 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.25.65:0
--local_ip_for_outbound_sockets=127.15.25.65
--webserver_interface=127.15.25.65
--webserver_port=0
--tserver_master_addrs=127.15.25.126:38419
--builtin_ntp_servers=127.15.25.84:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250622 13:59:57.732463 21027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 13:59:57.732972 21027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 13:59:57.733414 21027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 13:59:57.762073 21027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 13:59:57.762849 21027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.25.65
I20250622 13:59:57.794870 21027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.25.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.15.25.65
--webserver_port=0
--tserver_master_addrs=127.15.25.126:38419
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.25.65
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 13:59:57.796038 21027 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 13:59:57.797468 21027 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 13:59:57.813218 21034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:59:57.817827 21027 server_base.cc:1048] running on GCE node
W20250622 13:59:57.813957 21036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 13:59:57.813612 21033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 13:59:58.943583 21027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 13:59:58.946280 21027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 13:59:58.947741 21027 hybrid_clock.cc:648] HybridClock initialized: now 1750600798947681 us; error 62 us; skew 500 ppm
I20250622 13:59:58.948757 21027 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 13:59:58.956051 21027 webserver.cc:469] Webserver started at http://127.15.25.65:37555/ using document root <none> and password file <none>
I20250622 13:59:58.957113 21027 fs_manager.cc:362] Metadata directory not provided
I20250622 13:59:58.957343 21027 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 13:59:58.957918 21027 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 13:59:58.963428 21027 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b31ce92b592c4ff69b0da79dfcbd529b"
format_stamp: "Formatted at 2025-06-22 13:59:58 on dist-test-slave-6tph"
I20250622 13:59:58.964396 21027 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b31ce92b592c4ff69b0da79dfcbd529b"
format_stamp: "Formatted at 2025-06-22 13:59:58 on dist-test-slave-6tph"
I20250622 13:59:58.970737 21027 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20250622 13:59:58.975558 21043 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 13:59:58.976423 21027 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20250622 13:59:58.976807 21027 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b31ce92b592c4ff69b0da79dfcbd529b"
format_stamp: "Formatted at 2025-06-22 13:59:58 on dist-test-slave-6tph"
I20250622 13:59:58.977082 21027 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 13:59:59.030114 21027 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 13:59:59.031466 21027 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 13:59:59.031884 21027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 13:59:59.034235 21027 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 13:59:59.038156 21027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 13:59:59.038343 21027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 13:59:59.038604 21027 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 13:59:59.038758 21027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 13:59:59.159375 21027 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.65:44765
I20250622 13:59:59.159494 21155 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.65:44765 every 8 connection(s)
I20250622 13:59:59.161664 21027 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250622 13:59:59.166435 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 21027
I20250622 13:59:59.166916 15461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750600661728289-15461-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250622 13:59:59.180639 21156 heartbeater.cc:344] Connected to a master server at 127.15.25.126:38419
I20250622 13:59:59.181022 21156 heartbeater.cc:461] Registering TS with master...
I20250622 13:59:59.182042 21156 heartbeater.cc:507] Master 127.15.25.126:38419 requested a full tablet report, sending...
I20250622 13:59:59.184280 20968 ts_manager.cc:194] Registered new tserver with Master: b31ce92b592c4ff69b0da79dfcbd529b (127.15.25.65:44765)
I20250622 13:59:59.186012 20968 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.25.65:55455
I20250622 13:59:59.197403 15461 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250622 13:59:59.223881 20968 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54446:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250622 13:59:59.281143 21091 tablet_service.cc:1468] Processing CreateTablet for tablet ca9e2daf3f2e43ff89970d5c6098a62e (DEFAULT_TABLE table=TestTable [id=3504a42cc8044765af8de77332be1e0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 13:59:59.282848 21091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca9e2daf3f2e43ff89970d5c6098a62e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 13:59:59.306289 21171 tablet_bootstrap.cc:492] T ca9e2daf3f2e43ff89970d5c6098a62e P b31ce92b592c4ff69b0da79dfcbd529b: Bootstrap starting.
I20250622 13:59:59.311760 21171 tablet_bootstrap.cc:654] T ca9e2daf3f2e43ff89970d5c6098a62e P b31ce92b592c4ff69b0da79dfcbd529b: Neither blocks nor log segments found. Creating new log.
I20250622 13:59:59.313333 21171 log.cc:826] T ca9e2daf3f2e43ff89970d5c6098a62e P b31ce92b592c4ff69b0da79dfcbd529b: Log is configured to *not* fsync() on all Append() calls
I20250622 13:59:59.317044 21171 tablet_bootstrap.cc:492] T ca9e2daf3f2e43ff89970d5c6098a62e P b31ce92b592c4ff69b0da79dfcbd529b: No bootstrap required, opened a new log
I20250622 13:59:59.317344 21171 ts_tablet_manager.cc:1397] T ca9e2daf3f2e43ff89970d5c6098a62e P b31ce92b592c4ff69b0da79dfcbd529b: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20250622 13:59:59.333068 21171 raft_consensus.cc:357] T ca9e2daf3f2e43ff89970d5c6098a62e P b31ce92b592c4ff69b0da79dfcbd529b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b31ce92b592c4ff69b0da79dfcbd529b" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44765 } }
I20250622 13:59:59.333623 21171 raft_consensus.cc:383] T ca9e2daf3f2e43ff89970d5c6098a62e P b31ce92b592c4ff69b0da79dfcbd529b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 13:59:59.333813 21171 raft_consensus.cc:738] T ca9e2daf3f2e43ff89970d5c6098a62e P b31ce92b592c4ff69b0da79dfcbd529b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b31ce92b592c4ff69b0da79dfcbd529b, State: Initialized, Role: FOLLOWER
I20250622 13:59:59.334345 21171 consensus_queue.cc:260] T ca9e2daf3f2e43ff89970d5c6098a62e P b31ce92b592c4ff69b0da79dfcbd529b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b31ce92b592c4ff69b0da79dfcbd529b" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44765 } }
I20250622 13:59:59.334769 21171 raft_consensus.cc:397] T ca9e2daf3f2e43ff89970d5c6098a62e P b31ce92b592c4ff69b0da79dfcbd529b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 13:59:59.334959 21171 raft_consensus.cc:491] T ca9e2daf3f2e43ff89970d5c6098a62e P b31ce92b592c4ff69b0da79dfcbd529b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 13:59:59.335166 21171 raft_consensus.cc:3058] T ca9e2daf3f2e43ff89970d5c6098a62e P b31ce92b592c4ff69b0da79dfcbd529b [term 0 FOLLOWER]: Advancing to term 1
I20250622 13:59:59.338848 21171 raft_consensus.cc:513] T ca9e2daf3f2e43ff89970d5c6098a62e P b31ce92b592c4ff69b0da79dfcbd529b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b31ce92b592c4ff69b0da79dfcbd529b" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44765 } }
I20250622 13:59:59.339424 21171 leader_election.cc:304] T ca9e2daf3f2e43ff89970d5c6098a62e P b31ce92b592c4ff69b0da79dfcbd529b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b31ce92b592c4ff69b0da79dfcbd529b; no voters:
I20250622 13:59:59.340885 21171 leader_election.cc:290] T ca9e2daf3f2e43ff89970d5c6098a62e P b31ce92b592c4ff69b0da79dfcbd529b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 13:59:59.341221 21173 raft_consensus.cc:2802] T ca9e2daf3f2e43ff89970d5c6098a62e P b31ce92b592c4ff69b0da79dfcbd529b [term 1 FOLLOWER]: Leader election won for term 1
I20250622 13:59:59.344305 21156 heartbeater.cc:499] Master 127.15.25.126:38419 was elected leader, sending a full tablet report...
I20250622 13:59:59.343864 21173 raft_consensus.cc:695] T ca9e2daf3f2e43ff89970d5c6098a62e P b31ce92b592c4ff69b0da79dfcbd529b [term 1 LEADER]: Becoming Leader. State: Replica: b31ce92b592c4ff69b0da79dfcbd529b, State: Running, Role: LEADER
I20250622 13:59:59.344949 21171 ts_tablet_manager.cc:1428] T ca9e2daf3f2e43ff89970d5c6098a62e P b31ce92b592c4ff69b0da79dfcbd529b: Time spent starting tablet: real 0.027s user 0.021s sys 0.006s
I20250622 13:59:59.345225 21173 consensus_queue.cc:237] T ca9e2daf3f2e43ff89970d5c6098a62e P b31ce92b592c4ff69b0da79dfcbd529b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b31ce92b592c4ff69b0da79dfcbd529b" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44765 } }
I20250622 13:59:59.356419 20968 catalog_manager.cc:5582] T ca9e2daf3f2e43ff89970d5c6098a62e P b31ce92b592c4ff69b0da79dfcbd529b reported cstate change: term changed from 0 to 1, leader changed from <none> to b31ce92b592c4ff69b0da79dfcbd529b (127.15.25.65). New cstate: current_term: 1 leader_uuid: "b31ce92b592c4ff69b0da79dfcbd529b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b31ce92b592c4ff69b0da79dfcbd529b" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 44765 } health_report { overall_health: HEALTHY } } }
I20250622 13:59:59.376670 15461 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250622 13:59:59.379256 15461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver b31ce92b592c4ff69b0da79dfcbd529b to finish bootstrapping
I20250622 14:00:01.934053 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 21027
I20250622 14:00:01.957967 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 20935
2025-06-22T14:00:01Z chronyd exiting
[ OK ] ListTableCliSimpleParamTest.TestListTables/2 (6433 ms)
[----------] 1 test from ListTableCliSimpleParamTest (6433 ms total)
[----------] 1 test from ListTableCliParamTest
[ RUN ] ListTableCliParamTest.ListTabletWithPartitionInfo/4
I20250622 14:00:02.009264 15461 test_util.cc:276] Using random seed: -2098036582
[ OK ] ListTableCliParamTest.ListTabletWithPartitionInfo/4 (10 ms)
[----------] 1 test from ListTableCliParamTest (10 ms total)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest
[ RUN ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0
2025-06-22T14:00:02Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-22T14:00:02Z Disabled control of system clock
I20250622 14:00:02.055583 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.25.126:42519
--webserver_interface=127.15.25.126
--webserver_port=0
--builtin_ntp_servers=127.15.25.84:42217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.25.126:42519 with env {}
W20250622 14:00:02.335028 21199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 14:00:02.335547 21199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 14:00:02.335951 21199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 14:00:02.365711 21199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250622 14:00:02.366034 21199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 14:00:02.366307 21199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250622 14:00:02.366550 21199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250622 14:00:02.399766 21199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:42217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.25.126:42519
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.25.126:42519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.25.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 14:00:02.401055 21199 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 14:00:02.402607 21199 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 14:00:02.415210 21207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 14:00:02.415201 21206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 14:00:03.561240 21209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 14:00:03.563376 21208 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1145 milliseconds
I20250622 14:00:03.563469 21199 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 14:00:03.564539 21199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 14:00:03.566820 21199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 14:00:03.568128 21199 hybrid_clock.cc:648] HybridClock initialized: now 1750600803568085 us; error 48 us; skew 500 ppm
I20250622 14:00:03.568985 21199 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 14:00:03.578212 21199 webserver.cc:469] Webserver started at http://127.15.25.126:42899/ using document root <none> and password file <none>
I20250622 14:00:03.579105 21199 fs_manager.cc:362] Metadata directory not provided
I20250622 14:00:03.579301 21199 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 14:00:03.579700 21199 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 14:00:03.583561 21199 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/master-0/data/instance:
uuid: "b09c38a1f40a4b368c0086544e6e6e73"
format_stamp: "Formatted at 2025-06-22 14:00:03 on dist-test-slave-6tph"
I20250622 14:00:03.584523 21199 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/master-0/wal/instance:
uuid: "b09c38a1f40a4b368c0086544e6e6e73"
format_stamp: "Formatted at 2025-06-22 14:00:03 on dist-test-slave-6tph"
I20250622 14:00:03.591055 21199 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20250622 14:00:03.596058 21216 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 14:00:03.596992 21199 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250622 14:00:03.597287 21199 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/master-0/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/master-0/wal
uuid: "b09c38a1f40a4b368c0086544e6e6e73"
format_stamp: "Formatted at 2025-06-22 14:00:03 on dist-test-slave-6tph"
I20250622 14:00:03.597615 21199 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 14:00:03.649209 21199 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 14:00:03.650550 21199 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 14:00:03.650995 21199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 14:00:03.715837 21199 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.126:42519
I20250622 14:00:03.715906 21267 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.126:42519 every 8 connection(s)
I20250622 14:00:03.718163 21199 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/master-0/data/info.pb
I20250622 14:00:03.720507 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 21199
I20250622 14:00:03.721359 15461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/master-0/wal/instance
I20250622 14:00:03.723242 21268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 14:00:03.746084 21268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b09c38a1f40a4b368c0086544e6e6e73: Bootstrap starting.
I20250622 14:00:03.750924 21268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b09c38a1f40a4b368c0086544e6e6e73: Neither blocks nor log segments found. Creating new log.
I20250622 14:00:03.752399 21268 log.cc:826] T 00000000000000000000000000000000 P b09c38a1f40a4b368c0086544e6e6e73: Log is configured to *not* fsync() on all Append() calls
I20250622 14:00:03.756237 21268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b09c38a1f40a4b368c0086544e6e6e73: No bootstrap required, opened a new log
I20250622 14:00:03.771620 21268 raft_consensus.cc:357] T 00000000000000000000000000000000 P b09c38a1f40a4b368c0086544e6e6e73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09c38a1f40a4b368c0086544e6e6e73" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 42519 } }
I20250622 14:00:03.772249 21268 raft_consensus.cc:383] T 00000000000000000000000000000000 P b09c38a1f40a4b368c0086544e6e6e73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 14:00:03.772476 21268 raft_consensus.cc:738] T 00000000000000000000000000000000 P b09c38a1f40a4b368c0086544e6e6e73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09c38a1f40a4b368c0086544e6e6e73, State: Initialized, Role: FOLLOWER
I20250622 14:00:03.773092 21268 consensus_queue.cc:260] T 00000000000000000000000000000000 P b09c38a1f40a4b368c0086544e6e6e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09c38a1f40a4b368c0086544e6e6e73" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 42519 } }
I20250622 14:00:03.773561 21268 raft_consensus.cc:397] T 00000000000000000000000000000000 P b09c38a1f40a4b368c0086544e6e6e73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 14:00:03.773828 21268 raft_consensus.cc:491] T 00000000000000000000000000000000 P b09c38a1f40a4b368c0086544e6e6e73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 14:00:03.774086 21268 raft_consensus.cc:3058] T 00000000000000000000000000000000 P b09c38a1f40a4b368c0086544e6e6e73 [term 0 FOLLOWER]: Advancing to term 1
I20250622 14:00:03.777597 21268 raft_consensus.cc:513] T 00000000000000000000000000000000 P b09c38a1f40a4b368c0086544e6e6e73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09c38a1f40a4b368c0086544e6e6e73" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 42519 } }
I20250622 14:00:03.778337 21268 leader_election.cc:304] T 00000000000000000000000000000000 P b09c38a1f40a4b368c0086544e6e6e73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b09c38a1f40a4b368c0086544e6e6e73; no voters:
I20250622 14:00:03.780076 21268 leader_election.cc:290] T 00000000000000000000000000000000 P b09c38a1f40a4b368c0086544e6e6e73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 14:00:03.780689 21273 raft_consensus.cc:2802] T 00000000000000000000000000000000 P b09c38a1f40a4b368c0086544e6e6e73 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 14:00:03.782753 21273 raft_consensus.cc:695] T 00000000000000000000000000000000 P b09c38a1f40a4b368c0086544e6e6e73 [term 1 LEADER]: Becoming Leader. State: Replica: b09c38a1f40a4b368c0086544e6e6e73, State: Running, Role: LEADER
I20250622 14:00:03.783421 21273 consensus_queue.cc:237] T 00000000000000000000000000000000 P b09c38a1f40a4b368c0086544e6e6e73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09c38a1f40a4b368c0086544e6e6e73" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 42519 } }
I20250622 14:00:03.784328 21268 sys_catalog.cc:564] T 00000000000000000000000000000000 P b09c38a1f40a4b368c0086544e6e6e73 [sys.catalog]: configured and running, proceeding with master startup.
I20250622 14:00:03.792974 21274 sys_catalog.cc:455] T 00000000000000000000000000000000 P b09c38a1f40a4b368c0086544e6e6e73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b09c38a1f40a4b368c0086544e6e6e73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09c38a1f40a4b368c0086544e6e6e73" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 42519 } } }
I20250622 14:00:03.794752 21275 sys_catalog.cc:455] T 00000000000000000000000000000000 P b09c38a1f40a4b368c0086544e6e6e73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b09c38a1f40a4b368c0086544e6e6e73. Latest consensus state: current_term: 1 leader_uuid: "b09c38a1f40a4b368c0086544e6e6e73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09c38a1f40a4b368c0086544e6e6e73" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 42519 } } }
I20250622 14:00:03.795521 21274 sys_catalog.cc:458] T 00000000000000000000000000000000 P b09c38a1f40a4b368c0086544e6e6e73 [sys.catalog]: This master's current role is: LEADER
I20250622 14:00:03.795594 21275 sys_catalog.cc:458] T 00000000000000000000000000000000 P b09c38a1f40a4b368c0086544e6e6e73 [sys.catalog]: This master's current role is: LEADER
I20250622 14:00:03.797981 21283 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 14:00:03.808825 21283 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 14:00:03.823410 21283 catalog_manager.cc:1349] Generated new cluster ID: 8881cbddeb7b458d849d1ef90e2a5e14
I20250622 14:00:03.823699 21283 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 14:00:03.844715 21283 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 14:00:03.845947 21283 catalog_manager.cc:1506] Loading token signing keys...
I20250622 14:00:03.859865 21283 catalog_manager.cc:5955] T 00000000000000000000000000000000 P b09c38a1f40a4b368c0086544e6e6e73: Generated new TSK 0
I20250622 14:00:03.860632 21283 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 14:00:03.880501 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.25.65:0
--local_ip_for_outbound_sockets=127.15.25.65
--webserver_interface=127.15.25.65
--webserver_port=0
--tserver_master_addrs=127.15.25.126:42519
--builtin_ntp_servers=127.15.25.84:42217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20250622 14:00:04.158422 21292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 14:00:04.158939 21292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 14:00:04.159427 21292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 14:00:04.190593 21292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 14:00:04.191452 21292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.25.65
I20250622 14:00:04.226205 21292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:42217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.25.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.25.65
--webserver_port=0
--tserver_master_addrs=127.15.25.126:42519
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.25.65
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 14:00:04.227531 21292 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 14:00:04.229281 21292 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 14:00:04.245929 21299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 14:00:04.247596 21298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 14:00:04.250057 21292 server_base.cc:1048] running on GCE node
W20250622 14:00:04.251199 21301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 14:00:05.353768 21292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 14:00:05.356246 21292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 14:00:05.357677 21292 hybrid_clock.cc:648] HybridClock initialized: now 1750600805357632 us; error 44 us; skew 500 ppm
I20250622 14:00:05.358743 21292 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 14:00:05.365600 21292 webserver.cc:469] Webserver started at http://127.15.25.65:38619/ using document root <none> and password file <none>
I20250622 14:00:05.366683 21292 fs_manager.cc:362] Metadata directory not provided
I20250622 14:00:05.366915 21292 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 14:00:05.367411 21292 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 14:00:05.373698 21292 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-0/data/instance:
uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f"
format_stamp: "Formatted at 2025-06-22 14:00:05 on dist-test-slave-6tph"
I20250622 14:00:05.375072 21292 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-0/wal/instance:
uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f"
format_stamp: "Formatted at 2025-06-22 14:00:05 on dist-test-slave-6tph"
I20250622 14:00:05.383402 21292 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.000s sys 0.008s
I20250622 14:00:05.390475 21308 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 14:00:05.391657 21292 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250622 14:00:05.392047 21292 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-0/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-0/wal
uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f"
format_stamp: "Formatted at 2025-06-22 14:00:05 on dist-test-slave-6tph"
I20250622 14:00:05.392479 21292 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 14:00:05.451731 21292 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 14:00:05.453508 21292 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 14:00:05.453984 21292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 14:00:05.456668 21292 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 14:00:05.461536 21292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 14:00:05.461791 21292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20250622 14:00:05.462065 21292 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 14:00:05.462245 21292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 14:00:05.587671 21292 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.65:34225
I20250622 14:00:05.587853 21420 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.65:34225 every 8 connection(s)
I20250622 14:00:05.590317 21292 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-0/data/info.pb
I20250622 14:00:05.599073 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 21292
I20250622 14:00:05.599542 15461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-0/wal/instance
I20250622 14:00:05.606653 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.25.66:0
--local_ip_for_outbound_sockets=127.15.25.66
--webserver_interface=127.15.25.66
--webserver_port=0
--tserver_master_addrs=127.15.25.126:42519
--builtin_ntp_servers=127.15.25.84:42217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250622 14:00:05.611792 21421 heartbeater.cc:344] Connected to a master server at 127.15.25.126:42519
I20250622 14:00:05.612215 21421 heartbeater.cc:461] Registering TS with master...
I20250622 14:00:05.613184 21421 heartbeater.cc:507] Master 127.15.25.126:42519 requested a full tablet report, sending...
I20250622 14:00:05.615551 21233 ts_manager.cc:194] Registered new tserver with Master: a88bdd42d19b4fd9a56021e3ad01cd7f (127.15.25.65:34225)
I20250622 14:00:05.617337 21233 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.25.65:57591
W20250622 14:00:05.894650 21425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 14:00:05.895141 21425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 14:00:05.895643 21425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 14:00:05.924911 21425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 14:00:05.925683 21425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.25.66
I20250622 14:00:05.958804 21425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:42217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.25.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.25.66
--webserver_port=0
--tserver_master_addrs=127.15.25.126:42519
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.25.66
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 14:00:05.960084 21425 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 14:00:05.961661 21425 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 14:00:05.975787 21432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 14:00:06.621143 21421 heartbeater.cc:499] Master 127.15.25.126:42519 was elected leader, sending a full tablet report...
W20250622 14:00:05.976672 21431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 14:00:05.976693 21434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 14:00:05.978361 21425 server_base.cc:1048] running on GCE node
I20250622 14:00:07.081404 21425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 14:00:07.083381 21425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 14:00:07.084673 21425 hybrid_clock.cc:648] HybridClock initialized: now 1750600807084644 us; error 47 us; skew 500 ppm
I20250622 14:00:07.085373 21425 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 14:00:07.091013 21425 webserver.cc:469] Webserver started at http://127.15.25.66:38233/ using document root <none> and password file <none>
I20250622 14:00:07.091859 21425 fs_manager.cc:362] Metadata directory not provided
I20250622 14:00:07.092046 21425 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 14:00:07.092458 21425 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 14:00:07.096315 21425 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-1/data/instance:
uuid: "adcd2340ef804b518a235266c7499a71"
format_stamp: "Formatted at 2025-06-22 14:00:07 on dist-test-slave-6tph"
I20250622 14:00:07.097327 21425 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-1/wal/instance:
uuid: "adcd2340ef804b518a235266c7499a71"
format_stamp: "Formatted at 2025-06-22 14:00:07 on dist-test-slave-6tph"
I20250622 14:00:07.103828 21425 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250622 14:00:07.108909 21441 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 14:00:07.109851 21425 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250622 14:00:07.110131 21425 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-1/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-1/wal
uuid: "adcd2340ef804b518a235266c7499a71"
format_stamp: "Formatted at 2025-06-22 14:00:07 on dist-test-slave-6tph"
I20250622 14:00:07.110431 21425 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 14:00:07.154701 21425 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 14:00:07.155975 21425 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 14:00:07.156356 21425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 14:00:07.158609 21425 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 14:00:07.162416 21425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 14:00:07.162602 21425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 14:00:07.162823 21425 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 14:00:07.162972 21425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 14:00:07.283027 21425 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.66:45433
I20250622 14:00:07.283130 21553 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.66:45433 every 8 connection(s)
I20250622 14:00:07.285368 21425 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-1/data/info.pb
I20250622 14:00:07.287133 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 21425
I20250622 14:00:07.287657 15461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-1/wal/instance
I20250622 14:00:07.294534 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.25.67:0
--local_ip_for_outbound_sockets=127.15.25.67
--webserver_interface=127.15.25.67
--webserver_port=0
--tserver_master_addrs=127.15.25.126:42519
--builtin_ntp_servers=127.15.25.84:42217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250622 14:00:07.305200 21554 heartbeater.cc:344] Connected to a master server at 127.15.25.126:42519
I20250622 14:00:07.305755 21554 heartbeater.cc:461] Registering TS with master...
I20250622 14:00:07.307011 21554 heartbeater.cc:507] Master 127.15.25.126:42519 requested a full tablet report, sending...
I20250622 14:00:07.309537 21233 ts_manager.cc:194] Registered new tserver with Master: adcd2340ef804b518a235266c7499a71 (127.15.25.66:45433)
I20250622 14:00:07.311321 21233 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.25.66:48649
W20250622 14:00:07.575621 21558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 14:00:07.576076 21558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 14:00:07.576515 21558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 14:00:07.607990 21558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 14:00:07.608791 21558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.25.67
I20250622 14:00:07.644876 21558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:42217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.25.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.25.67
--webserver_port=0
--tserver_master_addrs=127.15.25.126:42519
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.25.67
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 14:00:07.646103 21558 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 14:00:07.647614 21558 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 14:00:07.662609 21564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 14:00:08.314554 21554 heartbeater.cc:499] Master 127.15.25.126:42519 was elected leader, sending a full tablet report...
W20250622 14:00:07.663354 21565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 14:00:07.664144 21558 server_base.cc:1048] running on GCE node
W20250622 14:00:07.663430 21567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 14:00:08.766635 21558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 14:00:08.769483 21558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 14:00:08.770911 21558 hybrid_clock.cc:648] HybridClock initialized: now 1750600808770877 us; error 48 us; skew 500 ppm
I20250622 14:00:08.771888 21558 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 14:00:08.781438 21558 webserver.cc:469] Webserver started at http://127.15.25.67:36297/ using document root <none> and password file <none>
I20250622 14:00:08.782290 21558 fs_manager.cc:362] Metadata directory not provided
I20250622 14:00:08.782454 21558 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 14:00:08.782819 21558 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 14:00:08.786832 21558 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-2/data/instance:
uuid: "9502119b0cf747b2a6e6dad2565c8f98"
format_stamp: "Formatted at 2025-06-22 14:00:08 on dist-test-slave-6tph"
I20250622 14:00:08.787830 21558 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-2/wal/instance:
uuid: "9502119b0cf747b2a6e6dad2565c8f98"
format_stamp: "Formatted at 2025-06-22 14:00:08 on dist-test-slave-6tph"
I20250622 14:00:08.794565 21558 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20250622 14:00:08.799842 21574 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 14:00:08.800916 21558 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20250622 14:00:08.801205 21558 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-2/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-2/wal
uuid: "9502119b0cf747b2a6e6dad2565c8f98"
format_stamp: "Formatted at 2025-06-22 14:00:08 on dist-test-slave-6tph"
I20250622 14:00:08.801524 21558 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 14:00:08.863315 21558 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 14:00:08.864666 21558 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 14:00:08.865093 21558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 14:00:08.867349 21558 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 14:00:08.870996 21558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 14:00:08.871165 21558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 14:00:08.871388 21558 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 14:00:08.871562 21558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 14:00:08.998740 21558 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.67:33783
I20250622 14:00:08.998845 21686 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.67:33783 every 8 connection(s)
I20250622 14:00:09.001286 21558 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-2/data/info.pb
I20250622 14:00:09.005496 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 21558
I20250622 14:00:09.006037 15461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-2/wal/instance
I20250622 14:00:09.022397 21687 heartbeater.cc:344] Connected to a master server at 127.15.25.126:42519
I20250622 14:00:09.022809 21687 heartbeater.cc:461] Registering TS with master...
I20250622 14:00:09.023840 21687 heartbeater.cc:507] Master 127.15.25.126:42519 requested a full tablet report, sending...
I20250622 14:00:09.025861 21233 ts_manager.cc:194] Registered new tserver with Master: 9502119b0cf747b2a6e6dad2565c8f98 (127.15.25.67:33783)
I20250622 14:00:09.027047 21233 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.25.67:33769
I20250622 14:00:09.040107 15461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250622 14:00:09.067099 15461 test_util.cc:276] Using random seed: -2090978746
I20250622 14:00:09.101929 21233 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44872:
name: "pre_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250622 14:00:09.104059 21233 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table pre_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250622 14:00:09.147286 21356 tablet_service.cc:1468] Processing CreateTablet for tablet 215cdfd3a26c4bc5bec0fa817a8b3e36 (DEFAULT_TABLE table=pre_rebuild [id=cf89f127849e47f189563fae377a03ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 14:00:09.148590 21489 tablet_service.cc:1468] Processing CreateTablet for tablet 215cdfd3a26c4bc5bec0fa817a8b3e36 (DEFAULT_TABLE table=pre_rebuild [id=cf89f127849e47f189563fae377a03ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 14:00:09.149830 21356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 215cdfd3a26c4bc5bec0fa817a8b3e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 14:00:09.150527 21489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 215cdfd3a26c4bc5bec0fa817a8b3e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 14:00:09.150079 21622 tablet_service.cc:1468] Processing CreateTablet for tablet 215cdfd3a26c4bc5bec0fa817a8b3e36 (DEFAULT_TABLE table=pre_rebuild [id=cf89f127849e47f189563fae377a03ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 14:00:09.151733 21622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 215cdfd3a26c4bc5bec0fa817a8b3e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 14:00:09.170291 21711 tablet_bootstrap.cc:492] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f: Bootstrap starting.
I20250622 14:00:09.176088 21712 tablet_bootstrap.cc:492] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71: Bootstrap starting.
I20250622 14:00:09.177143 21711 tablet_bootstrap.cc:654] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f: Neither blocks nor log segments found. Creating new log.
I20250622 14:00:09.178584 21713 tablet_bootstrap.cc:492] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P 9502119b0cf747b2a6e6dad2565c8f98: Bootstrap starting.
I20250622 14:00:09.179720 21711 log.cc:826] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f: Log is configured to *not* fsync() on all Append() calls
I20250622 14:00:09.183409 21712 tablet_bootstrap.cc:654] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71: Neither blocks nor log segments found. Creating new log.
I20250622 14:00:09.184238 21713 tablet_bootstrap.cc:654] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P 9502119b0cf747b2a6e6dad2565c8f98: Neither blocks nor log segments found. Creating new log.
I20250622 14:00:09.185590 21711 tablet_bootstrap.cc:492] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f: No bootstrap required, opened a new log
I20250622 14:00:09.185549 21712 log.cc:826] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71: Log is configured to *not* fsync() on all Append() calls
I20250622 14:00:09.186153 21711 ts_tablet_manager.cc:1397] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.004s
I20250622 14:00:09.186252 21713 log.cc:826] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P 9502119b0cf747b2a6e6dad2565c8f98: Log is configured to *not* fsync() on all Append() calls
I20250622 14:00:09.190901 21712 tablet_bootstrap.cc:492] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71: No bootstrap required, opened a new log
I20250622 14:00:09.191368 21713 tablet_bootstrap.cc:492] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P 9502119b0cf747b2a6e6dad2565c8f98: No bootstrap required, opened a new log
I20250622 14:00:09.191346 21712 ts_tablet_manager.cc:1397] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.004s
I20250622 14:00:09.191869 21713 ts_tablet_manager.cc:1397] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P 9502119b0cf747b2a6e6dad2565c8f98: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.003s
I20250622 14:00:09.204421 21711 raft_consensus.cc:357] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 34225 } } peers { permanent_uuid: "9502119b0cf747b2a6e6dad2565c8f98" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33783 } } peers { permanent_uuid: "adcd2340ef804b518a235266c7499a71" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45433 } }
I20250622 14:00:09.205025 21711 raft_consensus.cc:383] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 14:00:09.205263 21711 raft_consensus.cc:738] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a88bdd42d19b4fd9a56021e3ad01cd7f, State: Initialized, Role: FOLLOWER
I20250622 14:00:09.206075 21711 consensus_queue.cc:260] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 34225 } } peers { permanent_uuid: "9502119b0cf747b2a6e6dad2565c8f98" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33783 } } peers { permanent_uuid: "adcd2340ef804b518a235266c7499a71" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45433 } }
I20250622 14:00:09.209803 21711 ts_tablet_manager.cc:1428] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f: Time spent starting tablet: real 0.023s user 0.023s sys 0.000s
I20250622 14:00:09.215602 21713 raft_consensus.cc:357] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P 9502119b0cf747b2a6e6dad2565c8f98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 34225 } } peers { permanent_uuid: "9502119b0cf747b2a6e6dad2565c8f98" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33783 } } peers { permanent_uuid: "adcd2340ef804b518a235266c7499a71" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45433 } }
I20250622 14:00:09.216336 21713 raft_consensus.cc:383] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P 9502119b0cf747b2a6e6dad2565c8f98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 14:00:09.216626 21713 raft_consensus.cc:738] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P 9502119b0cf747b2a6e6dad2565c8f98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9502119b0cf747b2a6e6dad2565c8f98, State: Initialized, Role: FOLLOWER
I20250622 14:00:09.218236 21712 raft_consensus.cc:357] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 34225 } } peers { permanent_uuid: "9502119b0cf747b2a6e6dad2565c8f98" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33783 } } peers { permanent_uuid: "adcd2340ef804b518a235266c7499a71" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45433 } }
I20250622 14:00:09.217597 21713 consensus_queue.cc:260] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P 9502119b0cf747b2a6e6dad2565c8f98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 34225 } } peers { permanent_uuid: "9502119b0cf747b2a6e6dad2565c8f98" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33783 } } peers { permanent_uuid: "adcd2340ef804b518a235266c7499a71" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45433 } }
I20250622 14:00:09.219023 21712 raft_consensus.cc:383] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 14:00:09.219295 21712 raft_consensus.cc:738] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adcd2340ef804b518a235266c7499a71, State: Initialized, Role: FOLLOWER
I20250622 14:00:09.220315 21712 consensus_queue.cc:260] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 34225 } } peers { permanent_uuid: "9502119b0cf747b2a6e6dad2565c8f98" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33783 } } peers { permanent_uuid: "adcd2340ef804b518a235266c7499a71" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45433 } }
I20250622 14:00:09.223101 21687 heartbeater.cc:499] Master 127.15.25.126:42519 was elected leader, sending a full tablet report...
I20250622 14:00:09.224301 21713 ts_tablet_manager.cc:1428] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P 9502119b0cf747b2a6e6dad2565c8f98: Time spent starting tablet: real 0.032s user 0.026s sys 0.004s
I20250622 14:00:09.225700 21712 ts_tablet_manager.cc:1428] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71: Time spent starting tablet: real 0.034s user 0.029s sys 0.004s
W20250622 14:00:09.257133 21688 tablet.cc:2378] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P 9502119b0cf747b2a6e6dad2565c8f98: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 14:00:09.270954 21719 raft_consensus.cc:491] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 14:00:09.271384 21719 raft_consensus.cc:513] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 34225 } } peers { permanent_uuid: "9502119b0cf747b2a6e6dad2565c8f98" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33783 } } peers { permanent_uuid: "adcd2340ef804b518a235266c7499a71" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45433 } }
I20250622 14:00:09.273643 21719 leader_election.cc:290] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a88bdd42d19b4fd9a56021e3ad01cd7f (127.15.25.65:34225), 9502119b0cf747b2a6e6dad2565c8f98 (127.15.25.67:33783)
I20250622 14:00:09.284129 21642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "215cdfd3a26c4bc5bec0fa817a8b3e36" candidate_uuid: "adcd2340ef804b518a235266c7499a71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9502119b0cf747b2a6e6dad2565c8f98" is_pre_election: true
I20250622 14:00:09.284860 21642 raft_consensus.cc:2466] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P 9502119b0cf747b2a6e6dad2565c8f98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adcd2340ef804b518a235266c7499a71 in term 0.
I20250622 14:00:09.284904 21376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "215cdfd3a26c4bc5bec0fa817a8b3e36" candidate_uuid: "adcd2340ef804b518a235266c7499a71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" is_pre_election: true
I20250622 14:00:09.285620 21376 raft_consensus.cc:2466] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adcd2340ef804b518a235266c7499a71 in term 0.
I20250622 14:00:09.286129 21445 leader_election.cc:304] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9502119b0cf747b2a6e6dad2565c8f98, adcd2340ef804b518a235266c7499a71; no voters:
I20250622 14:00:09.286845 21719 raft_consensus.cc:2802] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 14:00:09.287143 21719 raft_consensus.cc:491] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 14:00:09.287413 21719 raft_consensus.cc:3058] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71 [term 0 FOLLOWER]: Advancing to term 1
W20250622 14:00:09.291742 21555 tablet.cc:2378] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 14:00:09.293862 21719 raft_consensus.cc:513] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 34225 } } peers { permanent_uuid: "9502119b0cf747b2a6e6dad2565c8f98" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33783 } } peers { permanent_uuid: "adcd2340ef804b518a235266c7499a71" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45433 } }
I20250622 14:00:09.295506 21719 leader_election.cc:290] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71 [CANDIDATE]: Term 1 election: Requested vote from peers a88bdd42d19b4fd9a56021e3ad01cd7f (127.15.25.65:34225), 9502119b0cf747b2a6e6dad2565c8f98 (127.15.25.67:33783)
I20250622 14:00:09.296208 21376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "215cdfd3a26c4bc5bec0fa817a8b3e36" candidate_uuid: "adcd2340ef804b518a235266c7499a71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f"
I20250622 14:00:09.296595 21376 raft_consensus.cc:3058] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f [term 0 FOLLOWER]: Advancing to term 1
I20250622 14:00:09.296455 21642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "215cdfd3a26c4bc5bec0fa817a8b3e36" candidate_uuid: "adcd2340ef804b518a235266c7499a71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9502119b0cf747b2a6e6dad2565c8f98"
I20250622 14:00:09.296938 21642 raft_consensus.cc:3058] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P 9502119b0cf747b2a6e6dad2565c8f98 [term 0 FOLLOWER]: Advancing to term 1
I20250622 14:00:09.301003 21376 raft_consensus.cc:2466] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adcd2340ef804b518a235266c7499a71 in term 1.
I20250622 14:00:09.301893 21444 leader_election.cc:304] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a88bdd42d19b4fd9a56021e3ad01cd7f, adcd2340ef804b518a235266c7499a71; no voters:
I20250622 14:00:09.302439 21719 raft_consensus.cc:2802] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 14:00:09.303249 21642 raft_consensus.cc:2466] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P 9502119b0cf747b2a6e6dad2565c8f98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adcd2340ef804b518a235266c7499a71 in term 1.
I20250622 14:00:09.304266 21719 raft_consensus.cc:695] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71 [term 1 LEADER]: Becoming Leader. State: Replica: adcd2340ef804b518a235266c7499a71, State: Running, Role: LEADER
I20250622 14:00:09.304982 21719 consensus_queue.cc:237] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 34225 } } peers { permanent_uuid: "9502119b0cf747b2a6e6dad2565c8f98" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33783 } } peers { permanent_uuid: "adcd2340ef804b518a235266c7499a71" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45433 } }
I20250622 14:00:09.317943 21232 catalog_manager.cc:5582] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71 reported cstate change: term changed from 0 to 1, leader changed from <none> to adcd2340ef804b518a235266c7499a71 (127.15.25.66). New cstate: current_term: 1 leader_uuid: "adcd2340ef804b518a235266c7499a71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 34225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9502119b0cf747b2a6e6dad2565c8f98" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "adcd2340ef804b518a235266c7499a71" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45433 } health_report { overall_health: HEALTHY } } }
W20250622 14:00:09.348270 21422 tablet.cc:2378] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 14:00:09.457333 21642 raft_consensus.cc:1273] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P 9502119b0cf747b2a6e6dad2565c8f98 [term 1 FOLLOWER]: Refusing update from remote peer adcd2340ef804b518a235266c7499a71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 14:00:09.457590 21376 raft_consensus.cc:1273] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f [term 1 FOLLOWER]: Refusing update from remote peer adcd2340ef804b518a235266c7499a71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 14:00:09.459064 21722 consensus_queue.cc:1035] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9502119b0cf747b2a6e6dad2565c8f98" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250622 14:00:09.459765 21719 consensus_queue.cc:1035] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 34225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 14:00:09.487468 21729 mvcc.cc:204] Tried to move back new op lower bound from 7170460915523039232 to 7170460914920628224. Current Snapshot: MvccSnapshot[applied={T|T < 7170460915523039232}]
I20250622 14:00:14.226361 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 21199
W20250622 14:00:14.567577 21757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 14:00:14.568145 21757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 14:00:14.604074 21554 heartbeater.cc:646] Failed to heartbeat to 127.15.25.126:42519 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.25.126:42519: connect: Connection refused (error 111)
W20250622 14:00:14.604549 21421 heartbeater.cc:646] Failed to heartbeat to 127.15.25.126:42519 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.25.126:42519: connect: Connection refused (error 111)
W20250622 14:00:14.624600 21757 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250622 14:00:14.651736 21687 heartbeater.cc:646] Failed to heartbeat to 127.15.25.126:42519 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.25.126:42519: connect: Connection refused (error 111)
W20250622 14:00:16.048146 21757 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.372s user 0.438s sys 0.921s
W20250622 14:00:16.048480 21757 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.373s user 0.438s sys 0.921s
I20250622 14:00:16.135354 21757 minidump.cc:252] Setting minidump size limit to 20M
I20250622 14:00:16.137856 21757 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 14:00:16.138855 21757 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 14:00:16.155164 21792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 14:00:16.161556 21793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 14:00:16.163627 21795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 14:00:16.269529 21757 server_base.cc:1048] running on GCE node
I20250622 14:00:16.270605 21757 hybrid_clock.cc:584] initializing the hybrid clock with 'system' time source
I20250622 14:00:16.271092 21757 hybrid_clock.cc:648] HybridClock initialized: now 1750600816271069 us; error 273016 us; skew 500 ppm
I20250622 14:00:16.271792 21757 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 14:00:16.276260 21757 webserver.cc:469] Webserver started at http://0.0.0.0:35261/ using document root <none> and password file <none>
I20250622 14:00:16.277076 21757 fs_manager.cc:362] Metadata directory not provided
I20250622 14:00:16.277298 21757 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 14:00:16.277818 21757 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 14:00:16.281921 21757 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/master-0/data/instance:
uuid: "2d9465de7cc54dfc9f15ded620288a37"
format_stamp: "Formatted at 2025-06-22 14:00:16 on dist-test-slave-6tph"
I20250622 14:00:16.282943 21757 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/master-0/wal/instance:
uuid: "2d9465de7cc54dfc9f15ded620288a37"
format_stamp: "Formatted at 2025-06-22 14:00:16 on dist-test-slave-6tph"
I20250622 14:00:16.288734 21757 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20250622 14:00:16.293769 21800 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 14:00:16.294615 21757 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250622 14:00:16.294919 21757 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/master-0/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/master-0/wal
uuid: "2d9465de7cc54dfc9f15ded620288a37"
format_stamp: "Formatted at 2025-06-22 14:00:16 on dist-test-slave-6tph"
I20250622 14:00:16.295209 21757 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 14:00:16.359073 21757 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 14:00:16.360486 21757 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 14:00:16.360898 21757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 14:00:16.365047 21757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 14:00:16.378201 21757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37: Bootstrap starting.
I20250622 14:00:16.382725 21757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37: Neither blocks nor log segments found. Creating new log.
I20250622 14:00:16.384157 21757 log.cc:826] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37: Log is configured to *not* fsync() on all Append() calls
I20250622 14:00:16.387672 21757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37: No bootstrap required, opened a new log
I20250622 14:00:16.402397 21757 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d9465de7cc54dfc9f15ded620288a37" member_type: VOTER }
I20250622 14:00:16.402849 21757 raft_consensus.cc:383] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 14:00:16.403059 21757 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d9465de7cc54dfc9f15ded620288a37, State: Initialized, Role: FOLLOWER
I20250622 14:00:16.403647 21757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d9465de7cc54dfc9f15ded620288a37" member_type: VOTER }
I20250622 14:00:16.404089 21757 raft_consensus.cc:397] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 14:00:16.404313 21757 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 14:00:16.404559 21757 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37 [term 0 FOLLOWER]: Advancing to term 1
I20250622 14:00:16.408035 21757 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d9465de7cc54dfc9f15ded620288a37" member_type: VOTER }
I20250622 14:00:16.408618 21757 leader_election.cc:304] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d9465de7cc54dfc9f15ded620288a37; no voters:
I20250622 14:00:16.410282 21757 leader_election.cc:290] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 14:00:16.410492 21807 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 14:00:16.413161 21807 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37 [term 1 LEADER]: Becoming Leader. State: Replica: 2d9465de7cc54dfc9f15ded620288a37, State: Running, Role: LEADER
I20250622 14:00:16.413916 21807 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d9465de7cc54dfc9f15ded620288a37" member_type: VOTER }
I20250622 14:00:16.419798 21809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d9465de7cc54dfc9f15ded620288a37. Latest consensus state: current_term: 1 leader_uuid: "2d9465de7cc54dfc9f15ded620288a37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d9465de7cc54dfc9f15ded620288a37" member_type: VOTER } }
I20250622 14:00:16.420521 21809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37 [sys.catalog]: This master's current role is: LEADER
I20250622 14:00:16.421255 21808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d9465de7cc54dfc9f15ded620288a37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d9465de7cc54dfc9f15ded620288a37" member_type: VOTER } }
I20250622 14:00:16.421805 21808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37 [sys.catalog]: This master's current role is: LEADER
I20250622 14:00:16.431452 21757 tablet_replica.cc:331] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37: stopping tablet replica
I20250622 14:00:16.432020 21757 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37 [term 1 LEADER]: Raft consensus shutting down.
I20250622 14:00:16.432474 21757 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 14:00:16.434441 21757 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250622 14:00:16.434880 21757 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250622 14:00:16.461969 21757 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
I20250622 14:00:17.490518 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 21292
I20250622 14:00:17.521449 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 21425
I20250622 14:00:17.558677 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 21558
I20250622 14:00:17.594547 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.25.126:42519
--webserver_interface=127.15.25.126
--webserver_port=42899
--builtin_ntp_servers=127.15.25.84:42217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.25.126:42519 with env {}
W20250622 14:00:17.882164 21816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 14:00:17.882758 21816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 14:00:17.883209 21816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 14:00:17.912811 21816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250622 14:00:17.913123 21816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 14:00:17.913353 21816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250622 14:00:17.913581 21816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250622 14:00:17.946332 21816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:42217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.25.126:42519
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.25.126:42519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.25.126
--webserver_port=42899
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 14:00:17.947573 21816 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 14:00:17.949071 21816 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 14:00:17.962558 21823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 14:00:17.962726 21822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 14:00:17.963413 21825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 14:00:19.066337 21824 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250622 14:00:19.066481 21816 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 14:00:19.070168 21816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 14:00:19.073076 21816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 14:00:19.074443 21816 hybrid_clock.cc:648] HybridClock initialized: now 1750600819074423 us; error 41 us; skew 500 ppm
I20250622 14:00:19.075132 21816 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 14:00:19.081458 21816 webserver.cc:469] Webserver started at http://127.15.25.126:42899/ using document root <none> and password file <none>
I20250622 14:00:19.082327 21816 fs_manager.cc:362] Metadata directory not provided
I20250622 14:00:19.082515 21816 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 14:00:19.089617 21816 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.007s sys 0.000s
I20250622 14:00:19.095222 21832 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 14:00:19.096201 21816 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250622 14:00:19.096482 21816 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/master-0/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/master-0/wal
uuid: "2d9465de7cc54dfc9f15ded620288a37"
format_stamp: "Formatted at 2025-06-22 14:00:16 on dist-test-slave-6tph"
I20250622 14:00:19.098641 21816 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 14:00:19.145984 21816 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 14:00:19.147279 21816 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 14:00:19.147644 21816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 14:00:19.212724 21816 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.126:42519
I20250622 14:00:19.212779 21883 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.126:42519 every 8 connection(s)
I20250622 14:00:19.215418 21816 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/master-0/data/info.pb
I20250622 14:00:19.222220 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 21816
I20250622 14:00:19.224149 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.25.65:34225
--local_ip_for_outbound_sockets=127.15.25.65
--tserver_master_addrs=127.15.25.126:42519
--webserver_port=38619
--webserver_interface=127.15.25.65
--builtin_ntp_servers=127.15.25.84:42217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250622 14:00:19.226600 21884 sys_catalog.cc:263] Verifying existing consensus state
I20250622 14:00:19.239677 21884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37: Bootstrap starting.
I20250622 14:00:19.248993 21884 log.cc:826] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37: Log is configured to *not* fsync() on all Append() calls
I20250622 14:00:19.260547 21884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 14:00:19.261276 21884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37: Bootstrap complete.
I20250622 14:00:19.280354 21884 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d9465de7cc54dfc9f15ded620288a37" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 42519 } }
I20250622 14:00:19.280966 21884 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d9465de7cc54dfc9f15ded620288a37, State: Initialized, Role: FOLLOWER
I20250622 14:00:19.281626 21884 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d9465de7cc54dfc9f15ded620288a37" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 42519 } }
I20250622 14:00:19.282131 21884 raft_consensus.cc:397] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 14:00:19.282369 21884 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 14:00:19.282630 21884 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37 [term 1 FOLLOWER]: Advancing to term 2
I20250622 14:00:19.286208 21884 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d9465de7cc54dfc9f15ded620288a37" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 42519 } }
I20250622 14:00:19.286770 21884 leader_election.cc:304] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d9465de7cc54dfc9f15ded620288a37; no voters:
I20250622 14:00:19.288759 21884 leader_election.cc:290] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250622 14:00:19.289077 21888 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37 [term 2 FOLLOWER]: Leader election won for term 2
I20250622 14:00:19.292407 21888 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37 [term 2 LEADER]: Becoming Leader. State: Replica: 2d9465de7cc54dfc9f15ded620288a37, State: Running, Role: LEADER
I20250622 14:00:19.293182 21888 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d9465de7cc54dfc9f15ded620288a37" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 42519 } }
I20250622 14:00:19.293901 21884 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37 [sys.catalog]: configured and running, proceeding with master startup.
I20250622 14:00:19.303637 21889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2d9465de7cc54dfc9f15ded620288a37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d9465de7cc54dfc9f15ded620288a37" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 42519 } } }
I20250622 14:00:19.304328 21889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37 [sys.catalog]: This master's current role is: LEADER
I20250622 14:00:19.305997 21890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d9465de7cc54dfc9f15ded620288a37. Latest consensus state: current_term: 2 leader_uuid: "2d9465de7cc54dfc9f15ded620288a37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d9465de7cc54dfc9f15ded620288a37" member_type: VOTER last_known_addr { host: "127.15.25.126" port: 42519 } } }
I20250622 14:00:19.306813 21890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37 [sys.catalog]: This master's current role is: LEADER
I20250622 14:00:19.319322 21895 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 14:00:19.331501 21895 catalog_manager.cc:671] Loaded metadata for table pre_rebuild [id=0aa5b3d575144d50b7f3f1c6898e9861]
I20250622 14:00:19.339313 21895 tablet_loader.cc:96] loaded metadata for tablet 215cdfd3a26c4bc5bec0fa817a8b3e36 (table pre_rebuild [id=0aa5b3d575144d50b7f3f1c6898e9861])
I20250622 14:00:19.341091 21895 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 14:00:19.371232 21895 catalog_manager.cc:1349] Generated new cluster ID: eb58830800e04254b4123cb1c68f3bbe
I20250622 14:00:19.371589 21895 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 14:00:19.378984 21907 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250622 14:00:19.410027 21895 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 14:00:19.411973 21895 catalog_manager.cc:1506] Loading token signing keys...
I20250622 14:00:19.428491 21895 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37: Generated new TSK 0
I20250622 14:00:19.429519 21895 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250622 14:00:19.549460 21886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 14:00:19.549993 21886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 14:00:19.550483 21886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 14:00:19.580413 21886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 14:00:19.581275 21886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.25.65
I20250622 14:00:19.615089 21886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:42217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.25.65:34225
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.25.65
--webserver_port=38619
--tserver_master_addrs=127.15.25.126:42519
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.25.65
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 14:00:19.616410 21886 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 14:00:19.618016 21886 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 14:00:19.635258 21914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 14:00:19.636437 21913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 14:00:19.639039 21916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 14:00:19.639014 21886 server_base.cc:1048] running on GCE node
I20250622 14:00:20.755692 21886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 14:00:20.758153 21886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 14:00:20.759559 21886 hybrid_clock.cc:648] HybridClock initialized: now 1750600820759488 us; error 67 us; skew 500 ppm
I20250622 14:00:20.760623 21886 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 14:00:20.771888 21886 webserver.cc:469] Webserver started at http://127.15.25.65:38619/ using document root <none> and password file <none>
I20250622 14:00:20.773151 21886 fs_manager.cc:362] Metadata directory not provided
I20250622 14:00:20.773358 21886 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 14:00:20.781235 21886 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20250622 14:00:20.785851 21923 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 14:00:20.786974 21886 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20250622 14:00:20.787257 21886 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-0/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-0/wal
uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f"
format_stamp: "Formatted at 2025-06-22 14:00:05 on dist-test-slave-6tph"
I20250622 14:00:20.788991 21886 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 14:00:20.832381 21886 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 14:00:20.833844 21886 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 14:00:20.834240 21886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 14:00:20.836978 21886 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 14:00:20.842877 21930 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250622 14:00:20.849622 21886 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250622 14:00:20.849866 21886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.002s
I20250622 14:00:20.850175 21886 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250622 14:00:20.854653 21886 ts_tablet_manager.cc:610] Registered 1 tablets
I20250622 14:00:20.854875 21886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.004s sys 0.000s
I20250622 14:00:20.855223 21930 tablet_bootstrap.cc:492] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f: Bootstrap starting.
I20250622 14:00:21.022780 21886 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.65:34225
I20250622 14:00:21.023015 22036 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.65:34225 every 8 connection(s)
I20250622 14:00:21.026484 21886 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-0/data/info.pb
I20250622 14:00:21.027962 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 21886
I20250622 14:00:21.029990 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.25.66:45433
--local_ip_for_outbound_sockets=127.15.25.66
--tserver_master_addrs=127.15.25.126:42519
--webserver_port=38233
--webserver_interface=127.15.25.66
--builtin_ntp_servers=127.15.25.84:42217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250622 14:00:21.094576 22037 heartbeater.cc:344] Connected to a master server at 127.15.25.126:42519
I20250622 14:00:21.095052 22037 heartbeater.cc:461] Registering TS with master...
I20250622 14:00:21.096128 22037 heartbeater.cc:507] Master 127.15.25.126:42519 requested a full tablet report, sending...
I20250622 14:00:21.100346 21849 ts_manager.cc:194] Registered new tserver with Master: a88bdd42d19b4fd9a56021e3ad01cd7f (127.15.25.65:34225)
I20250622 14:00:21.106330 21849 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.25.65:56861
I20250622 14:00:21.161581 21930 log.cc:826] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f: Log is configured to *not* fsync() on all Append() calls
W20250622 14:00:21.375800 22041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 14:00:21.376271 22041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 14:00:21.376737 22041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 14:00:21.406455 22041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 14:00:21.407311 22041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.25.66
I20250622 14:00:21.440397 22041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:42217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.25.66:45433
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.25.66
--webserver_port=38233
--tserver_master_addrs=127.15.25.126:42519
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.25.66
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 14:00:21.441732 22041 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 14:00:21.443328 22041 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 14:00:21.459806 22049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 14:00:22.110102 22037 heartbeater.cc:499] Master 127.15.25.126:42519 was elected leader, sending a full tablet report...
W20250622 14:00:21.461491 22048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 14:00:22.834152 22051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 14:00:22.835831 22050 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1377 milliseconds
W20250622 14:00:22.837448 22041 thread.cc:640] OpenStack (cloud detector) Time spent creating pthread: real 1.382s user 0.461s sys 0.875s
W20250622 14:00:22.837798 22041 thread.cc:606] OpenStack (cloud detector) Time spent starting thread: real 1.382s user 0.461s sys 0.877s
I20250622 14:00:22.838068 22041 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 14:00:22.839457 22041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 14:00:22.842271 22041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 14:00:22.845835 22041 hybrid_clock.cc:648] HybridClock initialized: now 1750600822845776 us; error 56 us; skew 500 ppm
I20250622 14:00:22.846933 22041 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 14:00:22.855839 22041 webserver.cc:469] Webserver started at http://127.15.25.66:38233/ using document root <none> and password file <none>
I20250622 14:00:22.856976 22041 fs_manager.cc:362] Metadata directory not provided
I20250622 14:00:22.857236 22041 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 14:00:22.867570 22041 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.006s sys 0.003s
I20250622 14:00:22.873572 22059 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 14:00:22.874847 22041 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20250622 14:00:22.875195 22041 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-1/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-1/wal
uuid: "adcd2340ef804b518a235266c7499a71"
format_stamp: "Formatted at 2025-06-22 14:00:07 on dist-test-slave-6tph"
I20250622 14:00:22.877691 22041 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 14:00:22.966696 22041 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 14:00:22.968143 22041 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 14:00:22.968576 22041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 14:00:22.971537 22041 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 14:00:22.978078 22066 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250622 14:00:22.985548 22041 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250622 14:00:22.985863 22041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.003s sys 0.000s
I20250622 14:00:22.986202 22041 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250622 14:00:22.993338 22041 ts_tablet_manager.cc:610] Registered 1 tablets
I20250622 14:00:22.993602 22041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.007s user 0.004s sys 0.000s
I20250622 14:00:22.994041 22066 tablet_bootstrap.cc:492] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71: Bootstrap starting.
I20250622 14:00:23.202236 22041 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.66:45433
I20250622 14:00:23.202386 22172 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.66:45433 every 8 connection(s)
I20250622 14:00:23.205628 22041 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-1/data/info.pb
I20250622 14:00:23.208442 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 22041
I20250622 14:00:23.210582 15461 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
/tmp/dist-test-task1wNjCM/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.25.67:33783
--local_ip_for_outbound_sockets=127.15.25.67
--tserver_master_addrs=127.15.25.126:42519
--webserver_port=36297
--webserver_interface=127.15.25.67
--builtin_ntp_servers=127.15.25.84:42217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250622 14:00:23.271410 22173 heartbeater.cc:344] Connected to a master server at 127.15.25.126:42519
I20250622 14:00:23.272058 22173 heartbeater.cc:461] Registering TS with master...
I20250622 14:00:23.273191 22173 heartbeater.cc:507] Master 127.15.25.126:42519 requested a full tablet report, sending...
I20250622 14:00:23.276966 21849 ts_manager.cc:194] Registered new tserver with Master: adcd2340ef804b518a235266c7499a71 (127.15.25.66:45433)
I20250622 14:00:23.279650 21849 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.25.66:57649
I20250622 14:00:23.294291 22066 log.cc:826] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71: Log is configured to *not* fsync() on all Append() calls
W20250622 14:00:23.663460 22177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 14:00:23.664110 22177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 14:00:23.664798 22177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 14:00:23.717573 22177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 14:00:23.718900 22177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.25.67
I20250622 14:00:23.773262 22177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.25.84:42217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.25.67:33783
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.25.67
--webserver_port=36297
--tserver_master_addrs=127.15.25.126:42519
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.25.67
--log_dir=/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 6717
TSAN enabled
I20250622 14:00:23.774766 22177 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 14:00:23.776639 22177 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 14:00:23.795003 22184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 14:00:23.986960 21930 tablet_bootstrap.cc:492] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 14:00:23.988088 21930 tablet_bootstrap.cc:492] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f: Bootstrap complete.
I20250622 14:00:23.989984 21930 ts_tablet_manager.cc:1397] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f: Time spent bootstrapping tablet: real 3.135s user 3.022s sys 0.052s
I20250622 14:00:24.010859 21930 raft_consensus.cc:357] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 34225 } } peers { permanent_uuid: "9502119b0cf747b2a6e6dad2565c8f98" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33783 } } peers { permanent_uuid: "adcd2340ef804b518a235266c7499a71" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45433 } }
I20250622 14:00:24.015146 21930 raft_consensus.cc:738] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a88bdd42d19b4fd9a56021e3ad01cd7f, State: Initialized, Role: FOLLOWER
I20250622 14:00:24.016412 21930 consensus_queue.cc:260] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 34225 } } peers { permanent_uuid: "9502119b0cf747b2a6e6dad2565c8f98" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33783 } } peers { permanent_uuid: "adcd2340ef804b518a235266c7499a71" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45433 } }
I20250622 14:00:24.046231 21930 ts_tablet_manager.cc:1428] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f: Time spent starting tablet: real 0.056s user 0.027s sys 0.008s
I20250622 14:00:24.283388 22173 heartbeater.cc:499] Master 127.15.25.126:42519 was elected leader, sending a full tablet report...
I20250622 14:00:25.658783 22191 raft_consensus.cc:491] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 14:00:25.670133 22191 raft_consensus.cc:513] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 34225 } } peers { permanent_uuid: "9502119b0cf747b2a6e6dad2565c8f98" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33783 } } peers { permanent_uuid: "adcd2340ef804b518a235266c7499a71" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45433 } }
I20250622 14:00:25.680202 22191 leader_election.cc:290] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9502119b0cf747b2a6e6dad2565c8f98 (127.15.25.67:33783), adcd2340ef804b518a235266c7499a71 (127.15.25.66:45433)
W20250622 14:00:25.697193 21927 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.25.67:33783: connect: Connection refused (error 111)
W20250622 14:00:25.712031 21927 leader_election.cc:336] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9502119b0cf747b2a6e6dad2565c8f98 (127.15.25.67:33783): Network error: Client connection negotiation failed: client connection to 127.15.25.67:33783: connect: Connection refused (error 111)
I20250622 14:00:25.706192 22128 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "215cdfd3a26c4bc5bec0fa817a8b3e36" candidate_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "adcd2340ef804b518a235266c7499a71" is_pre_election: true
W20250622 14:00:25.718340 21925 leader_election.cc:343] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer adcd2340ef804b518a235266c7499a71 (127.15.25.66:45433): Illegal state: must be running to vote when last-logged opid is not known
I20250622 14:00:25.718760 21925 leader_election.cc:304] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a88bdd42d19b4fd9a56021e3ad01cd7f; no voters: 9502119b0cf747b2a6e6dad2565c8f98, adcd2340ef804b518a235266c7499a71
I20250622 14:00:25.719528 22191 raft_consensus.cc:2747] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250622 14:00:25.197137 22183 debug-util.cc:398] Leaking SignalData structure 0x7b0800000a80 after lost signal to thread 22177
W20250622 14:00:25.805006 22183 kernel_stack_watchdog.cc:198] Thread 22177 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:641 for 393ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250622 14:00:25.805502 22177 thread.cc:640] OpenStack (cloud detector) Time spent creating pthread: real 2.009s user 0.701s sys 1.200s
W20250622 14:00:25.806043 22177 thread.cc:606] OpenStack (cloud detector) Time spent starting thread: real 2.009s user 0.701s sys 1.200s
W20250622 14:00:23.796594 22185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 14:00:25.807511 22187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 14:00:25.811965 22186 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 2013 milliseconds
I20250622 14:00:25.811996 22177 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 14:00:25.813226 22177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 14:00:25.815645 22177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 14:00:25.817081 22177 hybrid_clock.cc:648] HybridClock initialized: now 1750600825817037 us; error 53 us; skew 500 ppm
I20250622 14:00:25.817893 22177 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 14:00:25.827867 22177 webserver.cc:469] Webserver started at http://127.15.25.67:36297/ using document root <none> and password file <none>
I20250622 14:00:25.828842 22177 fs_manager.cc:362] Metadata directory not provided
I20250622 14:00:25.829051 22177 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 14:00:25.836342 22177 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.001s
I20250622 14:00:25.841260 22200 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 14:00:25.842211 22177 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250622 14:00:25.842481 22177 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-2/data,/tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-2/wal
uuid: "9502119b0cf747b2a6e6dad2565c8f98"
format_stamp: "Formatted at 2025-06-22 14:00:08 on dist-test-slave-6tph"
I20250622 14:00:25.844094 22177 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 14:00:25.891317 22177 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 14:00:25.892640 22177 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 14:00:25.893030 22177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 14:00:25.895893 22177 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 14:00:25.901657 22207 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250622 14:00:25.908401 22177 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250622 14:00:25.908572 22177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.002s sys 0.000s
I20250622 14:00:25.908841 22177 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250622 14:00:25.913141 22177 ts_tablet_manager.cc:610] Registered 1 tablets
I20250622 14:00:25.913388 22177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.000s sys 0.001s
I20250622 14:00:25.913779 22207 tablet_bootstrap.cc:492] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P 9502119b0cf747b2a6e6dad2565c8f98: Bootstrap starting.
I20250622 14:00:26.148640 22177 rpc_server.cc:307] RPC server started. Bound to: 127.15.25.67:33783
I20250622 14:00:26.148743 22313 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.25.67:33783 every 8 connection(s)
I20250622 14:00:26.151451 22177 server_base.cc:1180] Dumped server information to /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-2/data/info.pb
I20250622 14:00:26.158671 15461 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu as pid 22177
I20250622 14:00:26.191893 22314 heartbeater.cc:344] Connected to a master server at 127.15.25.126:42519
I20250622 14:00:26.192412 22314 heartbeater.cc:461] Registering TS with master...
I20250622 14:00:26.193650 22314 heartbeater.cc:507] Master 127.15.25.126:42519 requested a full tablet report, sending...
I20250622 14:00:26.196974 21848 ts_manager.cc:194] Registered new tserver with Master: 9502119b0cf747b2a6e6dad2565c8f98 (127.15.25.67:33783)
I20250622 14:00:26.199786 21848 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.25.67:40965
I20250622 14:00:26.209015 15461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250622 14:00:26.211809 22207 log.cc:826] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P 9502119b0cf747b2a6e6dad2565c8f98: Log is configured to *not* fsync() on all Append() calls
I20250622 14:00:26.586033 22066 tablet_bootstrap.cc:492] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 14:00:26.586797 22066 tablet_bootstrap.cc:492] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71: Bootstrap complete.
I20250622 14:00:26.588033 22066 ts_tablet_manager.cc:1397] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71: Time spent bootstrapping tablet: real 3.594s user 3.351s sys 0.067s
I20250622 14:00:26.593255 22066 raft_consensus.cc:357] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 34225 } } peers { permanent_uuid: "9502119b0cf747b2a6e6dad2565c8f98" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33783 } } peers { permanent_uuid: "adcd2340ef804b518a235266c7499a71" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45433 } }
I20250622 14:00:26.595075 22066 raft_consensus.cc:738] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: adcd2340ef804b518a235266c7499a71, State: Initialized, Role: FOLLOWER
I20250622 14:00:26.595671 22066 consensus_queue.cc:260] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 34225 } } peers { permanent_uuid: "9502119b0cf747b2a6e6dad2565c8f98" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33783 } } peers { permanent_uuid: "adcd2340ef804b518a235266c7499a71" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45433 } }
I20250622 14:00:26.598734 22066 ts_tablet_manager.cc:1428] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20250622 14:00:27.203965 22314 heartbeater.cc:499] Master 127.15.25.126:42519 was elected leader, sending a full tablet report...
I20250622 14:00:27.447965 22327 raft_consensus.cc:491] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 14:00:27.448359 22327 raft_consensus.cc:513] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 34225 } } peers { permanent_uuid: "9502119b0cf747b2a6e6dad2565c8f98" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33783 } } peers { permanent_uuid: "adcd2340ef804b518a235266c7499a71" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45433 } }
I20250622 14:00:27.449854 22327 leader_election.cc:290] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9502119b0cf747b2a6e6dad2565c8f98 (127.15.25.67:33783), adcd2340ef804b518a235266c7499a71 (127.15.25.66:45433)
I20250622 14:00:27.451520 22128 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "215cdfd3a26c4bc5bec0fa817a8b3e36" candidate_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "adcd2340ef804b518a235266c7499a71" is_pre_election: true
I20250622 14:00:27.452464 22128 raft_consensus.cc:2466] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a88bdd42d19b4fd9a56021e3ad01cd7f in term 1.
I20250622 14:00:27.454140 21925 leader_election.cc:304] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a88bdd42d19b4fd9a56021e3ad01cd7f, adcd2340ef804b518a235266c7499a71; no voters:
I20250622 14:00:27.455189 22327 raft_consensus.cc:2802] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250622 14:00:27.455618 22327 raft_consensus.cc:491] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 14:00:27.456063 22327 raft_consensus.cc:3058] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f [term 1 FOLLOWER]: Advancing to term 2
I20250622 14:00:27.465992 22327 raft_consensus.cc:513] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 34225 } } peers { permanent_uuid: "9502119b0cf747b2a6e6dad2565c8f98" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33783 } } peers { permanent_uuid: "adcd2340ef804b518a235266c7499a71" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45433 } }
I20250622 14:00:27.467492 22327 leader_election.cc:290] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f [CANDIDATE]: Term 2 election: Requested vote from peers 9502119b0cf747b2a6e6dad2565c8f98 (127.15.25.67:33783), adcd2340ef804b518a235266c7499a71 (127.15.25.66:45433)
I20250622 14:00:27.468417 22128 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "215cdfd3a26c4bc5bec0fa817a8b3e36" candidate_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "adcd2340ef804b518a235266c7499a71"
I20250622 14:00:27.468827 22128 raft_consensus.cc:3058] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71 [term 1 FOLLOWER]: Advancing to term 2
I20250622 14:00:27.477442 22128 raft_consensus.cc:2466] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a88bdd42d19b4fd9a56021e3ad01cd7f in term 2.
I20250622 14:00:27.478418 21925 leader_election.cc:304] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a88bdd42d19b4fd9a56021e3ad01cd7f, adcd2340ef804b518a235266c7499a71; no voters:
I20250622 14:00:27.479108 22327 raft_consensus.cc:2802] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f [term 2 FOLLOWER]: Leader election won for term 2
I20250622 14:00:27.480625 22327 raft_consensus.cc:695] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f [term 2 LEADER]: Becoming Leader. State: Replica: a88bdd42d19b4fd9a56021e3ad01cd7f, State: Running, Role: LEADER
I20250622 14:00:27.481521 22327 consensus_queue.cc:237] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 34225 } } peers { permanent_uuid: "9502119b0cf747b2a6e6dad2565c8f98" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33783 } } peers { permanent_uuid: "adcd2340ef804b518a235266c7499a71" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45433 } }
I20250622 14:00:27.473620 22268 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "215cdfd3a26c4bc5bec0fa817a8b3e36" candidate_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "9502119b0cf747b2a6e6dad2565c8f98"
I20250622 14:00:27.473415 22269 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "215cdfd3a26c4bc5bec0fa817a8b3e36" candidate_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "9502119b0cf747b2a6e6dad2565c8f98" is_pre_election: true
W20250622 14:00:27.485399 21927 leader_election.cc:343] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 9502119b0cf747b2a6e6dad2565c8f98 (127.15.25.67:33783): Illegal state: must be running to vote when last-logged opid is not known
I20250622 14:00:27.493114 21848 catalog_manager.cc:5582] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f reported cstate change: term changed from 0 to 2, leader changed from <none> to a88bdd42d19b4fd9a56021e3ad01cd7f (127.15.25.65), VOTER 9502119b0cf747b2a6e6dad2565c8f98 (127.15.25.67) added, VOTER a88bdd42d19b4fd9a56021e3ad01cd7f (127.15.25.65) added, VOTER adcd2340ef804b518a235266c7499a71 (127.15.25.66) added. New cstate: current_term: 2 leader_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 34225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9502119b0cf747b2a6e6dad2565c8f98" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "adcd2340ef804b518a235266c7499a71" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45433 } health_report { overall_health: UNKNOWN } } }
I20250622 14:00:27.954471 22128 raft_consensus.cc:1273] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71 [term 2 FOLLOWER]: Refusing update from remote peer a88bdd42d19b4fd9a56021e3ad01cd7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250622 14:00:27.956036 22327 consensus_queue.cc:1035] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "adcd2340ef804b518a235266c7499a71" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
W20250622 14:00:28.008602 21927 consensus_peers.cc:487] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f -> Peer 9502119b0cf747b2a6e6dad2565c8f98 (127.15.25.67:33783): Couldn't send request to peer 9502119b0cf747b2a6e6dad2565c8f98. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250622 14:00:28.046891 21992 consensus_queue.cc:237] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 2.206, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 34225 } } peers { permanent_uuid: "adcd2340ef804b518a235266c7499a71" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45433 } }
I20250622 14:00:28.052451 22128 raft_consensus.cc:1273] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71 [term 2 FOLLOWER]: Refusing update from remote peer a88bdd42d19b4fd9a56021e3ad01cd7f: Log matching property violated. Preceding OpId in replica: term: 2 index: 206. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
I20250622 14:00:28.053884 22345 consensus_queue.cc:1035] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "adcd2340ef804b518a235266c7499a71" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.000s
I20250622 14:00:28.059597 22345 raft_consensus.cc:2953] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f [term 2 LEADER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 9502119b0cf747b2a6e6dad2565c8f98 (127.15.25.67) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 34225 } } peers { permanent_uuid: "adcd2340ef804b518a235266c7499a71" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45433 } } }
I20250622 14:00:28.061868 22128 raft_consensus.cc:2953] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71 [term 2 FOLLOWER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 9502119b0cf747b2a6e6dad2565c8f98 (127.15.25.67) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 34225 } } peers { permanent_uuid: "adcd2340ef804b518a235266c7499a71" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45433 } } }
I20250622 14:00:28.077419 21835 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 215cdfd3a26c4bc5bec0fa817a8b3e36 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 14:00:28.081720 21848 catalog_manager.cc:5582] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71 reported cstate change: config changed from index -1 to 207, VOTER 9502119b0cf747b2a6e6dad2565c8f98 (127.15.25.67) evicted. New cstate: current_term: 2 leader_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 34225 } } peers { permanent_uuid: "adcd2340ef804b518a235266c7499a71" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45433 } } }
I20250622 14:00:28.114032 21992 consensus_queue.cc:237] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 2.207, Last appended by leader: 205, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 34225 } }
I20250622 14:00:28.116350 22345 raft_consensus.cc:2953] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f [term 2 LEADER]: Committing config change with OpId 2.208: config changed from index 207 to 208, VOTER adcd2340ef804b518a235266c7499a71 (127.15.25.66) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 34225 } } }
I20250622 14:00:28.124176 21835 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 215cdfd3a26c4bc5bec0fa817a8b3e36 with cas_config_opid_index 207: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 14:00:28.127429 21849 catalog_manager.cc:5582] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f reported cstate change: config changed from index 207 to 208, VOTER adcd2340ef804b518a235266c7499a71 (127.15.25.66) evicted. New cstate: current_term: 2 leader_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 34225 } health_report { overall_health: HEALTHY } } }
I20250622 14:00:28.131244 22249 tablet_service.cc:1515] Processing DeleteTablet for tablet 215cdfd3a26c4bc5bec0fa817a8b3e36 with delete_type TABLET_DATA_TOMBSTONED (TS 9502119b0cf747b2a6e6dad2565c8f98 not found in new config with opid_index 207) from {username='slave'} at 127.0.0.1:58414
W20250622 14:00:28.134742 21836 catalog_manager.cc:4908] TS 9502119b0cf747b2a6e6dad2565c8f98 (127.15.25.67:33783): delete failed for tablet 215cdfd3a26c4bc5bec0fa817a8b3e36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 215cdfd3a26c4bc5bec0fa817a8b3e36 already in progress: opening tablet
I20250622 14:00:28.146361 22207 tablet_bootstrap.cc:492] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P 9502119b0cf747b2a6e6dad2565c8f98: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 14:00:28.147639 22207 tablet_bootstrap.cc:492] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P 9502119b0cf747b2a6e6dad2565c8f98: Bootstrap complete.
I20250622 14:00:28.149710 22207 ts_tablet_manager.cc:1397] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P 9502119b0cf747b2a6e6dad2565c8f98: Time spent bootstrapping tablet: real 2.236s user 2.125s sys 0.072s
I20250622 14:00:28.157037 22207 raft_consensus.cc:357] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P 9502119b0cf747b2a6e6dad2565c8f98 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 34225 } } peers { permanent_uuid: "9502119b0cf747b2a6e6dad2565c8f98" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33783 } } peers { permanent_uuid: "adcd2340ef804b518a235266c7499a71" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45433 } }
I20250622 14:00:28.159952 22207 raft_consensus.cc:738] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P 9502119b0cf747b2a6e6dad2565c8f98 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9502119b0cf747b2a6e6dad2565c8f98, State: Initialized, Role: FOLLOWER
I20250622 14:00:28.160205 22108 tablet_service.cc:1515] Processing DeleteTablet for tablet 215cdfd3a26c4bc5bec0fa817a8b3e36 with delete_type TABLET_DATA_TOMBSTONED (TS adcd2340ef804b518a235266c7499a71 not found in new config with opid_index 208) from {username='slave'} at 127.0.0.1:47794
I20250622 14:00:28.160965 22207 consensus_queue.cc:260] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P 9502119b0cf747b2a6e6dad2565c8f98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 34225 } } peers { permanent_uuid: "9502119b0cf747b2a6e6dad2565c8f98" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33783 } } peers { permanent_uuid: "adcd2340ef804b518a235266c7499a71" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45433 } }
W20250622 14:00:28.162779 15461 scanner-internal.cc:458] Time spent opening tablet: real 1.887s user 0.005s sys 0.004s
I20250622 14:00:28.174935 22354 tablet_replica.cc:331] stopping tablet replica
I20250622 14:00:28.175760 22354 raft_consensus.cc:2241] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 14:00:28.176513 22354 raft_consensus.cc:2270] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 14:00:28.189105 22207 ts_tablet_manager.cc:1428] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P 9502119b0cf747b2a6e6dad2565c8f98: Time spent starting tablet: real 0.039s user 0.011s sys 0.006s
I20250622 14:00:28.201682 22249 tablet_service.cc:1515] Processing DeleteTablet for tablet 215cdfd3a26c4bc5bec0fa817a8b3e36 with delete_type TABLET_DATA_TOMBSTONED (Replica has no consensus available (current committed config index is 208)) from {username='slave'} at 127.0.0.1:58414
I20250622 14:00:28.202724 22352 tablet_replica.cc:331] stopping tablet replica
I20250622 14:00:28.203490 22352 raft_consensus.cc:2241] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P 9502119b0cf747b2a6e6dad2565c8f98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 14:00:28.203996 22352 raft_consensus.cc:2270] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P 9502119b0cf747b2a6e6dad2565c8f98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 14:00:28.235044 22354 ts_tablet_manager.cc:1905] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250622 14:00:28.251780 22352 ts_tablet_manager.cc:1905] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P 9502119b0cf747b2a6e6dad2565c8f98: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250622 14:00:28.261718 22354 ts_tablet_manager.cc:1918] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.207
I20250622 14:00:28.275961 22354 log.cc:1199] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P adcd2340ef804b518a235266c7499a71: Deleting WAL directory at /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-1/wal/wals/215cdfd3a26c4bc5bec0fa817a8b3e36
I20250622 14:00:28.277940 21834 catalog_manager.cc:4928] TS adcd2340ef804b518a235266c7499a71 (127.15.25.66:45433): tablet 215cdfd3a26c4bc5bec0fa817a8b3e36 (table pre_rebuild [id=0aa5b3d575144d50b7f3f1c6898e9861]) successfully deleted
I20250622 14:00:28.279960 22352 ts_tablet_manager.cc:1918] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P 9502119b0cf747b2a6e6dad2565c8f98: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.205
I20250622 14:00:28.280450 22352 log.cc:1199] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P 9502119b0cf747b2a6e6dad2565c8f98: Deleting WAL directory at /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-2/wal/wals/215cdfd3a26c4bc5bec0fa817a8b3e36
I20250622 14:00:28.282143 21836 catalog_manager.cc:4928] TS 9502119b0cf747b2a6e6dad2565c8f98 (127.15.25.67:33783): tablet 215cdfd3a26c4bc5bec0fa817a8b3e36 (table pre_rebuild [id=0aa5b3d575144d50b7f3f1c6898e9861]) successfully deleted
I20250622 14:00:28.753501 22249 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 14:00:28.769912 21972 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250622 14:00:28.775373 22108 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
2d9465de7cc54dfc9f15ded620288a37 | 127.15.25.126:42519 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.15.25.84:42217 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
9502119b0cf747b2a6e6dad2565c8f98 | 127.15.25.67:33783 | HEALTHY | <none> | 0 | 0
a88bdd42d19b4fd9a56021e3ad01cd7f | 127.15.25.65:34225 | HEALTHY | <none> | 1 | 0
adcd2340ef804b518a235266c7499a71 | 127.15.25.66:45433 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.15.25.65 | experimental | 127.15.25.65:34225
local_ip_for_outbound_sockets | 127.15.25.66 | experimental | 127.15.25.66:45433
local_ip_for_outbound_sockets | 127.15.25.67 | experimental | 127.15.25.67:33783
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.25.65:34225
server_dump_info_path | /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.25.66:45433
server_dump_info_path | /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.25.67:33783
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.15.25.84:42217 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------+----+---------+---------------+---------+------------+------------------+-------------
pre_rebuild | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 0
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 1
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250622 14:00:29.019455 15461 log_verifier.cc:126] Checking tablet 215cdfd3a26c4bc5bec0fa817a8b3e36
I20250622 14:00:29.273540 15461 log_verifier.cc:177] Verified matching terms for 208 ops in tablet 215cdfd3a26c4bc5bec0fa817a8b3e36
I20250622 14:00:29.275771 21849 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58288:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250622 14:00:29.276283 21849 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58288:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250622 14:00:29.287420 21849 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 2d9465de7cc54dfc9f15ded620288a37: Sending DeleteTablet for 1 replicas of tablet 215cdfd3a26c4bc5bec0fa817a8b3e36
I20250622 14:00:29.288980 21972 tablet_service.cc:1515] Processing DeleteTablet for tablet 215cdfd3a26c4bc5bec0fa817a8b3e36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-22 14:00:29 UTC) from {username='slave'} at 127.0.0.1:58558
I20250622 14:00:29.289060 15461 test_util.cc:276] Using random seed: -2070756786
I20250622 14:00:29.290535 22386 tablet_replica.cc:331] stopping tablet replica
I20250622 14:00:29.291100 22386 raft_consensus.cc:2241] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f [term 2 LEADER]: Raft consensus shutting down.
I20250622 14:00:29.291594 22386 raft_consensus.cc:2270] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 14:00:29.319806 22386 ts_tablet_manager.cc:1905] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250622 14:00:29.323096 21849 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58316:
name: "post_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250622 14:00:29.326393 21849 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table post_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250622 14:00:29.330397 22386 ts_tablet_manager.cc:1918] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.208
I20250622 14:00:29.330704 22386 log.cc:1199] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f: Deleting WAL directory at /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-0/wal/wals/215cdfd3a26c4bc5bec0fa817a8b3e36
I20250622 14:00:29.331575 22386 ts_tablet_manager.cc:1939] T 215cdfd3a26c4bc5bec0fa817a8b3e36 P a88bdd42d19b4fd9a56021e3ad01cd7f: Deleting consensus metadata
I20250622 14:00:29.334308 21835 catalog_manager.cc:4928] TS a88bdd42d19b4fd9a56021e3ad01cd7f (127.15.25.65:34225): tablet 215cdfd3a26c4bc5bec0fa817a8b3e36 (table pre_rebuild [id=0aa5b3d575144d50b7f3f1c6898e9861]) successfully deleted
I20250622 14:00:29.349995 22108 tablet_service.cc:1468] Processing CreateTablet for tablet 28b643ef15f64ba59a72d7915c15edeb (DEFAULT_TABLE table=post_rebuild [id=7f1e31f5c7114cb99df5b92c691f2bc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 14:00:29.350726 22249 tablet_service.cc:1468] Processing CreateTablet for tablet 28b643ef15f64ba59a72d7915c15edeb (DEFAULT_TABLE table=post_rebuild [id=7f1e31f5c7114cb99df5b92c691f2bc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 14:00:29.351354 22108 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28b643ef15f64ba59a72d7915c15edeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 14:00:29.351526 21972 tablet_service.cc:1468] Processing CreateTablet for tablet 28b643ef15f64ba59a72d7915c15edeb (DEFAULT_TABLE table=post_rebuild [id=7f1e31f5c7114cb99df5b92c691f2bc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 14:00:29.351963 22249 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28b643ef15f64ba59a72d7915c15edeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 14:00:29.352756 21972 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28b643ef15f64ba59a72d7915c15edeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 14:00:29.373004 22393 tablet_bootstrap.cc:492] T 28b643ef15f64ba59a72d7915c15edeb P a88bdd42d19b4fd9a56021e3ad01cd7f: Bootstrap starting.
I20250622 14:00:29.375639 22394 tablet_bootstrap.cc:492] T 28b643ef15f64ba59a72d7915c15edeb P adcd2340ef804b518a235266c7499a71: Bootstrap starting.
I20250622 14:00:29.378654 22395 tablet_bootstrap.cc:492] T 28b643ef15f64ba59a72d7915c15edeb P 9502119b0cf747b2a6e6dad2565c8f98: Bootstrap starting.
I20250622 14:00:29.382772 22395 tablet_bootstrap.cc:654] T 28b643ef15f64ba59a72d7915c15edeb P 9502119b0cf747b2a6e6dad2565c8f98: Neither blocks nor log segments found. Creating new log.
I20250622 14:00:29.383236 22393 tablet_bootstrap.cc:654] T 28b643ef15f64ba59a72d7915c15edeb P a88bdd42d19b4fd9a56021e3ad01cd7f: Neither blocks nor log segments found. Creating new log.
I20250622 14:00:29.387133 22394 tablet_bootstrap.cc:654] T 28b643ef15f64ba59a72d7915c15edeb P adcd2340ef804b518a235266c7499a71: Neither blocks nor log segments found. Creating new log.
I20250622 14:00:29.391489 22393 tablet_bootstrap.cc:492] T 28b643ef15f64ba59a72d7915c15edeb P a88bdd42d19b4fd9a56021e3ad01cd7f: No bootstrap required, opened a new log
I20250622 14:00:29.392056 22393 ts_tablet_manager.cc:1397] T 28b643ef15f64ba59a72d7915c15edeb P a88bdd42d19b4fd9a56021e3ad01cd7f: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.000s
I20250622 14:00:29.394695 22393 raft_consensus.cc:357] T 28b643ef15f64ba59a72d7915c15edeb P a88bdd42d19b4fd9a56021e3ad01cd7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adcd2340ef804b518a235266c7499a71" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45433 } } peers { permanent_uuid: "9502119b0cf747b2a6e6dad2565c8f98" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33783 } } peers { permanent_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 34225 } }
I20250622 14:00:29.396034 22393 raft_consensus.cc:383] T 28b643ef15f64ba59a72d7915c15edeb P a88bdd42d19b4fd9a56021e3ad01cd7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 14:00:29.396348 22393 raft_consensus.cc:738] T 28b643ef15f64ba59a72d7915c15edeb P a88bdd42d19b4fd9a56021e3ad01cd7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a88bdd42d19b4fd9a56021e3ad01cd7f, State: Initialized, Role: FOLLOWER
I20250622 14:00:29.397058 22393 consensus_queue.cc:260] T 28b643ef15f64ba59a72d7915c15edeb P a88bdd42d19b4fd9a56021e3ad01cd7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adcd2340ef804b518a235266c7499a71" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45433 } } peers { permanent_uuid: "9502119b0cf747b2a6e6dad2565c8f98" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33783 } } peers { permanent_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 34225 } }
I20250622 14:00:29.398348 22395 tablet_bootstrap.cc:492] T 28b643ef15f64ba59a72d7915c15edeb P 9502119b0cf747b2a6e6dad2565c8f98: No bootstrap required, opened a new log
I20250622 14:00:29.398823 22395 ts_tablet_manager.cc:1397] T 28b643ef15f64ba59a72d7915c15edeb P 9502119b0cf747b2a6e6dad2565c8f98: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.005s
I20250622 14:00:29.402288 22395 raft_consensus.cc:357] T 28b643ef15f64ba59a72d7915c15edeb P 9502119b0cf747b2a6e6dad2565c8f98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adcd2340ef804b518a235266c7499a71" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45433 } } peers { permanent_uuid: "9502119b0cf747b2a6e6dad2565c8f98" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33783 } } peers { permanent_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 34225 } }
I20250622 14:00:29.402887 22395 raft_consensus.cc:383] T 28b643ef15f64ba59a72d7915c15edeb P 9502119b0cf747b2a6e6dad2565c8f98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 14:00:29.403151 22395 raft_consensus.cc:738] T 28b643ef15f64ba59a72d7915c15edeb P 9502119b0cf747b2a6e6dad2565c8f98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9502119b0cf747b2a6e6dad2565c8f98, State: Initialized, Role: FOLLOWER
I20250622 14:00:29.403806 22395 consensus_queue.cc:260] T 28b643ef15f64ba59a72d7915c15edeb P 9502119b0cf747b2a6e6dad2565c8f98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adcd2340ef804b518a235266c7499a71" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45433 } } peers { permanent_uuid: "9502119b0cf747b2a6e6dad2565c8f98" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33783 } } peers { permanent_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 34225 } }
I20250622 14:00:29.407536 22393 ts_tablet_manager.cc:1428] T 28b643ef15f64ba59a72d7915c15edeb P a88bdd42d19b4fd9a56021e3ad01cd7f: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20250622 14:00:29.408222 22394 tablet_bootstrap.cc:492] T 28b643ef15f64ba59a72d7915c15edeb P adcd2340ef804b518a235266c7499a71: No bootstrap required, opened a new log
I20250622 14:00:29.408622 22394 ts_tablet_manager.cc:1397] T 28b643ef15f64ba59a72d7915c15edeb P adcd2340ef804b518a235266c7499a71: Time spent bootstrapping tablet: real 0.033s user 0.006s sys 0.016s
I20250622 14:00:29.411165 22394 raft_consensus.cc:357] T 28b643ef15f64ba59a72d7915c15edeb P adcd2340ef804b518a235266c7499a71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adcd2340ef804b518a235266c7499a71" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45433 } } peers { permanent_uuid: "9502119b0cf747b2a6e6dad2565c8f98" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33783 } } peers { permanent_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 34225 } }
I20250622 14:00:29.411927 22394 raft_consensus.cc:383] T 28b643ef15f64ba59a72d7915c15edeb P adcd2340ef804b518a235266c7499a71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 14:00:29.412251 22394 raft_consensus.cc:738] T 28b643ef15f64ba59a72d7915c15edeb P adcd2340ef804b518a235266c7499a71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adcd2340ef804b518a235266c7499a71, State: Initialized, Role: FOLLOWER
I20250622 14:00:29.412925 22394 consensus_queue.cc:260] T 28b643ef15f64ba59a72d7915c15edeb P adcd2340ef804b518a235266c7499a71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adcd2340ef804b518a235266c7499a71" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45433 } } peers { permanent_uuid: "9502119b0cf747b2a6e6dad2565c8f98" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33783 } } peers { permanent_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 34225 } }
I20250622 14:00:29.415160 22395 ts_tablet_manager.cc:1428] T 28b643ef15f64ba59a72d7915c15edeb P 9502119b0cf747b2a6e6dad2565c8f98: Time spent starting tablet: real 0.016s user 0.008s sys 0.000s
I20250622 14:00:29.418959 22394 ts_tablet_manager.cc:1428] T 28b643ef15f64ba59a72d7915c15edeb P adcd2340ef804b518a235266c7499a71: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
W20250622 14:00:29.429828 22315 tablet.cc:2378] T 28b643ef15f64ba59a72d7915c15edeb P 9502119b0cf747b2a6e6dad2565c8f98: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 14:00:29.476413 22399 raft_consensus.cc:491] T 28b643ef15f64ba59a72d7915c15edeb P a88bdd42d19b4fd9a56021e3ad01cd7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 14:00:29.476754 22399 raft_consensus.cc:513] T 28b643ef15f64ba59a72d7915c15edeb P a88bdd42d19b4fd9a56021e3ad01cd7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adcd2340ef804b518a235266c7499a71" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45433 } } peers { permanent_uuid: "9502119b0cf747b2a6e6dad2565c8f98" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33783 } } peers { permanent_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 34225 } }
I20250622 14:00:29.478178 22399 leader_election.cc:290] T 28b643ef15f64ba59a72d7915c15edeb P a88bdd42d19b4fd9a56021e3ad01cd7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers adcd2340ef804b518a235266c7499a71 (127.15.25.66:45433), 9502119b0cf747b2a6e6dad2565c8f98 (127.15.25.67:33783)
I20250622 14:00:29.479028 22128 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28b643ef15f64ba59a72d7915c15edeb" candidate_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adcd2340ef804b518a235266c7499a71" is_pre_election: true
I20250622 14:00:29.479166 22269 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28b643ef15f64ba59a72d7915c15edeb" candidate_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9502119b0cf747b2a6e6dad2565c8f98" is_pre_election: true
I20250622 14:00:29.479453 22128 raft_consensus.cc:2466] T 28b643ef15f64ba59a72d7915c15edeb P adcd2340ef804b518a235266c7499a71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a88bdd42d19b4fd9a56021e3ad01cd7f in term 0.
I20250622 14:00:29.479797 22269 raft_consensus.cc:2466] T 28b643ef15f64ba59a72d7915c15edeb P 9502119b0cf747b2a6e6dad2565c8f98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a88bdd42d19b4fd9a56021e3ad01cd7f in term 0.
I20250622 14:00:29.480270 21925 leader_election.cc:304] T 28b643ef15f64ba59a72d7915c15edeb P a88bdd42d19b4fd9a56021e3ad01cd7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a88bdd42d19b4fd9a56021e3ad01cd7f, adcd2340ef804b518a235266c7499a71; no voters:
I20250622 14:00:29.480911 22399 raft_consensus.cc:2802] T 28b643ef15f64ba59a72d7915c15edeb P a88bdd42d19b4fd9a56021e3ad01cd7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 14:00:29.481209 22399 raft_consensus.cc:491] T 28b643ef15f64ba59a72d7915c15edeb P a88bdd42d19b4fd9a56021e3ad01cd7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 14:00:29.481452 22399 raft_consensus.cc:3058] T 28b643ef15f64ba59a72d7915c15edeb P a88bdd42d19b4fd9a56021e3ad01cd7f [term 0 FOLLOWER]: Advancing to term 1
I20250622 14:00:29.485278 22399 raft_consensus.cc:513] T 28b643ef15f64ba59a72d7915c15edeb P a88bdd42d19b4fd9a56021e3ad01cd7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adcd2340ef804b518a235266c7499a71" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45433 } } peers { permanent_uuid: "9502119b0cf747b2a6e6dad2565c8f98" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33783 } } peers { permanent_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 34225 } }
W20250622 14:00:29.485761 22174 tablet.cc:2378] T 28b643ef15f64ba59a72d7915c15edeb P adcd2340ef804b518a235266c7499a71: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 14:00:29.486863 22399 leader_election.cc:290] T 28b643ef15f64ba59a72d7915c15edeb P a88bdd42d19b4fd9a56021e3ad01cd7f [CANDIDATE]: Term 1 election: Requested vote from peers adcd2340ef804b518a235266c7499a71 (127.15.25.66:45433), 9502119b0cf747b2a6e6dad2565c8f98 (127.15.25.67:33783)
I20250622 14:00:29.487551 22128 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28b643ef15f64ba59a72d7915c15edeb" candidate_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adcd2340ef804b518a235266c7499a71"
I20250622 14:00:29.487720 22269 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28b643ef15f64ba59a72d7915c15edeb" candidate_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9502119b0cf747b2a6e6dad2565c8f98"
I20250622 14:00:29.488021 22128 raft_consensus.cc:3058] T 28b643ef15f64ba59a72d7915c15edeb P adcd2340ef804b518a235266c7499a71 [term 0 FOLLOWER]: Advancing to term 1
I20250622 14:00:29.488128 22269 raft_consensus.cc:3058] T 28b643ef15f64ba59a72d7915c15edeb P 9502119b0cf747b2a6e6dad2565c8f98 [term 0 FOLLOWER]: Advancing to term 1
I20250622 14:00:29.493709 22269 raft_consensus.cc:2466] T 28b643ef15f64ba59a72d7915c15edeb P 9502119b0cf747b2a6e6dad2565c8f98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a88bdd42d19b4fd9a56021e3ad01cd7f in term 1.
I20250622 14:00:29.494002 22128 raft_consensus.cc:2466] T 28b643ef15f64ba59a72d7915c15edeb P adcd2340ef804b518a235266c7499a71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a88bdd42d19b4fd9a56021e3ad01cd7f in term 1.
I20250622 14:00:29.494402 21927 leader_election.cc:304] T 28b643ef15f64ba59a72d7915c15edeb P a88bdd42d19b4fd9a56021e3ad01cd7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9502119b0cf747b2a6e6dad2565c8f98, a88bdd42d19b4fd9a56021e3ad01cd7f; no voters:
I20250622 14:00:29.495025 22399 raft_consensus.cc:2802] T 28b643ef15f64ba59a72d7915c15edeb P a88bdd42d19b4fd9a56021e3ad01cd7f [term 1 FOLLOWER]: Leader election won for term 1
I20250622 14:00:29.496232 22399 raft_consensus.cc:695] T 28b643ef15f64ba59a72d7915c15edeb P a88bdd42d19b4fd9a56021e3ad01cd7f [term 1 LEADER]: Becoming Leader. State: Replica: a88bdd42d19b4fd9a56021e3ad01cd7f, State: Running, Role: LEADER
I20250622 14:00:29.496855 22399 consensus_queue.cc:237] T 28b643ef15f64ba59a72d7915c15edeb P a88bdd42d19b4fd9a56021e3ad01cd7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adcd2340ef804b518a235266c7499a71" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45433 } } peers { permanent_uuid: "9502119b0cf747b2a6e6dad2565c8f98" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33783 } } peers { permanent_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 34225 } }
I20250622 14:00:29.502918 21847 catalog_manager.cc:5582] T 28b643ef15f64ba59a72d7915c15edeb P a88bdd42d19b4fd9a56021e3ad01cd7f reported cstate change: term changed from 0 to 1, leader changed from <none> to a88bdd42d19b4fd9a56021e3ad01cd7f (127.15.25.65). New cstate: current_term: 1 leader_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adcd2340ef804b518a235266c7499a71" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9502119b0cf747b2a6e6dad2565c8f98" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a88bdd42d19b4fd9a56021e3ad01cd7f" member_type: VOTER last_known_addr { host: "127.15.25.65" port: 34225 } health_report { overall_health: HEALTHY } } }
W20250622 14:00:29.598651 22038 tablet.cc:2378] T 28b643ef15f64ba59a72d7915c15edeb P a88bdd42d19b4fd9a56021e3ad01cd7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 14:00:29.697177 22128 raft_consensus.cc:1273] T 28b643ef15f64ba59a72d7915c15edeb P adcd2340ef804b518a235266c7499a71 [term 1 FOLLOWER]: Refusing update from remote peer a88bdd42d19b4fd9a56021e3ad01cd7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 14:00:29.697429 22269 raft_consensus.cc:1273] T 28b643ef15f64ba59a72d7915c15edeb P 9502119b0cf747b2a6e6dad2565c8f98 [term 1 FOLLOWER]: Refusing update from remote peer a88bdd42d19b4fd9a56021e3ad01cd7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 14:00:29.698699 22402 consensus_queue.cc:1035] T 28b643ef15f64ba59a72d7915c15edeb P a88bdd42d19b4fd9a56021e3ad01cd7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "adcd2340ef804b518a235266c7499a71" member_type: VOTER last_known_addr { host: "127.15.25.66" port: 45433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250622 14:00:29.699936 22399 consensus_queue.cc:1035] T 28b643ef15f64ba59a72d7915c15edeb P a88bdd42d19b4fd9a56021e3ad01cd7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9502119b0cf747b2a6e6dad2565c8f98" member_type: VOTER last_known_addr { host: "127.15.25.67" port: 33783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 14:00:29.736764 22411 mvcc.cc:204] Tried to move back new op lower bound from 7170460998421258240 to 7170460997624815616. Current Snapshot: MvccSnapshot[applied={T|T < 7170460998421258240}]
I20250622 14:00:29.748090 22412 mvcc.cc:204] Tried to move back new op lower bound from 7170460998421258240 to 7170460997624815616. Current Snapshot: MvccSnapshot[applied={T|T < 7170460998421258240}]
W20250622 14:00:30.578552 21927 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.25.67:33783, user_credentials={real_user=slave}} blocked reactor thread for 113386us
I20250622 14:00:34.520228 22249 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 14:00:34.520764 21972 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250622 14:00:34.522148 22108 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
2d9465de7cc54dfc9f15ded620288a37 | 127.15.25.126:42519 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.15.25.84:42217 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
9502119b0cf747b2a6e6dad2565c8f98 | 127.15.25.67:33783 | HEALTHY | <none> | 0 | 0
a88bdd42d19b4fd9a56021e3ad01cd7f | 127.15.25.65:34225 | HEALTHY | <none> | 1 | 0
adcd2340ef804b518a235266c7499a71 | 127.15.25.66:45433 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.15.25.65 | experimental | 127.15.25.65:34225
local_ip_for_outbound_sockets | 127.15.25.66 | experimental | 127.15.25.66:45433
local_ip_for_outbound_sockets | 127.15.25.67 | experimental | 127.15.25.67:33783
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.25.65:34225
server_dump_info_path | /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.25.66:45433
server_dump_info_path | /tmp/dist-test-task1wNjCM/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750600661728289-15461-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.25.67:33783
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.15.25.84:42217 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
--------------+----+---------+---------------+---------+------------+------------------+-------------
post_rebuild | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250622 14:00:34.703222 15461 log_verifier.cc:126] Checking tablet 215cdfd3a26c4bc5bec0fa817a8b3e36
I20250622 14:00:34.704145 15461 log_verifier.cc:177] Verified matching terms for 0 ops in tablet 215cdfd3a26c4bc5bec0fa817a8b3e36
I20250622 14:00:34.704342 15461 log_verifier.cc:126] Checking tablet 28b643ef15f64ba59a72d7915c15edeb
I20250622 14:00:35.408429 15461 log_verifier.cc:177] Verified matching terms for 205 ops in tablet 28b643ef15f64ba59a72d7915c15edeb
I20250622 14:00:35.434461 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 21886
I20250622 14:00:35.477473 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 22041
I20250622 14:00:35.514452 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 22177
I20250622 14:00:35.548695 15461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1wNjCM/build/tsan/bin/kudu with pid 21816
2025-06-22T14:00:35Z chronyd exiting
[ OK ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0 (33583 ms)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest (33583 ms total)
[----------] Global test environment tear-down
[==========] 9 tests from 5 test suites ran. (173810 ms total)
[ PASSED ] 8 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AdminCliTest.TestRebuildTables
1 FAILED TEST
I20250622 14:00:35.608287 15461 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/meta_cache.cc:302: suppressed but not reported on 2 messages since previous log ~45 seconds ago