Note: This is test shard 6 of 8.
[==========] Running 9 tests from 5 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from AdminCliTest
[ RUN ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250625 13:57:12.333294 13935 test_util.cc:276] Using random seed: -765750408
W20250625 13:57:13.548122 13935 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.170s user 0.458s sys 0.711s
W20250625 13:57:13.548467 13935 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.171s user 0.458s sys 0.711s
I20250625 13:57:13.563184 13935 ts_itest-base.cc:115] Starting cluster with:
I20250625 13:57:13.563373 13935 ts_itest-base.cc:116] --------------
I20250625 13:57:13.563529 13935 ts_itest-base.cc:117] 4 tablet servers
I20250625 13:57:13.563688 13935 ts_itest-base.cc:118] 3 replicas per TS
I20250625 13:57:13.563838 13935 ts_itest-base.cc:119] --------------
2025-06-25T13:57:13Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-25T13:57:13Z Disabled control of system clock
I20250625 13:57:13.602535 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.155.254:34591
--webserver_interface=127.13.155.254
--webserver_port=0
--builtin_ntp_servers=127.13.155.212:33671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.155.254:34591 with env {}
W20250625 13:57:13.919530 13949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:57:13.920127 13949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:57:13.920588 13949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:57:13.951174 13949 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 13:57:13.951490 13949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:57:13.951759 13949 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 13:57:13.952006 13949 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 13:57:13.986188 13949 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:33671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.155.254:34591
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.155.254:34591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.13.155.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:57:13.987488 13949 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:57:13.989094 13949 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:57:14.005321 13955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:57:14.005764 13956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:57:14.005776 13958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:57:14.006776 13949 server_base.cc:1048] running on GCE node
I20250625 13:57:15.177814 13949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:57:15.180779 13949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:57:15.182112 13949 hybrid_clock.cc:648] HybridClock initialized: now 1750859835182062 us; error 61 us; skew 500 ppm
I20250625 13:57:15.182924 13949 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:57:15.189438 13949 webserver.cc:469] Webserver started at http://127.13.155.254:35059/ using document root <none> and password file <none>
I20250625 13:57:15.190353 13949 fs_manager.cc:362] Metadata directory not provided
I20250625 13:57:15.190610 13949 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:57:15.191052 13949 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:57:15.195499 13949 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "f536eb4978b84fa287cd63c9d3878c6a"
format_stamp: "Formatted at 2025-06-25 13:57:15 on dist-test-slave-k028"
I20250625 13:57:15.196542 13949 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "f536eb4978b84fa287cd63c9d3878c6a"
format_stamp: "Formatted at 2025-06-25 13:57:15 on dist-test-slave-k028"
I20250625 13:57:15.204090 13949 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.001s sys 0.007s
I20250625 13:57:15.209887 13965 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:15.210925 13949 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250625 13:57:15.211263 13949 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f536eb4978b84fa287cd63c9d3878c6a"
format_stamp: "Formatted at 2025-06-25 13:57:15 on dist-test-slave-k028"
I20250625 13:57:15.211606 13949 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:57:15.261960 13949 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:57:15.263445 13949 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:57:15.263876 13949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:57:15.342548 13949 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.254:34591
I20250625 13:57:15.342618 14016 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.254:34591 every 8 connection(s)
I20250625 13:57:15.345449 13949 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250625 13:57:15.351506 14017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:57:15.354076 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 13949
I20250625 13:57:15.354542 13935 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250625 13:57:15.370021 14017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a: Bootstrap starting.
I20250625 13:57:15.375360 14017 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a: Neither blocks nor log segments found. Creating new log.
I20250625 13:57:15.376950 14017 log.cc:826] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a: Log is configured to *not* fsync() on all Append() calls
I20250625 13:57:15.382627 14017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a: No bootstrap required, opened a new log
I20250625 13:57:15.403903 14017 raft_consensus.cc:357] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f536eb4978b84fa287cd63c9d3878c6a" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 34591 } }
I20250625 13:57:15.404505 14017 raft_consensus.cc:383] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:57:15.404699 14017 raft_consensus.cc:738] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f536eb4978b84fa287cd63c9d3878c6a, State: Initialized, Role: FOLLOWER
I20250625 13:57:15.405333 14017 consensus_queue.cc:260] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f536eb4978b84fa287cd63c9d3878c6a" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 34591 } }
I20250625 13:57:15.405791 14017 raft_consensus.cc:397] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 13:57:15.406055 14017 raft_consensus.cc:491] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 13:57:15.406348 14017 raft_consensus.cc:3058] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:57:15.410297 14017 raft_consensus.cc:513] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f536eb4978b84fa287cd63c9d3878c6a" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 34591 } }
I20250625 13:57:15.410981 14017 leader_election.cc:304] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f536eb4978b84fa287cd63c9d3878c6a; no voters:
I20250625 13:57:15.412940 14017 leader_election.cc:290] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 13:57:15.415015 14022 raft_consensus.cc:2802] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:57:15.417635 14022 raft_consensus.cc:695] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a [term 1 LEADER]: Becoming Leader. State: Replica: f536eb4978b84fa287cd63c9d3878c6a, State: Running, Role: LEADER
I20250625 13:57:15.417824 14017 sys_catalog.cc:564] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a [sys.catalog]: configured and running, proceeding with master startup.
I20250625 13:57:15.418525 14022 consensus_queue.cc:237] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f536eb4978b84fa287cd63c9d3878c6a" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 34591 } }
I20250625 13:57:15.431975 14023 sys_catalog.cc:455] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f536eb4978b84fa287cd63c9d3878c6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f536eb4978b84fa287cd63c9d3878c6a" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 34591 } } }
I20250625 13:57:15.432807 14023 sys_catalog.cc:458] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a [sys.catalog]: This master's current role is: LEADER
I20250625 13:57:15.432428 14024 sys_catalog.cc:455] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f536eb4978b84fa287cd63c9d3878c6a. Latest consensus state: current_term: 1 leader_uuid: "f536eb4978b84fa287cd63c9d3878c6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f536eb4978b84fa287cd63c9d3878c6a" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 34591 } } }
I20250625 13:57:15.433203 14024 sys_catalog.cc:458] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a [sys.catalog]: This master's current role is: LEADER
I20250625 13:57:15.438503 14032 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 13:57:15.450944 14032 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 13:57:15.468214 14032 catalog_manager.cc:1349] Generated new cluster ID: 6f7fbade18654200947085c151ba07e0
I20250625 13:57:15.468523 14032 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 13:57:15.491184 14032 catalog_manager.cc:1372] Generated new certificate authority record
I20250625 13:57:15.492892 14032 catalog_manager.cc:1506] Loading token signing keys...
I20250625 13:57:15.518038 14032 catalog_manager.cc:5955] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a: Generated new TSK 0
I20250625 13:57:15.519161 14032 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 13:57:15.530999 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.155.193:0
--local_ip_for_outbound_sockets=127.13.155.193
--webserver_interface=127.13.155.193
--webserver_port=0
--tserver_master_addrs=127.13.155.254:34591
--builtin_ntp_servers=127.13.155.212:33671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250625 13:57:15.841090 14041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:57:15.841604 14041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:57:15.842330 14041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:57:15.873754 14041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:57:15.874660 14041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.155.193
I20250625 13:57:15.910544 14041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:33671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.155.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.13.155.193
--webserver_port=0
--tserver_master_addrs=127.13.155.254:34591
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.155.193
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:57:15.911851 14041 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:57:15.913486 14041 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:57:15.932974 14048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:57:15.933776 14050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:57:15.935310 14047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:57:17.447546 14049 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1513 milliseconds
I20250625 13:57:17.447649 14041 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 13:57:17.448812 14041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:57:17.451484 14041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:57:17.452898 14041 hybrid_clock.cc:648] HybridClock initialized: now 1750859837452856 us; error 58 us; skew 500 ppm
I20250625 13:57:17.453652 14041 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:57:17.460558 14041 webserver.cc:469] Webserver started at http://127.13.155.193:33939/ using document root <none> and password file <none>
I20250625 13:57:17.461426 14041 fs_manager.cc:362] Metadata directory not provided
I20250625 13:57:17.461608 14041 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:57:17.462018 14041 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:57:17.467306 14041 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "4b149ff2436c4feeb6efd63182aa92a4"
format_stamp: "Formatted at 2025-06-25 13:57:17 on dist-test-slave-k028"
I20250625 13:57:17.468446 14041 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "4b149ff2436c4feeb6efd63182aa92a4"
format_stamp: "Formatted at 2025-06-25 13:57:17 on dist-test-slave-k028"
I20250625 13:57:17.475338 14041 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20250625 13:57:17.481870 14057 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:17.482908 14041 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.001s
I20250625 13:57:17.483186 14041 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "4b149ff2436c4feeb6efd63182aa92a4"
format_stamp: "Formatted at 2025-06-25 13:57:17 on dist-test-slave-k028"
I20250625 13:57:17.483502 14041 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:57:17.534962 14041 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:57:17.536325 14041 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:57:17.537174 14041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:57:17.539616 14041 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 13:57:17.543921 14041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 13:57:17.544126 14041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:17.544315 14041 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 13:57:17.544510 14041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:17.693544 14041 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.193:46151
I20250625 13:57:17.693648 14169 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.193:46151 every 8 connection(s)
I20250625 13:57:17.696272 14041 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250625 13:57:17.704524 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 14041
I20250625 13:57:17.704960 13935 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250625 13:57:17.711014 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.155.194:0
--local_ip_for_outbound_sockets=127.13.155.194
--webserver_interface=127.13.155.194
--webserver_port=0
--tserver_master_addrs=127.13.155.254:34591
--builtin_ntp_servers=127.13.155.212:33671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 13:57:17.719192 14170 heartbeater.cc:344] Connected to a master server at 127.13.155.254:34591
I20250625 13:57:17.719744 14170 heartbeater.cc:461] Registering TS with master...
I20250625 13:57:17.721038 14170 heartbeater.cc:507] Master 127.13.155.254:34591 requested a full tablet report, sending...
I20250625 13:57:17.723793 13982 ts_manager.cc:194] Registered new tserver with Master: 4b149ff2436c4feeb6efd63182aa92a4 (127.13.155.193:46151)
I20250625 13:57:17.725610 13982 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.155.193:57275
W20250625 13:57:18.015763 14174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:57:18.016291 14174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:57:18.016798 14174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:57:18.049660 14174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:57:18.050559 14174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.155.194
I20250625 13:57:18.085258 14174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:33671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.155.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.13.155.194
--webserver_port=0
--tserver_master_addrs=127.13.155.254:34591
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.155.194
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:57:18.086581 14174 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:57:18.088189 14174 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:57:18.105041 14181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:57:18.729599 14170 heartbeater.cc:499] Master 127.13.155.254:34591 was elected leader, sending a full tablet report...
W20250625 13:57:18.106029 14183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:57:18.106961 14174 server_base.cc:1048] running on GCE node
W20250625 13:57:18.106233 14180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:57:19.268844 14174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:57:19.271044 14174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:57:19.272424 14174 hybrid_clock.cc:648] HybridClock initialized: now 1750859839272404 us; error 42 us; skew 500 ppm
I20250625 13:57:19.273195 14174 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:57:19.283555 14174 webserver.cc:469] Webserver started at http://127.13.155.194:45255/ using document root <none> and password file <none>
I20250625 13:57:19.284420 14174 fs_manager.cc:362] Metadata directory not provided
I20250625 13:57:19.284618 14174 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:57:19.285022 14174 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:57:19.289325 14174 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "245f46b721f44c14a4aad846784626d6"
format_stamp: "Formatted at 2025-06-25 13:57:19 on dist-test-slave-k028"
I20250625 13:57:19.290342 14174 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "245f46b721f44c14a4aad846784626d6"
format_stamp: "Formatted at 2025-06-25 13:57:19 on dist-test-slave-k028"
I20250625 13:57:19.297603 14174 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20250625 13:57:19.303907 14190 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:19.304896 14174 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250625 13:57:19.305161 14174 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "245f46b721f44c14a4aad846784626d6"
format_stamp: "Formatted at 2025-06-25 13:57:19 on dist-test-slave-k028"
I20250625 13:57:19.305449 14174 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:57:19.365455 14174 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:57:19.366874 14174 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:57:19.367291 14174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:57:19.370157 14174 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 13:57:19.374004 14174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 13:57:19.374193 14174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:19.374385 14174 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 13:57:19.374558 14174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250625 13:57:19.522430 14174 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.194:35519
I20250625 13:57:19.522583 14302 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.194:35519 every 8 connection(s)
I20250625 13:57:19.524943 14174 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250625 13:57:19.534389 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 14174
I20250625 13:57:19.534914 13935 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250625 13:57:19.541164 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.155.195:0
--local_ip_for_outbound_sockets=127.13.155.195
--webserver_interface=127.13.155.195
--webserver_port=0
--tserver_master_addrs=127.13.155.254:34591
--builtin_ntp_servers=127.13.155.212:33671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 13:57:19.549268 14303 heartbeater.cc:344] Connected to a master server at 127.13.155.254:34591
I20250625 13:57:19.549729 14303 heartbeater.cc:461] Registering TS with master...
I20250625 13:57:19.550771 14303 heartbeater.cc:507] Master 127.13.155.254:34591 requested a full tablet report, sending...
I20250625 13:57:19.552943 13982 ts_manager.cc:194] Registered new tserver with Master: 245f46b721f44c14a4aad846784626d6 (127.13.155.194:35519)
I20250625 13:57:19.554136 13982 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.155.194:44839
W20250625 13:57:19.851413 14307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:57:19.851889 14307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:57:19.852396 14307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:57:19.883175 14307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:57:19.884011 14307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.155.195
I20250625 13:57:19.917963 14307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:33671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.155.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.13.155.195
--webserver_port=0
--tserver_master_addrs=127.13.155.254:34591
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.155.195
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:57:19.919394 14307 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:57:19.920976 14307 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:57:19.937501 14313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:57:20.557518 14303 heartbeater.cc:499] Master 127.13.155.254:34591 was elected leader, sending a full tablet report...
W20250625 13:57:19.938668 14314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:57:19.939749 14307 server_base.cc:1048] running on GCE node
W20250625 13:57:19.939427 14316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:57:21.106272 14307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:57:21.108453 14307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:57:21.109799 14307 hybrid_clock.cc:648] HybridClock initialized: now 1750859841109761 us; error 59 us; skew 500 ppm
I20250625 13:57:21.110677 14307 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:57:21.117786 14307 webserver.cc:469] Webserver started at http://127.13.155.195:35339/ using document root <none> and password file <none>
I20250625 13:57:21.118762 14307 fs_manager.cc:362] Metadata directory not provided
I20250625 13:57:21.119000 14307 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:57:21.119421 14307 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:57:21.123783 14307 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "fa46df1c4e334e9c9e12db8c48ce2249"
format_stamp: "Formatted at 2025-06-25 13:57:21 on dist-test-slave-k028"
I20250625 13:57:21.124827 14307 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "fa46df1c4e334e9c9e12db8c48ce2249"
format_stamp: "Formatted at 2025-06-25 13:57:21 on dist-test-slave-k028"
I20250625 13:57:21.131707 14307 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250625 13:57:21.137796 14323 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:21.139073 14307 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250625 13:57:21.139376 14307 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "fa46df1c4e334e9c9e12db8c48ce2249"
format_stamp: "Formatted at 2025-06-25 13:57:21 on dist-test-slave-k028"
I20250625 13:57:21.139714 14307 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:57:21.187489 14307 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:57:21.188957 14307 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:57:21.189390 14307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:57:21.191952 14307 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 13:57:21.196410 14307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 13:57:21.196637 14307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:21.196884 14307 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 13:57:21.197084 14307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:21.362150 14307 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.195:36725
I20250625 13:57:21.362251 14435 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.195:36725 every 8 connection(s)
I20250625 13:57:21.364987 14307 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250625 13:57:21.374593 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 14307
I20250625 13:57:21.374975 13935 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250625 13:57:21.380455 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.155.196:0
--local_ip_for_outbound_sockets=127.13.155.196
--webserver_interface=127.13.155.196
--webserver_port=0
--tserver_master_addrs=127.13.155.254:34591
--builtin_ntp_servers=127.13.155.212:33671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 13:57:21.388317 14436 heartbeater.cc:344] Connected to a master server at 127.13.155.254:34591
I20250625 13:57:21.388715 14436 heartbeater.cc:461] Registering TS with master...
I20250625 13:57:21.389662 14436 heartbeater.cc:507] Master 127.13.155.254:34591 requested a full tablet report, sending...
I20250625 13:57:21.391680 13982 ts_manager.cc:194] Registered new tserver with Master: fa46df1c4e334e9c9e12db8c48ce2249 (127.13.155.195:36725)
I20250625 13:57:21.392856 13982 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.155.195:52513
W20250625 13:57:21.685693 14440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:57:21.686194 14440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:57:21.686722 14440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:57:21.718034 14440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:57:21.718920 14440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.155.196
I20250625 13:57:21.755209 14440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:33671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.155.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.13.155.196
--webserver_port=0
--tserver_master_addrs=127.13.155.254:34591
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.155.196
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:57:21.756484 14440 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:57:21.758224 14440 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:57:21.774219 14446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:57:22.395994 14436 heartbeater.cc:499] Master 127.13.155.254:34591 was elected leader, sending a full tablet report...
W20250625 13:57:21.774338 14447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:57:22.986255 14449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:57:22.988794 14448 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1211 milliseconds
I20250625 13:57:22.988903 14440 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 13:57:22.990145 14440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:57:22.992311 14440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:57:22.993674 14440 hybrid_clock.cc:648] HybridClock initialized: now 1750859842993632 us; error 42 us; skew 500 ppm
I20250625 13:57:22.994462 14440 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:57:23.000917 14440 webserver.cc:469] Webserver started at http://127.13.155.196:33693/ using document root <none> and password file <none>
I20250625 13:57:23.001829 14440 fs_manager.cc:362] Metadata directory not provided
I20250625 13:57:23.002038 14440 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:57:23.002507 14440 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:57:23.006870 14440 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "69298260f4934e35a7f68c21c2205139"
format_stamp: "Formatted at 2025-06-25 13:57:23 on dist-test-slave-k028"
I20250625 13:57:23.008069 14440 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "69298260f4934e35a7f68c21c2205139"
format_stamp: "Formatted at 2025-06-25 13:57:23 on dist-test-slave-k028"
I20250625 13:57:23.015547 14440 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20250625 13:57:23.021569 14457 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:23.022562 14440 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20250625 13:57:23.022866 14440 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "69298260f4934e35a7f68c21c2205139"
format_stamp: "Formatted at 2025-06-25 13:57:23 on dist-test-slave-k028"
I20250625 13:57:23.023217 14440 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:57:23.072736 14440 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:57:23.074743 14440 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:57:23.075172 14440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:57:23.077502 14440 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 13:57:23.081851 14440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 13:57:23.082054 14440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:23.082301 14440 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 13:57:23.082479 14440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:23.235033 14440 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.196:39099
I20250625 13:57:23.235141 14569 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.196:39099 every 8 connection(s)
I20250625 13:57:23.237767 14440 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250625 13:57:23.245554 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 14440
I20250625 13:57:23.246086 13935 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250625 13:57:23.259267 14570 heartbeater.cc:344] Connected to a master server at 127.13.155.254:34591
I20250625 13:57:23.259703 14570 heartbeater.cc:461] Registering TS with master...
I20250625 13:57:23.260833 14570 heartbeater.cc:507] Master 127.13.155.254:34591 requested a full tablet report, sending...
I20250625 13:57:23.262895 13982 ts_manager.cc:194] Registered new tserver with Master: 69298260f4934e35a7f68c21c2205139 (127.13.155.196:39099)
I20250625 13:57:23.264195 13982 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.155.196:46371
I20250625 13:57:23.266580 13935 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250625 13:57:23.307981 13982 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49144:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250625 13:57:23.399955 14105 tablet_service.cc:1468] Processing CreateTablet for tablet cdd115530ef34645968765005dd3caf3 (DEFAULT_TABLE table=TestTable [id=6548b90a615c4e37a94459d5aa3733b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:57:23.405186 14105 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdd115530ef34645968765005dd3caf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:57:23.411113 14371 tablet_service.cc:1468] Processing CreateTablet for tablet cdd115530ef34645968765005dd3caf3 (DEFAULT_TABLE table=TestTable [id=6548b90a615c4e37a94459d5aa3733b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:57:23.414211 14238 tablet_service.cc:1468] Processing CreateTablet for tablet cdd115530ef34645968765005dd3caf3 (DEFAULT_TABLE table=TestTable [id=6548b90a615c4e37a94459d5aa3733b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:57:23.416198 14238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdd115530ef34645968765005dd3caf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:57:23.417784 14371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdd115530ef34645968765005dd3caf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:57:23.437224 14589 tablet_bootstrap.cc:492] T cdd115530ef34645968765005dd3caf3 P 4b149ff2436c4feeb6efd63182aa92a4: Bootstrap starting.
I20250625 13:57:23.443576 14590 tablet_bootstrap.cc:492] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6: Bootstrap starting.
I20250625 13:57:23.445122 14589 tablet_bootstrap.cc:654] T cdd115530ef34645968765005dd3caf3 P 4b149ff2436c4feeb6efd63182aa92a4: Neither blocks nor log segments found. Creating new log.
I20250625 13:57:23.447496 14589 log.cc:826] T cdd115530ef34645968765005dd3caf3 P 4b149ff2436c4feeb6efd63182aa92a4: Log is configured to *not* fsync() on all Append() calls
I20250625 13:57:23.452649 14592 tablet_bootstrap.cc:492] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249: Bootstrap starting.
I20250625 13:57:23.453641 14589 tablet_bootstrap.cc:492] T cdd115530ef34645968765005dd3caf3 P 4b149ff2436c4feeb6efd63182aa92a4: No bootstrap required, opened a new log
I20250625 13:57:23.453795 14590 tablet_bootstrap.cc:654] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6: Neither blocks nor log segments found. Creating new log.
I20250625 13:57:23.454169 14589 ts_tablet_manager.cc:1397] T cdd115530ef34645968765005dd3caf3 P 4b149ff2436c4feeb6efd63182aa92a4: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.008s
I20250625 13:57:23.456554 14590 log.cc:826] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6: Log is configured to *not* fsync() on all Append() calls
I20250625 13:57:23.459146 14592 tablet_bootstrap.cc:654] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249: Neither blocks nor log segments found. Creating new log.
I20250625 13:57:23.461712 14592 log.cc:826] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249: Log is configured to *not* fsync() on all Append() calls
I20250625 13:57:23.463107 14590 tablet_bootstrap.cc:492] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6: No bootstrap required, opened a new log
I20250625 13:57:23.463682 14590 ts_tablet_manager.cc:1397] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6: Time spent bootstrapping tablet: real 0.021s user 0.014s sys 0.003s
I20250625 13:57:23.468436 14592 tablet_bootstrap.cc:492] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249: No bootstrap required, opened a new log
I20250625 13:57:23.468963 14592 ts_tablet_manager.cc:1397] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.000s
I20250625 13:57:23.477288 14589 raft_consensus.cc:357] T cdd115530ef34645968765005dd3caf3 P 4b149ff2436c4feeb6efd63182aa92a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b149ff2436c4feeb6efd63182aa92a4" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 46151 } } peers { permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 } } peers { permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36725 } }
I20250625 13:57:23.478267 14589 raft_consensus.cc:383] T cdd115530ef34645968765005dd3caf3 P 4b149ff2436c4feeb6efd63182aa92a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:57:23.478621 14589 raft_consensus.cc:738] T cdd115530ef34645968765005dd3caf3 P 4b149ff2436c4feeb6efd63182aa92a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b149ff2436c4feeb6efd63182aa92a4, State: Initialized, Role: FOLLOWER
I20250625 13:57:23.479552 14589 consensus_queue.cc:260] T cdd115530ef34645968765005dd3caf3 P 4b149ff2436c4feeb6efd63182aa92a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b149ff2436c4feeb6efd63182aa92a4" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 46151 } } peers { permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 } } peers { permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36725 } }
I20250625 13:57:23.485157 14589 ts_tablet_manager.cc:1428] T cdd115530ef34645968765005dd3caf3 P 4b149ff2436c4feeb6efd63182aa92a4: Time spent starting tablet: real 0.031s user 0.016s sys 0.014s
I20250625 13:57:23.491281 14590 raft_consensus.cc:357] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b149ff2436c4feeb6efd63182aa92a4" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 46151 } } peers { permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 } } peers { permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36725 } }
I20250625 13:57:23.492204 14590 raft_consensus.cc:383] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:57:23.492551 14590 raft_consensus.cc:738] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 245f46b721f44c14a4aad846784626d6, State: Initialized, Role: FOLLOWER
I20250625 13:57:23.493427 14590 consensus_queue.cc:260] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b149ff2436c4feeb6efd63182aa92a4" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 46151 } } peers { permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 } } peers { permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36725 } }
I20250625 13:57:23.497715 14590 ts_tablet_manager.cc:1428] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6: Time spent starting tablet: real 0.034s user 0.029s sys 0.004s
I20250625 13:57:23.497967 14592 raft_consensus.cc:357] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b149ff2436c4feeb6efd63182aa92a4" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 46151 } } peers { permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 } } peers { permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36725 } }
I20250625 13:57:23.498704 14592 raft_consensus.cc:383] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:57:23.498942 14592 raft_consensus.cc:738] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa46df1c4e334e9c9e12db8c48ce2249, State: Initialized, Role: FOLLOWER
I20250625 13:57:23.499617 14592 consensus_queue.cc:260] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b149ff2436c4feeb6efd63182aa92a4" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 46151 } } peers { permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 } } peers { permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36725 } }
I20250625 13:57:23.503687 14592 ts_tablet_manager.cc:1428] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249: Time spent starting tablet: real 0.034s user 0.027s sys 0.008s
W20250625 13:57:23.534258 14304 tablet.cc:2378] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 13:57:23.556344 14595 raft_consensus.cc:491] T cdd115530ef34645968765005dd3caf3 P 4b149ff2436c4feeb6efd63182aa92a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:57:23.556879 14595 raft_consensus.cc:513] T cdd115530ef34645968765005dd3caf3 P 4b149ff2436c4feeb6efd63182aa92a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b149ff2436c4feeb6efd63182aa92a4" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 46151 } } peers { permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 } } peers { permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36725 } }
I20250625 13:57:23.560040 14595 leader_election.cc:290] T cdd115530ef34645968765005dd3caf3 P 4b149ff2436c4feeb6efd63182aa92a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 245f46b721f44c14a4aad846784626d6 (127.13.155.194:35519), fa46df1c4e334e9c9e12db8c48ce2249 (127.13.155.195:36725)
I20250625 13:57:23.571166 14258 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdd115530ef34645968765005dd3caf3" candidate_uuid: "4b149ff2436c4feeb6efd63182aa92a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "245f46b721f44c14a4aad846784626d6" is_pre_election: true
I20250625 13:57:23.572053 14258 raft_consensus.cc:2466] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b149ff2436c4feeb6efd63182aa92a4 in term 0.
I20250625 13:57:23.572492 14391 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdd115530ef34645968765005dd3caf3" candidate_uuid: "4b149ff2436c4feeb6efd63182aa92a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" is_pre_election: true
I20250625 13:57:23.573158 14058 leader_election.cc:304] T cdd115530ef34645968765005dd3caf3 P 4b149ff2436c4feeb6efd63182aa92a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 245f46b721f44c14a4aad846784626d6, 4b149ff2436c4feeb6efd63182aa92a4; no voters:
I20250625 13:57:23.573257 14391 raft_consensus.cc:2466] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b149ff2436c4feeb6efd63182aa92a4 in term 0.
I20250625 13:57:23.574355 14595 raft_consensus.cc:2802] T cdd115530ef34645968765005dd3caf3 P 4b149ff2436c4feeb6efd63182aa92a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:57:23.574644 14595 raft_consensus.cc:491] T cdd115530ef34645968765005dd3caf3 P 4b149ff2436c4feeb6efd63182aa92a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:57:23.574910 14595 raft_consensus.cc:3058] T cdd115530ef34645968765005dd3caf3 P 4b149ff2436c4feeb6efd63182aa92a4 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:57:23.578965 14595 raft_consensus.cc:513] T cdd115530ef34645968765005dd3caf3 P 4b149ff2436c4feeb6efd63182aa92a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b149ff2436c4feeb6efd63182aa92a4" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 46151 } } peers { permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 } } peers { permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36725 } }
I20250625 13:57:23.580600 14595 leader_election.cc:290] T cdd115530ef34645968765005dd3caf3 P 4b149ff2436c4feeb6efd63182aa92a4 [CANDIDATE]: Term 1 election: Requested vote from peers 245f46b721f44c14a4aad846784626d6 (127.13.155.194:35519), fa46df1c4e334e9c9e12db8c48ce2249 (127.13.155.195:36725)
I20250625 13:57:23.581393 14258 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdd115530ef34645968765005dd3caf3" candidate_uuid: "4b149ff2436c4feeb6efd63182aa92a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "245f46b721f44c14a4aad846784626d6"
I20250625 13:57:23.581456 14391 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdd115530ef34645968765005dd3caf3" candidate_uuid: "4b149ff2436c4feeb6efd63182aa92a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa46df1c4e334e9c9e12db8c48ce2249"
I20250625 13:57:23.581920 14391 raft_consensus.cc:3058] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:57:23.581933 14258 raft_consensus.cc:3058] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:57:23.588984 14391 raft_consensus.cc:2466] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b149ff2436c4feeb6efd63182aa92a4 in term 1.
I20250625 13:57:23.589136 14258 raft_consensus.cc:2466] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b149ff2436c4feeb6efd63182aa92a4 in term 1.
I20250625 13:57:23.590049 14059 leader_election.cc:304] T cdd115530ef34645968765005dd3caf3 P 4b149ff2436c4feeb6efd63182aa92a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b149ff2436c4feeb6efd63182aa92a4, fa46df1c4e334e9c9e12db8c48ce2249; no voters:
I20250625 13:57:23.590704 14595 raft_consensus.cc:2802] T cdd115530ef34645968765005dd3caf3 P 4b149ff2436c4feeb6efd63182aa92a4 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:57:23.592590 14595 raft_consensus.cc:695] T cdd115530ef34645968765005dd3caf3 P 4b149ff2436c4feeb6efd63182aa92a4 [term 1 LEADER]: Becoming Leader. State: Replica: 4b149ff2436c4feeb6efd63182aa92a4, State: Running, Role: LEADER
I20250625 13:57:23.593300 14595 consensus_queue.cc:237] T cdd115530ef34645968765005dd3caf3 P 4b149ff2436c4feeb6efd63182aa92a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b149ff2436c4feeb6efd63182aa92a4" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 46151 } } peers { permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 } } peers { permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36725 } }
I20250625 13:57:23.605165 13981 catalog_manager.cc:5582] T cdd115530ef34645968765005dd3caf3 P 4b149ff2436c4feeb6efd63182aa92a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b149ff2436c4feeb6efd63182aa92a4 (127.13.155.193). New cstate: current_term: 1 leader_uuid: "4b149ff2436c4feeb6efd63182aa92a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b149ff2436c4feeb6efd63182aa92a4" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 46151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36725 } health_report { overall_health: UNKNOWN } } }
W20250625 13:57:23.622715 14437 tablet.cc:2378] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 13:57:23.653020 13935 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250625 13:57:23.656563 13935 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 4b149ff2436c4feeb6efd63182aa92a4 to finish bootstrapping
I20250625 13:57:23.670751 13935 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 245f46b721f44c14a4aad846784626d6 to finish bootstrapping
I20250625 13:57:23.681469 13935 ts_itest-base.cc:246] Waiting for 1 tablets on tserver fa46df1c4e334e9c9e12db8c48ce2249 to finish bootstrapping
I20250625 13:57:23.692669 13935 kudu-admin-test.cc:709] Waiting for Master to see the current replicas...
W20250625 13:57:23.707142 14171 tablet.cc:2378] T cdd115530ef34645968765005dd3caf3 P 4b149ff2436c4feeb6efd63182aa92a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 13:57:23.696130 13935 kudu-admin-test.cc:716] Tablet locations:
tablet_locations {
tablet_id: "cdd115530ef34645968765005dd3caf3"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: LEADER
}
interned_replicas {
ts_info_idx: 1
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "4b149ff2436c4feeb6efd63182aa92a4"
rpc_addresses {
host: "127.13.155.193"
port: 46151
}
}
ts_infos {
permanent_uuid: "245f46b721f44c14a4aad846784626d6"
rpc_addresses {
host: "127.13.155.194"
port: 35519
}
}
ts_infos {
permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249"
rpc_addresses {
host: "127.13.155.195"
port: 36725
}
}
I20250625 13:57:23.975158 14595 consensus_queue.cc:1035] T cdd115530ef34645968765005dd3caf3 P 4b149ff2436c4feeb6efd63182aa92a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:57:23.991053 14600 consensus_queue.cc:1035] T cdd115530ef34645968765005dd3caf3 P 4b149ff2436c4feeb6efd63182aa92a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:57:23.998901 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 14041
W20250625 13:57:24.028172 14194 connection.cc:537] server connection from 127.13.155.193:35079 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250625 13:57:24.029361 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 13949
I20250625 13:57:24.058650 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.155.254:34591
--webserver_interface=127.13.155.254
--webserver_port=35059
--builtin_ntp_servers=127.13.155.212:33671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.155.254:34591 with env {}
I20250625 13:57:24.267352 14570 heartbeater.cc:499] Master 127.13.155.254:34591 was elected leader, sending a full tablet report...
W20250625 13:57:24.271044 14570 heartbeater.cc:646] Failed to heartbeat to 127.13.155.254:34591 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.155.254:34591: connect: Connection refused (error 111)
W20250625 13:57:24.355723 14610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:57:24.356309 14610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:57:24.356717 14610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:57:24.389281 14610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 13:57:24.389601 14610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:57:24.389860 14610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 13:57:24.390087 14610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 13:57:24.424782 14610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:33671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.155.254:34591
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.155.254:34591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.13.155.254
--webserver_port=35059
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:57:24.426119 14610 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:57:24.427745 14610 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:57:24.443130 14617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:57:25.007931 14436 heartbeater.cc:646] Failed to heartbeat to 127.13.155.254:34591 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.155.254:34591: connect: Connection refused (error 111)
W20250625 13:57:25.018414 14303 heartbeater.cc:646] Failed to heartbeat to 127.13.155.254:34591 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.155.254:34591: connect: Connection refused (error 111)
I20250625 13:57:25.488296 14626 raft_consensus.cc:491] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b149ff2436c4feeb6efd63182aa92a4)
I20250625 13:57:25.488684 14626 raft_consensus.cc:513] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b149ff2436c4feeb6efd63182aa92a4" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 46151 } } peers { permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 } } peers { permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36725 } }
I20250625 13:57:25.490900 14626 leader_election.cc:290] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b149ff2436c4feeb6efd63182aa92a4 (127.13.155.193:46151), 245f46b721f44c14a4aad846784626d6 (127.13.155.194:35519)
W20250625 13:57:25.494083 14325 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.155.193:46151: connect: Connection refused (error 111)
W20250625 13:57:25.508867 14325 leader_election.cc:336] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4b149ff2436c4feeb6efd63182aa92a4 (127.13.155.193:46151): Network error: Client connection negotiation failed: client connection to 127.13.155.193:46151: connect: Connection refused (error 111)
I20250625 13:57:25.508980 14630 raft_consensus.cc:491] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b149ff2436c4feeb6efd63182aa92a4)
I20250625 13:57:25.509373 14630 raft_consensus.cc:513] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b149ff2436c4feeb6efd63182aa92a4" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 46151 } } peers { permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 } } peers { permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36725 } }
I20250625 13:57:25.511730 14630 leader_election.cc:290] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b149ff2436c4feeb6efd63182aa92a4 (127.13.155.193:46151), fa46df1c4e334e9c9e12db8c48ce2249 (127.13.155.195:36725)
W20250625 13:57:25.516131 14192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.155.193:46151: connect: Connection refused (error 111)
I20250625 13:57:25.517571 14258 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdd115530ef34645968765005dd3caf3" candidate_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "245f46b721f44c14a4aad846784626d6" is_pre_election: true
I20250625 13:57:25.518339 14258 raft_consensus.cc:2466] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa46df1c4e334e9c9e12db8c48ce2249 in term 1.
I20250625 13:57:25.520239 14324 leader_election.cc:304] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 245f46b721f44c14a4aad846784626d6, fa46df1c4e334e9c9e12db8c48ce2249; no voters: 4b149ff2436c4feeb6efd63182aa92a4
I20250625 13:57:25.522668 14626 raft_consensus.cc:2802] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250625 13:57:25.522994 14626 raft_consensus.cc:491] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4b149ff2436c4feeb6efd63182aa92a4)
I20250625 13:57:25.523301 14626 raft_consensus.cc:3058] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [term 1 FOLLOWER]: Advancing to term 2
W20250625 13:57:25.529526 14192 leader_election.cc:336] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4b149ff2436c4feeb6efd63182aa92a4 (127.13.155.193:46151): Network error: Client connection negotiation failed: client connection to 127.13.155.193:46151: connect: Connection refused (error 111)
I20250625 13:57:25.529726 14626 raft_consensus.cc:513] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b149ff2436c4feeb6efd63182aa92a4" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 46151 } } peers { permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 } } peers { permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36725 } }
I20250625 13:57:25.530826 14391 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdd115530ef34645968765005dd3caf3" candidate_uuid: "245f46b721f44c14a4aad846784626d6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" is_pre_election: true
I20250625 13:57:25.531433 14391 raft_consensus.cc:2391] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 245f46b721f44c14a4aad846784626d6 in current term 2: Already voted for candidate fa46df1c4e334e9c9e12db8c48ce2249 in this term.
I20250625 13:57:25.532611 14192 leader_election.cc:304] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 245f46b721f44c14a4aad846784626d6; no voters: 4b149ff2436c4feeb6efd63182aa92a4, fa46df1c4e334e9c9e12db8c48ce2249
I20250625 13:57:25.533401 14630 raft_consensus.cc:3058] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6 [term 1 FOLLOWER]: Advancing to term 2
W20250625 13:57:25.535503 14325 leader_election.cc:336] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4b149ff2436c4feeb6efd63182aa92a4 (127.13.155.193:46151): Network error: Client connection negotiation failed: client connection to 127.13.155.193:46151: connect: Connection refused (error 111)
I20250625 13:57:25.536303 14626 leader_election.cc:290] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [CANDIDATE]: Term 2 election: Requested vote from peers 4b149ff2436c4feeb6efd63182aa92a4 (127.13.155.193:46151), 245f46b721f44c14a4aad846784626d6 (127.13.155.194:35519)
I20250625 13:57:25.537856 14258 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdd115530ef34645968765005dd3caf3" candidate_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "245f46b721f44c14a4aad846784626d6"
I20250625 13:57:25.539939 14630 raft_consensus.cc:2747] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 13:57:25.546639 14258 raft_consensus.cc:2466] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa46df1c4e334e9c9e12db8c48ce2249 in term 2.
I20250625 13:57:25.550014 14324 leader_election.cc:304] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 245f46b721f44c14a4aad846784626d6, fa46df1c4e334e9c9e12db8c48ce2249; no voters: 4b149ff2436c4feeb6efd63182aa92a4
I20250625 13:57:25.550830 14626 raft_consensus.cc:2802] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [term 2 FOLLOWER]: Leader election won for term 2
I20250625 13:57:25.552805 14626 raft_consensus.cc:695] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [term 2 LEADER]: Becoming Leader. State: Replica: fa46df1c4e334e9c9e12db8c48ce2249, State: Running, Role: LEADER
I20250625 13:57:25.553681 14626 consensus_queue.cc:237] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b149ff2436c4feeb6efd63182aa92a4" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 46151 } } peers { permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 } } peers { permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36725 } }
W20250625 13:57:24.443161 14620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:57:24.443816 14618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:57:24.445369 14610 server_base.cc:1048] running on GCE node
I20250625 13:57:25.657918 14610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:57:25.660486 14610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:57:25.661831 14610 hybrid_clock.cc:648] HybridClock initialized: now 1750859845661789 us; error 58 us; skew 500 ppm
I20250625 13:57:25.662653 14610 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:57:25.672755 14610 webserver.cc:469] Webserver started at http://127.13.155.254:35059/ using document root <none> and password file <none>
I20250625 13:57:25.673790 14610 fs_manager.cc:362] Metadata directory not provided
I20250625 13:57:25.674017 14610 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:57:25.681664 14610 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20250625 13:57:25.686050 14640 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:25.687081 14610 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250625 13:57:25.687394 14610 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f536eb4978b84fa287cd63c9d3878c6a"
format_stamp: "Formatted at 2025-06-25 13:57:15 on dist-test-slave-k028"
I20250625 13:57:25.689392 14610 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:57:25.748613 14610 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:57:25.750115 14610 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:57:25.750557 14610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:57:25.823890 14610 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.254:34591
I20250625 13:57:25.823976 14691 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.254:34591 every 8 connection(s)
I20250625 13:57:25.826777 14610 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250625 13:57:25.832469 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 14610
I20250625 13:57:25.833073 13935 kudu-admin-test.cc:735] Forcing unsafe config change on tserver fa46df1c4e334e9c9e12db8c48ce2249
I20250625 13:57:25.838909 14692 sys_catalog.cc:263] Verifying existing consensus state
I20250625 13:57:25.844897 14692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a: Bootstrap starting.
I20250625 13:57:25.882454 14692 log.cc:826] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a: Log is configured to *not* fsync() on all Append() calls
I20250625 13:57:25.904623 14692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 13:57:25.905419 14692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a: Bootstrap complete.
I20250625 13:57:25.926412 14692 raft_consensus.cc:357] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f536eb4978b84fa287cd63c9d3878c6a" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 34591 } }
I20250625 13:57:25.928514 14692 raft_consensus.cc:738] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f536eb4978b84fa287cd63c9d3878c6a, State: Initialized, Role: FOLLOWER
I20250625 13:57:25.929380 14692 consensus_queue.cc:260] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f536eb4978b84fa287cd63c9d3878c6a" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 34591 } }
I20250625 13:57:25.929873 14692 raft_consensus.cc:397] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 13:57:25.930145 14692 raft_consensus.cc:491] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 13:57:25.930444 14692 raft_consensus.cc:3058] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a [term 1 FOLLOWER]: Advancing to term 2
I20250625 13:57:25.936235 14692 raft_consensus.cc:513] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f536eb4978b84fa287cd63c9d3878c6a" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 34591 } }
I20250625 13:57:25.936951 14692 leader_election.cc:304] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f536eb4978b84fa287cd63c9d3878c6a; no voters:
I20250625 13:57:25.939492 14692 leader_election.cc:290] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a [CANDIDATE]: Term 2 election: Requested vote from peers
I20250625 13:57:25.940065 14696 raft_consensus.cc:2802] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a [term 2 FOLLOWER]: Leader election won for term 2
I20250625 13:57:25.943428 14696 raft_consensus.cc:695] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a [term 2 LEADER]: Becoming Leader. State: Replica: f536eb4978b84fa287cd63c9d3878c6a, State: Running, Role: LEADER
I20250625 13:57:25.944352 14696 consensus_queue.cc:237] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f536eb4978b84fa287cd63c9d3878c6a" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 34591 } }
I20250625 13:57:25.945096 14692 sys_catalog.cc:564] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a [sys.catalog]: configured and running, proceeding with master startup.
I20250625 13:57:25.954751 14697 sys_catalog.cc:455] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f536eb4978b84fa287cd63c9d3878c6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f536eb4978b84fa287cd63c9d3878c6a" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 34591 } } }
I20250625 13:57:25.955605 14697 sys_catalog.cc:458] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a [sys.catalog]: This master's current role is: LEADER
I20250625 13:57:25.957828 14698 sys_catalog.cc:455] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f536eb4978b84fa287cd63c9d3878c6a. Latest consensus state: current_term: 2 leader_uuid: "f536eb4978b84fa287cd63c9d3878c6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f536eb4978b84fa287cd63c9d3878c6a" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 34591 } } }
I20250625 13:57:25.958545 14698 sys_catalog.cc:458] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a [sys.catalog]: This master's current role is: LEADER
I20250625 13:57:25.965586 14703 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 13:57:25.978745 14703 catalog_manager.cc:671] Loaded metadata for table TestTable [id=6548b90a615c4e37a94459d5aa3733b2]
I20250625 13:57:25.987174 14703 tablet_loader.cc:96] loaded metadata for tablet cdd115530ef34645968765005dd3caf3 (table TestTable [id=6548b90a615c4e37a94459d5aa3733b2])
I20250625 13:57:25.988868 14703 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 13:57:25.993921 14703 catalog_manager.cc:1261] Loaded cluster ID: 6f7fbade18654200947085c151ba07e0
I20250625 13:57:25.994309 14703 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 13:57:26.002346 14703 catalog_manager.cc:1506] Loading token signing keys...
I20250625 13:57:26.007614 14703 catalog_manager.cc:5966] T 00000000000000000000000000000000 P f536eb4978b84fa287cd63c9d3878c6a: Loaded TSK: 0
I20250625 13:57:26.009948 14703 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 13:57:26.025930 14713 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250625 13:57:26.036638 14303 heartbeater.cc:344] Connected to a master server at 127.13.155.254:34591
I20250625 13:57:26.041551 14657 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "245f46b721f44c14a4aad846784626d6" instance_seqno: 1750859839488423) as {username='slave'} at 127.13.155.194:52545; Asking this server to re-register.
I20250625 13:57:26.043275 14303 heartbeater.cc:461] Registering TS with master...
I20250625 13:57:26.043802 14303 heartbeater.cc:507] Master 127.13.155.254:34591 requested a full tablet report, sending...
I20250625 13:57:26.046514 14657 ts_manager.cc:194] Registered new tserver with Master: 245f46b721f44c14a4aad846784626d6 (127.13.155.194:35519)
W20250625 13:57:26.068958 14325 consensus_peers.cc:487] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 -> Peer 4b149ff2436c4feeb6efd63182aa92a4 (127.13.155.193:46151): Couldn't send request to peer 4b149ff2436c4feeb6efd63182aa92a4. Status: Network error: Client connection negotiation failed: client connection to 127.13.155.193:46151: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250625 13:57:26.150955 14258 raft_consensus.cc:1273] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6 [term 2 FOLLOWER]: Refusing update from remote peer fa46df1c4e334e9c9e12db8c48ce2249: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250625 13:57:26.152439 14715 consensus_queue.cc:1035] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [LEADER]: Connected to new peer: Peer: permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250625 13:57:26.160998 14657 catalog_manager.cc:5582] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6 reported cstate change: term changed from 1 to 2, leader changed from 4b149ff2436c4feeb6efd63182aa92a4 (127.13.155.193) to fa46df1c4e334e9c9e12db8c48ce2249 (127.13.155.195). New cstate: current_term: 2 leader_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b149ff2436c4feeb6efd63182aa92a4" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 46151 } } peers { permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 } } peers { permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36725 } } }
I20250625 13:57:26.205448 14436 heartbeater.cc:344] Connected to a master server at 127.13.155.254:34591
I20250625 13:57:26.210309 14656 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" instance_seqno: 1750859841319854) as {username='slave'} at 127.13.155.195:34391; Asking this server to re-register.
I20250625 13:57:26.212201 14436 heartbeater.cc:461] Registering TS with master...
I20250625 13:57:26.212904 14436 heartbeater.cc:507] Master 127.13.155.254:34591 requested a full tablet report, sending...
I20250625 13:57:26.217842 14656 ts_manager.cc:194] Registered new tserver with Master: fa46df1c4e334e9c9e12db8c48ce2249 (127.13.155.195:36725)
W20250625 13:57:26.271934 14694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:57:26.272567 14694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250625 13:57:26.288605 14570 heartbeater.cc:344] Connected to a master server at 127.13.155.254:34591
I20250625 13:57:26.289126 14570 heartbeater.cc:499] Master 127.13.155.254:34591 was elected leader, sending a full tablet report...
I20250625 13:57:26.291036 14656 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "69298260f4934e35a7f68c21c2205139" instance_seqno: 1750859843199112) as {username='slave'} at 127.13.155.196:39111; Asking this server to re-register.
I20250625 13:57:26.292906 14570 heartbeater.cc:461] Registering TS with master...
I20250625 13:57:26.293480 14570 heartbeater.cc:507] Master 127.13.155.254:34591 requested a full tablet report, sending...
I20250625 13:57:26.295536 14656 ts_manager.cc:194] Registered new tserver with Master: 69298260f4934e35a7f68c21c2205139 (127.13.155.196:39099)
W20250625 13:57:26.310593 14694 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250625 13:57:27.757249 14729 debug-util.cc:398] Leaking SignalData structure 0x7b0800036040 after lost signal to thread 14694
W20250625 13:57:27.757766 14729 kernel_stack_watchdog.cc:198] Thread 14694 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:641 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250625 13:57:27.975378 14694 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.622s user 0.591s sys 1.019s
W20250625 13:57:28.096063 14694 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.742s user 0.595s sys 1.027s
I20250625 13:57:28.133817 14391 tablet_service.cc:1905] Received UnsafeChangeConfig RPC: dest_uuid: "fa46df1c4e334e9c9e12db8c48ce2249"
tablet_id: "cdd115530ef34645968765005dd3caf3"
caller_id: "kudu-tools"
new_config {
peers {
permanent_uuid: "245f46b721f44c14a4aad846784626d6"
}
peers {
permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249"
}
}
from {username='slave'} at 127.0.0.1:52264
W20250625 13:57:28.134855 14391 raft_consensus.cc:2216] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [term 2 LEADER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b149ff2436c4feeb6efd63182aa92a4" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 46151 } } peers { permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 } } peers { permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36725 } }NEW CONFIG: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 } } peers { permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36725 } } unsafe_config_change: true
I20250625 13:57:28.135639 14391 raft_consensus.cc:3053] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [term 2 LEADER]: Stepping down as leader of term 2
I20250625 13:57:28.135860 14391 raft_consensus.cc:738] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fa46df1c4e334e9c9e12db8c48ce2249, State: Running, Role: LEADER
I20250625 13:57:28.136518 14391 consensus_queue.cc:260] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b149ff2436c4feeb6efd63182aa92a4" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 46151 } } peers { permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 } } peers { permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36725 } }
I20250625 13:57:28.137404 14391 raft_consensus.cc:3058] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [term 2 FOLLOWER]: Advancing to term 3
W20250625 13:57:28.959374 14729 debug-util.cc:398] Leaking SignalData structure 0x7b0800036060 after lost signal to thread 14694
I20250625 13:57:29.568245 14754 raft_consensus.cc:491] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fa46df1c4e334e9c9e12db8c48ce2249)
I20250625 13:57:29.568783 14754 raft_consensus.cc:513] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b149ff2436c4feeb6efd63182aa92a4" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 46151 } } peers { permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 } } peers { permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36725 } }
I20250625 13:57:29.570271 14754 leader_election.cc:290] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4b149ff2436c4feeb6efd63182aa92a4 (127.13.155.193:46151), fa46df1c4e334e9c9e12db8c48ce2249 (127.13.155.195:36725)
I20250625 13:57:29.572619 14391 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdd115530ef34645968765005dd3caf3" candidate_uuid: "245f46b721f44c14a4aad846784626d6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" is_pre_election: true
W20250625 13:57:29.575634 14192 leader_election.cc:336] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4b149ff2436c4feeb6efd63182aa92a4 (127.13.155.193:46151): Network error: Client connection negotiation failed: client connection to 127.13.155.193:46151: connect: Connection refused (error 111)
I20250625 13:57:29.575950 14192 leader_election.cc:304] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 245f46b721f44c14a4aad846784626d6; no voters: 4b149ff2436c4feeb6efd63182aa92a4, fa46df1c4e334e9c9e12db8c48ce2249
I20250625 13:57:29.576472 14754 raft_consensus.cc:2747] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250625 13:57:29.643504 14757 raft_consensus.cc:491] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools)
I20250625 13:57:29.643899 14757 raft_consensus.cc:513] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 } } peers { permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36725 } } unsafe_config_change: true
I20250625 13:57:29.644932 14757 leader_election.cc:290] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 245f46b721f44c14a4aad846784626d6 (127.13.155.194:35519)
I20250625 13:57:29.646029 14258 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdd115530ef34645968765005dd3caf3" candidate_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "245f46b721f44c14a4aad846784626d6" is_pre_election: true
I20250625 13:57:29.646641 14258 raft_consensus.cc:2466] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa46df1c4e334e9c9e12db8c48ce2249 in term 2.
I20250625 13:57:29.647646 14324 leader_election.cc:304] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 245f46b721f44c14a4aad846784626d6, fa46df1c4e334e9c9e12db8c48ce2249; no voters:
I20250625 13:57:29.648195 14757 raft_consensus.cc:2802] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250625 13:57:29.648461 14757 raft_consensus.cc:491] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [term 3 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools)
I20250625 13:57:29.648685 14757 raft_consensus.cc:3058] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [term 3 FOLLOWER]: Advancing to term 4
I20250625 13:57:29.652657 14757 raft_consensus.cc:513] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 } } peers { permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36725 } } unsafe_config_change: true
I20250625 13:57:29.653575 14757 leader_election.cc:290] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [CANDIDATE]: Term 4 election: Requested vote from peers 245f46b721f44c14a4aad846784626d6 (127.13.155.194:35519)
I20250625 13:57:29.654588 14258 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdd115530ef34645968765005dd3caf3" candidate_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "245f46b721f44c14a4aad846784626d6"
I20250625 13:57:29.654975 14258 raft_consensus.cc:3058] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6 [term 2 FOLLOWER]: Advancing to term 4
I20250625 13:57:29.659420 14258 raft_consensus.cc:2466] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa46df1c4e334e9c9e12db8c48ce2249 in term 4.
I20250625 13:57:29.660240 14324 leader_election.cc:304] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 245f46b721f44c14a4aad846784626d6, fa46df1c4e334e9c9e12db8c48ce2249; no voters:
I20250625 13:57:29.660792 14757 raft_consensus.cc:2802] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [term 4 FOLLOWER]: Leader election won for term 4
I20250625 13:57:29.661517 14757 raft_consensus.cc:695] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [term 4 LEADER]: Becoming Leader. State: Replica: fa46df1c4e334e9c9e12db8c48ce2249, State: Running, Role: LEADER
I20250625 13:57:29.662065 14757 consensus_queue.cc:237] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 } } peers { permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36725 } } unsafe_config_change: true
I20250625 13:57:29.667291 14656 catalog_manager.cc:5582] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 reported cstate change: term changed from 2 to 4, now has a pending config: VOTER 245f46b721f44c14a4aad846784626d6 (127.13.155.194), VOTER fa46df1c4e334e9c9e12db8c48ce2249 (127.13.155.195). New cstate: current_term: 4 leader_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b149ff2436c4feeb6efd63182aa92a4" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 46151 } } peers { permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36725 } health_report { overall_health: HEALTHY } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 } } peers { permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36725 } } unsafe_config_change: true }
I20250625 13:57:30.134634 14258 raft_consensus.cc:1273] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6 [term 4 FOLLOWER]: Refusing update from remote peer fa46df1c4e334e9c9e12db8c48ce2249: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250625 13:57:30.135936 14757 consensus_queue.cc:1035] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [LEADER]: Connected to new peer: Peer: permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20250625 13:57:30.143851 14758 raft_consensus.cc:2953] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [term 4 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER 4b149ff2436c4feeb6efd63182aa92a4 (127.13.155.193) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 } } peers { permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36725 } } unsafe_config_change: true }
I20250625 13:57:30.144922 14258 raft_consensus.cc:2953] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6 [term 4 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER 4b149ff2436c4feeb6efd63182aa92a4 (127.13.155.193) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 } } peers { permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36725 } } unsafe_config_change: true }
I20250625 13:57:30.155124 14656 catalog_manager.cc:5582] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6 reported cstate change: config changed from index -1 to 3, VOTER 4b149ff2436c4feeb6efd63182aa92a4 (127.13.155.193) evicted, no longer has a pending config: VOTER 245f46b721f44c14a4aad846784626d6 (127.13.155.194), VOTER fa46df1c4e334e9c9e12db8c48ce2249 (127.13.155.195). New cstate: current_term: 4 leader_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 } } peers { permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36725 } } unsafe_config_change: true }
W20250625 13:57:30.162941 14656 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet cdd115530ef34645968765005dd3caf3 on TS 4b149ff2436c4feeb6efd63182aa92a4: Not found: failed to reset TS proxy: Could not find TS for UUID 4b149ff2436c4feeb6efd63182aa92a4
I20250625 13:57:30.178753 14391 consensus_queue.cc:237] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 } } peers { permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36725 } } peers { permanent_uuid: "69298260f4934e35a7f68c21c2205139" member_type: NON_VOTER last_known_addr { host: "127.13.155.196" port: 39099 } attrs { promote: true } } unsafe_config_change: true
I20250625 13:57:30.185669 14258 raft_consensus.cc:1273] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6 [term 4 FOLLOWER]: Refusing update from remote peer fa46df1c4e334e9c9e12db8c48ce2249: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250625 13:57:30.186902 14757 consensus_queue.cc:1035] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [LEADER]: Connected to new peer: Peer: permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250625 13:57:30.192359 14758 raft_consensus.cc:2953] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER 69298260f4934e35a7f68c21c2205139 (127.13.155.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 } } peers { permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36725 } } peers { permanent_uuid: "69298260f4934e35a7f68c21c2205139" member_type: NON_VOTER last_known_addr { host: "127.13.155.196" port: 39099 } attrs { promote: true } } unsafe_config_change: true }
I20250625 13:57:30.193647 14258 raft_consensus.cc:2953] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER 69298260f4934e35a7f68c21c2205139 (127.13.155.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 } } peers { permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36725 } } peers { permanent_uuid: "69298260f4934e35a7f68c21c2205139" member_type: NON_VOTER last_known_addr { host: "127.13.155.196" port: 39099 } attrs { promote: true } } unsafe_config_change: true }
W20250625 13:57:30.195593 14324 consensus_peers.cc:487] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 -> Peer 69298260f4934e35a7f68c21c2205139 (127.13.155.196:39099): Couldn't send request to peer 69298260f4934e35a7f68c21c2205139. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cdd115530ef34645968765005dd3caf3. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:57:30.199146 14642 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet cdd115530ef34645968765005dd3caf3 with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250625 13:57:30.202033 14657 catalog_manager.cc:5582] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 reported cstate change: config changed from index 3 to 5, NON_VOTER 69298260f4934e35a7f68c21c2205139 (127.13.155.196) added. New cstate: current_term: 4 leader_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69298260f4934e35a7f68c21c2205139" member_type: NON_VOTER last_known_addr { host: "127.13.155.196" port: 39099 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true }
W20250625 13:57:30.215734 14642 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet cdd115530ef34645968765005dd3caf3 on TS 4b149ff2436c4feeb6efd63182aa92a4 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 4b149ff2436c4feeb6efd63182aa92a4
I20250625 13:57:30.684839 14774 ts_tablet_manager.cc:927] T cdd115530ef34645968765005dd3caf3 P 69298260f4934e35a7f68c21c2205139: Initiating tablet copy from peer fa46df1c4e334e9c9e12db8c48ce2249 (127.13.155.195:36725)
I20250625 13:57:30.687088 14774 tablet_copy_client.cc:323] T cdd115530ef34645968765005dd3caf3 P 69298260f4934e35a7f68c21c2205139: tablet copy: Beginning tablet copy session from remote peer at address 127.13.155.195:36725
I20250625 13:57:30.697546 14411 tablet_copy_service.cc:140] P fa46df1c4e334e9c9e12db8c48ce2249: Received BeginTabletCopySession request for tablet cdd115530ef34645968765005dd3caf3 from peer 69298260f4934e35a7f68c21c2205139 ({username='slave'} at 127.13.155.196:40463)
I20250625 13:57:30.697984 14411 tablet_copy_service.cc:161] P fa46df1c4e334e9c9e12db8c48ce2249: Beginning new tablet copy session on tablet cdd115530ef34645968765005dd3caf3 from peer 69298260f4934e35a7f68c21c2205139 at {username='slave'} at 127.13.155.196:40463: session id = 69298260f4934e35a7f68c21c2205139-cdd115530ef34645968765005dd3caf3
I20250625 13:57:30.703011 14411 tablet_copy_source_session.cc:215] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249: Tablet Copy: opened 0 blocks and 1 log segments
I20250625 13:57:30.707628 14774 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdd115530ef34645968765005dd3caf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:57:30.724589 14774 tablet_copy_client.cc:806] T cdd115530ef34645968765005dd3caf3 P 69298260f4934e35a7f68c21c2205139: tablet copy: Starting download of 0 data blocks...
I20250625 13:57:30.725123 14774 tablet_copy_client.cc:670] T cdd115530ef34645968765005dd3caf3 P 69298260f4934e35a7f68c21c2205139: tablet copy: Starting download of 1 WAL segments...
I20250625 13:57:30.728449 14774 tablet_copy_client.cc:538] T cdd115530ef34645968765005dd3caf3 P 69298260f4934e35a7f68c21c2205139: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250625 13:57:30.734303 14774 tablet_bootstrap.cc:492] T cdd115530ef34645968765005dd3caf3 P 69298260f4934e35a7f68c21c2205139: Bootstrap starting.
I20250625 13:57:30.745527 14774 log.cc:826] T cdd115530ef34645968765005dd3caf3 P 69298260f4934e35a7f68c21c2205139: Log is configured to *not* fsync() on all Append() calls
I20250625 13:57:30.756031 14774 tablet_bootstrap.cc:492] T cdd115530ef34645968765005dd3caf3 P 69298260f4934e35a7f68c21c2205139: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 13:57:30.756719 14774 tablet_bootstrap.cc:492] T cdd115530ef34645968765005dd3caf3 P 69298260f4934e35a7f68c21c2205139: Bootstrap complete.
I20250625 13:57:30.757290 14774 ts_tablet_manager.cc:1397] T cdd115530ef34645968765005dd3caf3 P 69298260f4934e35a7f68c21c2205139: Time spent bootstrapping tablet: real 0.023s user 0.015s sys 0.008s
I20250625 13:57:30.773416 14774 raft_consensus.cc:357] T cdd115530ef34645968765005dd3caf3 P 69298260f4934e35a7f68c21c2205139 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 } } peers { permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36725 } } peers { permanent_uuid: "69298260f4934e35a7f68c21c2205139" member_type: NON_VOTER last_known_addr { host: "127.13.155.196" port: 39099 } attrs { promote: true } } unsafe_config_change: true
I20250625 13:57:30.774184 14774 raft_consensus.cc:738] T cdd115530ef34645968765005dd3caf3 P 69298260f4934e35a7f68c21c2205139 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 69298260f4934e35a7f68c21c2205139, State: Initialized, Role: LEARNER
I20250625 13:57:30.774835 14774 consensus_queue.cc:260] T cdd115530ef34645968765005dd3caf3 P 69298260f4934e35a7f68c21c2205139 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 } } peers { permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36725 } } peers { permanent_uuid: "69298260f4934e35a7f68c21c2205139" member_type: NON_VOTER last_known_addr { host: "127.13.155.196" port: 39099 } attrs { promote: true } } unsafe_config_change: true
I20250625 13:57:30.778201 14774 ts_tablet_manager.cc:1428] T cdd115530ef34645968765005dd3caf3 P 69298260f4934e35a7f68c21c2205139: Time spent starting tablet: real 0.021s user 0.022s sys 0.000s
I20250625 13:57:30.779866 14411 tablet_copy_service.cc:342] P fa46df1c4e334e9c9e12db8c48ce2249: Request end of tablet copy session 69298260f4934e35a7f68c21c2205139-cdd115530ef34645968765005dd3caf3 received from {username='slave'} at 127.13.155.196:40463
I20250625 13:57:30.780207 14411 tablet_copy_service.cc:434] P fa46df1c4e334e9c9e12db8c48ce2249: ending tablet copy session 69298260f4934e35a7f68c21c2205139-cdd115530ef34645968765005dd3caf3 on tablet cdd115530ef34645968765005dd3caf3 with peer 69298260f4934e35a7f68c21c2205139
I20250625 13:57:31.103453 14525 raft_consensus.cc:1215] T cdd115530ef34645968765005dd3caf3 P 69298260f4934e35a7f68c21c2205139 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.4->[4.5-4.5] Dedup: 4.5->[]
W20250625 13:57:31.382793 14642 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet cdd115530ef34645968765005dd3caf3 on TS 4b149ff2436c4feeb6efd63182aa92a4 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 4b149ff2436c4feeb6efd63182aa92a4
I20250625 13:57:31.558853 14780 raft_consensus.cc:1062] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249: attempting to promote NON_VOTER 69298260f4934e35a7f68c21c2205139 to VOTER
I20250625 13:57:31.560354 14780 consensus_queue.cc:237] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 } } peers { permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36725 } } peers { permanent_uuid: "69298260f4934e35a7f68c21c2205139" member_type: VOTER last_known_addr { host: "127.13.155.196" port: 39099 } attrs { promote: false } } unsafe_config_change: true
I20250625 13:57:31.565045 14525 raft_consensus.cc:1273] T cdd115530ef34645968765005dd3caf3 P 69298260f4934e35a7f68c21c2205139 [term 4 LEARNER]: Refusing update from remote peer fa46df1c4e334e9c9e12db8c48ce2249: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250625 13:57:31.565233 14258 raft_consensus.cc:1273] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6 [term 4 FOLLOWER]: Refusing update from remote peer fa46df1c4e334e9c9e12db8c48ce2249: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250625 13:57:31.566319 14781 consensus_queue.cc:1035] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69298260f4934e35a7f68c21c2205139" member_type: VOTER last_known_addr { host: "127.13.155.196" port: 39099 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250625 13:57:31.567072 14758 consensus_queue.cc:1035] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [LEADER]: Connected to new peer: Peer: permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250625 13:57:31.573196 14781 raft_consensus.cc:2953] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 69298260f4934e35a7f68c21c2205139 (127.13.155.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 } } peers { permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36725 } } peers { permanent_uuid: "69298260f4934e35a7f68c21c2205139" member_type: VOTER last_known_addr { host: "127.13.155.196" port: 39099 } attrs { promote: false } } unsafe_config_change: true }
I20250625 13:57:31.574654 14258 raft_consensus.cc:2953] T cdd115530ef34645968765005dd3caf3 P 245f46b721f44c14a4aad846784626d6 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 69298260f4934e35a7f68c21c2205139 (127.13.155.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 } } peers { permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36725 } } peers { permanent_uuid: "69298260f4934e35a7f68c21c2205139" member_type: VOTER last_known_addr { host: "127.13.155.196" port: 39099 } attrs { promote: false } } unsafe_config_change: true }
I20250625 13:57:31.576550 14525 raft_consensus.cc:2953] T cdd115530ef34645968765005dd3caf3 P 69298260f4934e35a7f68c21c2205139 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 69298260f4934e35a7f68c21c2205139 (127.13.155.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 } } peers { permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36725 } } peers { permanent_uuid: "69298260f4934e35a7f68c21c2205139" member_type: VOTER last_known_addr { host: "127.13.155.196" port: 39099 } attrs { promote: false } } unsafe_config_change: true }
I20250625 13:57:31.584263 14657 catalog_manager.cc:5582] T cdd115530ef34645968765005dd3caf3 P fa46df1c4e334e9c9e12db8c48ce2249 reported cstate change: config changed from index 5 to 6, 69298260f4934e35a7f68c21c2205139 (127.13.155.196) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "245f46b721f44c14a4aad846784626d6" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69298260f4934e35a7f68c21c2205139" member_type: VOTER last_known_addr { host: "127.13.155.196" port: 39099 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
I20250625 13:57:31.665671 13935 kudu-admin-test.cc:751] Waiting for Master to see new config...
I20250625 13:57:31.687639 13935 kudu-admin-test.cc:756] Tablet locations:
tablet_locations {
tablet_id: "cdd115530ef34645968765005dd3caf3"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: LEADER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "245f46b721f44c14a4aad846784626d6"
rpc_addresses {
host: "127.13.155.194"
port: 35519
}
}
ts_infos {
permanent_uuid: "fa46df1c4e334e9c9e12db8c48ce2249"
rpc_addresses {
host: "127.13.155.195"
port: 36725
}
}
ts_infos {
permanent_uuid: "69298260f4934e35a7f68c21c2205139"
rpc_addresses {
host: "127.13.155.196"
port: 39099
}
}
I20250625 13:57:31.690956 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 14174
I20250625 13:57:31.714524 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 14307
I20250625 13:57:31.742117 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 14440
I20250625 13:57:31.766690 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 14610
2025-06-25T13:57:31Z chronyd exiting
[ OK ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes (19501 ms)
[ RUN ] AdminCliTest.TestGracefulSpecificLeaderStepDown
I20250625 13:57:31.832551 13935 test_util.cc:276] Using random seed: -746251058
I20250625 13:57:31.838300 13935 ts_itest-base.cc:115] Starting cluster with:
I20250625 13:57:31.838491 13935 ts_itest-base.cc:116] --------------
I20250625 13:57:31.838678 13935 ts_itest-base.cc:117] 3 tablet servers
I20250625 13:57:31.838821 13935 ts_itest-base.cc:118] 3 replicas per TS
I20250625 13:57:31.838959 13935 ts_itest-base.cc:119] --------------
2025-06-25T13:57:31Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-25T13:57:31Z Disabled control of system clock
I20250625 13:57:31.874347 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.155.254:46565
--webserver_interface=127.13.155.254
--webserver_port=0
--builtin_ntp_servers=127.13.155.212:38013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.155.254:46565
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250625 13:57:32.171908 14800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:57:32.172544 14800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:57:32.173022 14800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:57:32.203804 14800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 13:57:32.204170 14800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:57:32.204437 14800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 13:57:32.204691 14800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 13:57:32.239359 14800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:38013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.13.155.254:46565
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.155.254:46565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.13.155.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:57:32.240720 14800 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:57:32.242340 14800 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:57:32.256410 14806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:57:32.258798 14800 server_base.cc:1048] running on GCE node
W20250625 13:57:32.256739 14807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:57:33.437561 14809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:57:33.440094 14800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:57:33.443120 14800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:57:33.444564 14800 hybrid_clock.cc:648] HybridClock initialized: now 1750859853444518 us; error 51 us; skew 500 ppm
I20250625 13:57:33.445338 14800 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:57:33.452020 14800 webserver.cc:469] Webserver started at http://127.13.155.254:37037/ using document root <none> and password file <none>
I20250625 13:57:33.452899 14800 fs_manager.cc:362] Metadata directory not provided
I20250625 13:57:33.453086 14800 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:57:33.453506 14800 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:57:33.457790 14800 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d42e34185a54422b89819206324ac4d0"
format_stamp: "Formatted at 2025-06-25 13:57:33 on dist-test-slave-k028"
I20250625 13:57:33.458853 14800 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d42e34185a54422b89819206324ac4d0"
format_stamp: "Formatted at 2025-06-25 13:57:33 on dist-test-slave-k028"
I20250625 13:57:33.465863 14800 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250625 13:57:33.471311 14817 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:33.472286 14800 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250625 13:57:33.472606 14800 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d42e34185a54422b89819206324ac4d0"
format_stamp: "Formatted at 2025-06-25 13:57:33 on dist-test-slave-k028"
I20250625 13:57:33.472910 14800 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:57:33.554111 14800 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:57:33.555660 14800 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:57:33.556051 14800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:57:33.625322 14800 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.254:46565
I20250625 13:57:33.625401 14868 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.254:46565 every 8 connection(s)
I20250625 13:57:33.627959 14800 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250625 13:57:33.632865 14869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:57:33.633174 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 14800
I20250625 13:57:33.633716 13935 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250625 13:57:33.659556 14869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d42e34185a54422b89819206324ac4d0: Bootstrap starting.
I20250625 13:57:33.664904 14869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d42e34185a54422b89819206324ac4d0: Neither blocks nor log segments found. Creating new log.
I20250625 13:57:33.666873 14869 log.cc:826] T 00000000000000000000000000000000 P d42e34185a54422b89819206324ac4d0: Log is configured to *not* fsync() on all Append() calls
I20250625 13:57:33.670975 14869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d42e34185a54422b89819206324ac4d0: No bootstrap required, opened a new log
I20250625 13:57:33.689116 14869 raft_consensus.cc:357] T 00000000000000000000000000000000 P d42e34185a54422b89819206324ac4d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42e34185a54422b89819206324ac4d0" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 46565 } }
I20250625 13:57:33.689756 14869 raft_consensus.cc:383] T 00000000000000000000000000000000 P d42e34185a54422b89819206324ac4d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:57:33.689955 14869 raft_consensus.cc:738] T 00000000000000000000000000000000 P d42e34185a54422b89819206324ac4d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d42e34185a54422b89819206324ac4d0, State: Initialized, Role: FOLLOWER
I20250625 13:57:33.690567 14869 consensus_queue.cc:260] T 00000000000000000000000000000000 P d42e34185a54422b89819206324ac4d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42e34185a54422b89819206324ac4d0" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 46565 } }
I20250625 13:57:33.691049 14869 raft_consensus.cc:397] T 00000000000000000000000000000000 P d42e34185a54422b89819206324ac4d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 13:57:33.691258 14869 raft_consensus.cc:491] T 00000000000000000000000000000000 P d42e34185a54422b89819206324ac4d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 13:57:33.691505 14869 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d42e34185a54422b89819206324ac4d0 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:57:33.695327 14869 raft_consensus.cc:513] T 00000000000000000000000000000000 P d42e34185a54422b89819206324ac4d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42e34185a54422b89819206324ac4d0" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 46565 } }
I20250625 13:57:33.695955 14869 leader_election.cc:304] T 00000000000000000000000000000000 P d42e34185a54422b89819206324ac4d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d42e34185a54422b89819206324ac4d0; no voters:
I20250625 13:57:33.697577 14869 leader_election.cc:290] T 00000000000000000000000000000000 P d42e34185a54422b89819206324ac4d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 13:57:33.698372 14874 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d42e34185a54422b89819206324ac4d0 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:57:33.700627 14874 raft_consensus.cc:695] T 00000000000000000000000000000000 P d42e34185a54422b89819206324ac4d0 [term 1 LEADER]: Becoming Leader. State: Replica: d42e34185a54422b89819206324ac4d0, State: Running, Role: LEADER
I20250625 13:57:33.701371 14874 consensus_queue.cc:237] T 00000000000000000000000000000000 P d42e34185a54422b89819206324ac4d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42e34185a54422b89819206324ac4d0" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 46565 } }
I20250625 13:57:33.702342 14869 sys_catalog.cc:564] T 00000000000000000000000000000000 P d42e34185a54422b89819206324ac4d0 [sys.catalog]: configured and running, proceeding with master startup.
I20250625 13:57:33.711519 14876 sys_catalog.cc:455] T 00000000000000000000000000000000 P d42e34185a54422b89819206324ac4d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d42e34185a54422b89819206324ac4d0. Latest consensus state: current_term: 1 leader_uuid: "d42e34185a54422b89819206324ac4d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42e34185a54422b89819206324ac4d0" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 46565 } } }
I20250625 13:57:33.712044 14875 sys_catalog.cc:455] T 00000000000000000000000000000000 P d42e34185a54422b89819206324ac4d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d42e34185a54422b89819206324ac4d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42e34185a54422b89819206324ac4d0" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 46565 } } }
I20250625 13:57:33.712600 14876 sys_catalog.cc:458] T 00000000000000000000000000000000 P d42e34185a54422b89819206324ac4d0 [sys.catalog]: This master's current role is: LEADER
I20250625 13:57:33.712611 14875 sys_catalog.cc:458] T 00000000000000000000000000000000 P d42e34185a54422b89819206324ac4d0 [sys.catalog]: This master's current role is: LEADER
I20250625 13:57:33.715682 14882 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 13:57:33.727936 14882 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 13:57:33.744742 14882 catalog_manager.cc:1349] Generated new cluster ID: 0798b5f06eee4d65ac8199811aca3df8
I20250625 13:57:33.745061 14882 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 13:57:33.758750 14882 catalog_manager.cc:1372] Generated new certificate authority record
I20250625 13:57:33.760193 14882 catalog_manager.cc:1506] Loading token signing keys...
I20250625 13:57:33.780257 14882 catalog_manager.cc:5955] T 00000000000000000000000000000000 P d42e34185a54422b89819206324ac4d0: Generated new TSK 0
I20250625 13:57:33.781353 14882 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 13:57:33.801124 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.155.193:0
--local_ip_for_outbound_sockets=127.13.155.193
--webserver_interface=127.13.155.193
--webserver_port=0
--tserver_master_addrs=127.13.155.254:46565
--builtin_ntp_servers=127.13.155.212:38013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250625 13:57:34.138938 14893 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250625 13:57:34.139572 14893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:57:34.139832 14893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:57:34.140306 14893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:57:34.171662 14893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:57:34.172502 14893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.155.193
I20250625 13:57:34.208920 14893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:38013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.155.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.13.155.193
--webserver_port=0
--tserver_master_addrs=127.13.155.254:46565
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.155.193
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:57:34.210325 14893 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:57:34.212028 14893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:57:34.230525 14900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:57:34.231001 14899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:57:34.234259 14902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:57:35.757272 14901 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1526 milliseconds
I20250625 13:57:35.757390 14893 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 13:57:35.758677 14893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:57:35.761389 14893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:57:35.762817 14893 hybrid_clock.cc:648] HybridClock initialized: now 1750859855762776 us; error 62 us; skew 500 ppm
I20250625 13:57:35.763608 14893 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:57:35.771010 14893 webserver.cc:469] Webserver started at http://127.13.155.193:46145/ using document root <none> and password file <none>
I20250625 13:57:35.771976 14893 fs_manager.cc:362] Metadata directory not provided
I20250625 13:57:35.772178 14893 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:57:35.772653 14893 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:57:35.777114 14893 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "7db1844c3d504045a83764e12984d3d1"
format_stamp: "Formatted at 2025-06-25 13:57:35 on dist-test-slave-k028"
I20250625 13:57:35.778288 14893 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "7db1844c3d504045a83764e12984d3d1"
format_stamp: "Formatted at 2025-06-25 13:57:35 on dist-test-slave-k028"
I20250625 13:57:35.785769 14893 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20250625 13:57:35.792716 14909 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:35.793735 14893 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250625 13:57:35.794051 14893 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "7db1844c3d504045a83764e12984d3d1"
format_stamp: "Formatted at 2025-06-25 13:57:35 on dist-test-slave-k028"
I20250625 13:57:35.794421 14893 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:57:35.844846 14893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:57:35.846310 14893 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:57:35.846750 14893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:57:35.849536 14893 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 13:57:35.853569 14893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 13:57:35.853796 14893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:35.854012 14893 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 13:57:35.854149 14893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:35.986943 14893 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.193:44087
I20250625 13:57:35.987074 15021 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.193:44087 every 8 connection(s)
I20250625 13:57:35.989538 14893 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250625 13:57:35.997421 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 14893
I20250625 13:57:35.997892 13935 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250625 13:57:36.005597 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.155.194:0
--local_ip_for_outbound_sockets=127.13.155.194
--webserver_interface=127.13.155.194
--webserver_port=0
--tserver_master_addrs=127.13.155.254:46565
--builtin_ntp_servers=127.13.155.212:38013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250625 13:57:36.011655 15022 heartbeater.cc:344] Connected to a master server at 127.13.155.254:46565
I20250625 13:57:36.012076 15022 heartbeater.cc:461] Registering TS with master...
I20250625 13:57:36.013054 15022 heartbeater.cc:507] Master 127.13.155.254:46565 requested a full tablet report, sending...
I20250625 13:57:36.015532 14834 ts_manager.cc:194] Registered new tserver with Master: 7db1844c3d504045a83764e12984d3d1 (127.13.155.193:44087)
I20250625 13:57:36.017599 14834 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.155.193:39247
W20250625 13:57:36.316596 15026 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250625 13:57:36.317227 15026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:57:36.317478 15026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:57:36.317989 15026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:57:36.354564 15026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:57:36.355652 15026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.155.194
I20250625 13:57:36.390987 15026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:38013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.155.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.13.155.194
--webserver_port=0
--tserver_master_addrs=127.13.155.254:46565
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.155.194
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:57:36.392349 15026 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:57:36.394107 15026 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:57:36.409451 15033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:57:37.021021 15022 heartbeater.cc:499] Master 127.13.155.254:46565 was elected leader, sending a full tablet report...
I20250625 13:57:36.412003 15026 server_base.cc:1048] running on GCE node
W20250625 13:57:36.410923 15035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:57:36.409766 15032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:57:37.608417 15026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:57:37.611150 15026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:57:37.612586 15026 hybrid_clock.cc:648] HybridClock initialized: now 1750859857612550 us; error 55 us; skew 500 ppm
I20250625 13:57:37.613382 15026 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:57:37.626413 15026 webserver.cc:469] Webserver started at http://127.13.155.194:41103/ using document root <none> and password file <none>
I20250625 13:57:37.627430 15026 fs_manager.cc:362] Metadata directory not provided
I20250625 13:57:37.627633 15026 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:57:37.628137 15026 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:57:37.632670 15026 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "def62bf4b0e1482583782117fe76f8fa"
format_stamp: "Formatted at 2025-06-25 13:57:37 on dist-test-slave-k028"
I20250625 13:57:37.633744 15026 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "def62bf4b0e1482583782117fe76f8fa"
format_stamp: "Formatted at 2025-06-25 13:57:37 on dist-test-slave-k028"
I20250625 13:57:37.640995 15026 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250625 13:57:37.646865 15042 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:37.647926 15026 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250625 13:57:37.648224 15026 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "def62bf4b0e1482583782117fe76f8fa"
format_stamp: "Formatted at 2025-06-25 13:57:37 on dist-test-slave-k028"
I20250625 13:57:37.648571 15026 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:57:37.719211 15026 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:57:37.720657 15026 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:57:37.721071 15026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:57:37.723559 15026 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 13:57:37.727582 15026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 13:57:37.727769 15026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:37.728006 15026 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 13:57:37.728227 15026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:37.866030 15026 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.194:46795
I20250625 13:57:37.866129 15154 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.194:46795 every 8 connection(s)
I20250625 13:57:37.868876 15026 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250625 13:57:37.870106 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 15026
I20250625 13:57:37.870508 13935 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250625 13:57:37.879629 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.155.195:0
--local_ip_for_outbound_sockets=127.13.155.195
--webserver_interface=127.13.155.195
--webserver_port=0
--tserver_master_addrs=127.13.155.254:46565
--builtin_ntp_servers=127.13.155.212:38013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250625 13:57:37.895880 15155 heartbeater.cc:344] Connected to a master server at 127.13.155.254:46565
I20250625 13:57:37.896440 15155 heartbeater.cc:461] Registering TS with master...
I20250625 13:57:37.897822 15155 heartbeater.cc:507] Master 127.13.155.254:46565 requested a full tablet report, sending...
I20250625 13:57:37.901551 14834 ts_manager.cc:194] Registered new tserver with Master: def62bf4b0e1482583782117fe76f8fa (127.13.155.194:46795)
I20250625 13:57:37.903605 14834 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.155.194:39375
W20250625 13:57:38.193053 15159 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250625 13:57:38.193677 15159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:57:38.193900 15159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:57:38.194314 15159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:57:38.226490 15159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:57:38.227300 15159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.155.195
I20250625 13:57:38.262825 15159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:38013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.155.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.13.155.195
--webserver_port=0
--tserver_master_addrs=127.13.155.254:46565
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.155.195
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:57:38.264204 15159 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:57:38.265889 15159 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:57:38.282322 15165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:57:38.907768 15155 heartbeater.cc:499] Master 127.13.155.254:46565 was elected leader, sending a full tablet report...
W20250625 13:57:38.283418 15168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:57:38.282428 15166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:57:39.533111 15167 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1248 milliseconds
I20250625 13:57:39.533210 15159 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 13:57:39.534399 15159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:57:39.537083 15159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:57:39.538540 15159 hybrid_clock.cc:648] HybridClock initialized: now 1750859859538498 us; error 56 us; skew 500 ppm
I20250625 13:57:39.539314 15159 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:57:39.545678 15159 webserver.cc:469] Webserver started at http://127.13.155.195:33547/ using document root <none> and password file <none>
I20250625 13:57:39.546607 15159 fs_manager.cc:362] Metadata directory not provided
I20250625 13:57:39.546809 15159 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:57:39.547250 15159 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:57:39.551550 15159 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "bb7f70044dca419a82e9f09444e30ee3"
format_stamp: "Formatted at 2025-06-25 13:57:39 on dist-test-slave-k028"
I20250625 13:57:39.552613 15159 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "bb7f70044dca419a82e9f09444e30ee3"
format_stamp: "Formatted at 2025-06-25 13:57:39 on dist-test-slave-k028"
I20250625 13:57:39.559494 15159 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250625 13:57:39.565033 15175 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:39.566021 15159 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250625 13:57:39.566321 15159 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "bb7f70044dca419a82e9f09444e30ee3"
format_stamp: "Formatted at 2025-06-25 13:57:39 on dist-test-slave-k028"
I20250625 13:57:39.566706 15159 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:57:39.620817 15159 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:57:39.622279 15159 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:57:39.622742 15159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:57:39.625845 15159 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 13:57:39.631577 15159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 13:57:39.631842 15159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:39.632120 15159 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 13:57:39.632325 15159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:39.767030 15159 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.195:39171
I20250625 13:57:39.767125 15287 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.195:39171 every 8 connection(s)
I20250625 13:57:39.769562 15159 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250625 13:57:39.779173 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 15159
I20250625 13:57:39.779525 13935 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250625 13:57:39.790526 15288 heartbeater.cc:344] Connected to a master server at 127.13.155.254:46565
I20250625 13:57:39.790953 15288 heartbeater.cc:461] Registering TS with master...
I20250625 13:57:39.792091 15288 heartbeater.cc:507] Master 127.13.155.254:46565 requested a full tablet report, sending...
I20250625 13:57:39.794092 14834 ts_manager.cc:194] Registered new tserver with Master: bb7f70044dca419a82e9f09444e30ee3 (127.13.155.195:39171)
I20250625 13:57:39.795374 14834 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.155.195:59685
I20250625 13:57:39.799392 13935 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 13:57:39.833969 14834 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49160:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250625 13:57:39.852252 14834 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:57:39.904948 14957 tablet_service.cc:1468] Processing CreateTablet for tablet 9efac11a67ca46998df938da5a0db53a (DEFAULT_TABLE table=TestTable [id=90e57a8b14744c3897ab16218c20b05e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:57:39.905815 15090 tablet_service.cc:1468] Processing CreateTablet for tablet 9efac11a67ca46998df938da5a0db53a (DEFAULT_TABLE table=TestTable [id=90e57a8b14744c3897ab16218c20b05e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:57:39.907052 14957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9efac11a67ca46998df938da5a0db53a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:57:39.907546 15090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9efac11a67ca46998df938da5a0db53a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:57:39.912945 15223 tablet_service.cc:1468] Processing CreateTablet for tablet 9efac11a67ca46998df938da5a0db53a (DEFAULT_TABLE table=TestTable [id=90e57a8b14744c3897ab16218c20b05e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:57:39.915751 15223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9efac11a67ca46998df938da5a0db53a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:57:39.929632 15307 tablet_bootstrap.cc:492] T 9efac11a67ca46998df938da5a0db53a P def62bf4b0e1482583782117fe76f8fa: Bootstrap starting.
I20250625 13:57:39.934208 15308 tablet_bootstrap.cc:492] T 9efac11a67ca46998df938da5a0db53a P 7db1844c3d504045a83764e12984d3d1: Bootstrap starting.
I20250625 13:57:39.936718 15307 tablet_bootstrap.cc:654] T 9efac11a67ca46998df938da5a0db53a P def62bf4b0e1482583782117fe76f8fa: Neither blocks nor log segments found. Creating new log.
I20250625 13:57:39.939491 15307 log.cc:826] T 9efac11a67ca46998df938da5a0db53a P def62bf4b0e1482583782117fe76f8fa: Log is configured to *not* fsync() on all Append() calls
I20250625 13:57:39.941952 15308 tablet_bootstrap.cc:654] T 9efac11a67ca46998df938da5a0db53a P 7db1844c3d504045a83764e12984d3d1: Neither blocks nor log segments found. Creating new log.
I20250625 13:57:39.942832 15309 tablet_bootstrap.cc:492] T 9efac11a67ca46998df938da5a0db53a P bb7f70044dca419a82e9f09444e30ee3: Bootstrap starting.
I20250625 13:57:39.944912 15308 log.cc:826] T 9efac11a67ca46998df938da5a0db53a P 7db1844c3d504045a83764e12984d3d1: Log is configured to *not* fsync() on all Append() calls
I20250625 13:57:39.948424 15309 tablet_bootstrap.cc:654] T 9efac11a67ca46998df938da5a0db53a P bb7f70044dca419a82e9f09444e30ee3: Neither blocks nor log segments found. Creating new log.
I20250625 13:57:39.949498 15307 tablet_bootstrap.cc:492] T 9efac11a67ca46998df938da5a0db53a P def62bf4b0e1482583782117fe76f8fa: No bootstrap required, opened a new log
I20250625 13:57:39.949973 15307 ts_tablet_manager.cc:1397] T 9efac11a67ca46998df938da5a0db53a P def62bf4b0e1482583782117fe76f8fa: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.010s
I20250625 13:57:39.951771 15309 log.cc:826] T 9efac11a67ca46998df938da5a0db53a P bb7f70044dca419a82e9f09444e30ee3: Log is configured to *not* fsync() on all Append() calls
I20250625 13:57:39.952203 15308 tablet_bootstrap.cc:492] T 9efac11a67ca46998df938da5a0db53a P 7db1844c3d504045a83764e12984d3d1: No bootstrap required, opened a new log
I20250625 13:57:39.952677 15308 ts_tablet_manager.cc:1397] T 9efac11a67ca46998df938da5a0db53a P 7db1844c3d504045a83764e12984d3d1: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.000s
I20250625 13:57:39.956589 15309 tablet_bootstrap.cc:492] T 9efac11a67ca46998df938da5a0db53a P bb7f70044dca419a82e9f09444e30ee3: No bootstrap required, opened a new log
I20250625 13:57:39.956974 15309 ts_tablet_manager.cc:1397] T 9efac11a67ca46998df938da5a0db53a P bb7f70044dca419a82e9f09444e30ee3: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.011s
I20250625 13:57:39.974567 15309 raft_consensus.cc:357] T 9efac11a67ca46998df938da5a0db53a P bb7f70044dca419a82e9f09444e30ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db1844c3d504045a83764e12984d3d1" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44087 } } peers { permanent_uuid: "bb7f70044dca419a82e9f09444e30ee3" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 39171 } } peers { permanent_uuid: "def62bf4b0e1482583782117fe76f8fa" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 46795 } }
I20250625 13:57:39.975267 15309 raft_consensus.cc:738] T 9efac11a67ca46998df938da5a0db53a P bb7f70044dca419a82e9f09444e30ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb7f70044dca419a82e9f09444e30ee3, State: Initialized, Role: FOLLOWER
I20250625 13:57:39.975865 15309 consensus_queue.cc:260] T 9efac11a67ca46998df938da5a0db53a P bb7f70044dca419a82e9f09444e30ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db1844c3d504045a83764e12984d3d1" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44087 } } peers { permanent_uuid: "bb7f70044dca419a82e9f09444e30ee3" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 39171 } } peers { permanent_uuid: "def62bf4b0e1482583782117fe76f8fa" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 46795 } }
I20250625 13:57:39.976604 15307 raft_consensus.cc:357] T 9efac11a67ca46998df938da5a0db53a P def62bf4b0e1482583782117fe76f8fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db1844c3d504045a83764e12984d3d1" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44087 } } peers { permanent_uuid: "bb7f70044dca419a82e9f09444e30ee3" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 39171 } } peers { permanent_uuid: "def62bf4b0e1482583782117fe76f8fa" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 46795 } }
I20250625 13:57:39.977571 15307 raft_consensus.cc:738] T 9efac11a67ca46998df938da5a0db53a P def62bf4b0e1482583782117fe76f8fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: def62bf4b0e1482583782117fe76f8fa, State: Initialized, Role: FOLLOWER
I20250625 13:57:39.978982 15288 heartbeater.cc:499] Master 127.13.155.254:46565 was elected leader, sending a full tablet report...
I20250625 13:57:39.978323 15307 consensus_queue.cc:260] T 9efac11a67ca46998df938da5a0db53a P def62bf4b0e1482583782117fe76f8fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db1844c3d504045a83764e12984d3d1" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44087 } } peers { permanent_uuid: "bb7f70044dca419a82e9f09444e30ee3" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 39171 } } peers { permanent_uuid: "def62bf4b0e1482583782117fe76f8fa" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 46795 } }
I20250625 13:57:39.979054 15308 raft_consensus.cc:357] T 9efac11a67ca46998df938da5a0db53a P 7db1844c3d504045a83764e12984d3d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db1844c3d504045a83764e12984d3d1" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44087 } } peers { permanent_uuid: "bb7f70044dca419a82e9f09444e30ee3" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 39171 } } peers { permanent_uuid: "def62bf4b0e1482583782117fe76f8fa" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 46795 } }
I20250625 13:57:39.979993 15308 raft_consensus.cc:738] T 9efac11a67ca46998df938da5a0db53a P 7db1844c3d504045a83764e12984d3d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7db1844c3d504045a83764e12984d3d1, State: Initialized, Role: FOLLOWER
I20250625 13:57:39.980233 15309 ts_tablet_manager.cc:1428] T 9efac11a67ca46998df938da5a0db53a P bb7f70044dca419a82e9f09444e30ee3: Time spent starting tablet: real 0.023s user 0.022s sys 0.001s
I20250625 13:57:39.980718 15308 consensus_queue.cc:260] T 9efac11a67ca46998df938da5a0db53a P 7db1844c3d504045a83764e12984d3d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db1844c3d504045a83764e12984d3d1" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44087 } } peers { permanent_uuid: "bb7f70044dca419a82e9f09444e30ee3" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 39171 } } peers { permanent_uuid: "def62bf4b0e1482583782117fe76f8fa" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 46795 } }
I20250625 13:57:39.983461 15307 ts_tablet_manager.cc:1428] T 9efac11a67ca46998df938da5a0db53a P def62bf4b0e1482583782117fe76f8fa: Time spent starting tablet: real 0.033s user 0.025s sys 0.007s
I20250625 13:57:39.987757 15308 ts_tablet_manager.cc:1428] T 9efac11a67ca46998df938da5a0db53a P 7db1844c3d504045a83764e12984d3d1: Time spent starting tablet: real 0.035s user 0.028s sys 0.004s
W20250625 13:57:39.998797 15023 tablet.cc:2378] T 9efac11a67ca46998df938da5a0db53a P 7db1844c3d504045a83764e12984d3d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 13:57:40.023200 13935 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
W20250625 13:57:40.024151 15289 tablet.cc:2378] T 9efac11a67ca46998df938da5a0db53a P bb7f70044dca419a82e9f09444e30ee3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 13:57:40.026418 13935 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 7db1844c3d504045a83764e12984d3d1 to finish bootstrapping
I20250625 13:57:40.039237 13935 ts_itest-base.cc:246] Waiting for 1 tablets on tserver def62bf4b0e1482583782117fe76f8fa to finish bootstrapping
I20250625 13:57:40.049405 13935 ts_itest-base.cc:246] Waiting for 1 tablets on tserver bb7f70044dca419a82e9f09444e30ee3 to finish bootstrapping
I20250625 13:57:40.089278 14977 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "9efac11a67ca46998df938da5a0db53a"
dest_uuid: "7db1844c3d504045a83764e12984d3d1"
from {username='slave'} at 127.0.0.1:42502
I20250625 13:57:40.089840 14977 raft_consensus.cc:491] T 9efac11a67ca46998df938da5a0db53a P 7db1844c3d504045a83764e12984d3d1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250625 13:57:40.090139 14977 raft_consensus.cc:3058] T 9efac11a67ca46998df938da5a0db53a P 7db1844c3d504045a83764e12984d3d1 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:57:40.094450 14977 raft_consensus.cc:513] T 9efac11a67ca46998df938da5a0db53a P 7db1844c3d504045a83764e12984d3d1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db1844c3d504045a83764e12984d3d1" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44087 } } peers { permanent_uuid: "bb7f70044dca419a82e9f09444e30ee3" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 39171 } } peers { permanent_uuid: "def62bf4b0e1482583782117fe76f8fa" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 46795 } }
I20250625 13:57:40.096833 14977 leader_election.cc:290] T 9efac11a67ca46998df938da5a0db53a P 7db1844c3d504045a83764e12984d3d1 [CANDIDATE]: Term 1 election: Requested vote from peers bb7f70044dca419a82e9f09444e30ee3 (127.13.155.195:39171), def62bf4b0e1482583782117fe76f8fa (127.13.155.194:46795)
I20250625 13:57:40.107856 13935 cluster_itest_util.cc:257] Not converged past 1 yet: 0.0 0.0 0.0
I20250625 13:57:40.109002 15243 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9efac11a67ca46998df938da5a0db53a" candidate_uuid: "7db1844c3d504045a83764e12984d3d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bb7f70044dca419a82e9f09444e30ee3"
I20250625 13:57:40.109619 15243 raft_consensus.cc:3058] T 9efac11a67ca46998df938da5a0db53a P bb7f70044dca419a82e9f09444e30ee3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:57:40.111444 15110 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9efac11a67ca46998df938da5a0db53a" candidate_uuid: "7db1844c3d504045a83764e12984d3d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "def62bf4b0e1482583782117fe76f8fa"
I20250625 13:57:40.112004 15110 raft_consensus.cc:3058] T 9efac11a67ca46998df938da5a0db53a P def62bf4b0e1482583782117fe76f8fa [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:57:40.114408 15243 raft_consensus.cc:2466] T 9efac11a67ca46998df938da5a0db53a P bb7f70044dca419a82e9f09444e30ee3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7db1844c3d504045a83764e12984d3d1 in term 1.
I20250625 13:57:40.115594 14910 leader_election.cc:304] T 9efac11a67ca46998df938da5a0db53a P 7db1844c3d504045a83764e12984d3d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7db1844c3d504045a83764e12984d3d1, bb7f70044dca419a82e9f09444e30ee3; no voters:
I20250625 13:57:40.116571 15315 raft_consensus.cc:2802] T 9efac11a67ca46998df938da5a0db53a P 7db1844c3d504045a83764e12984d3d1 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:57:40.116973 15110 raft_consensus.cc:2466] T 9efac11a67ca46998df938da5a0db53a P def62bf4b0e1482583782117fe76f8fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7db1844c3d504045a83764e12984d3d1 in term 1.
I20250625 13:57:40.118726 15315 raft_consensus.cc:695] T 9efac11a67ca46998df938da5a0db53a P 7db1844c3d504045a83764e12984d3d1 [term 1 LEADER]: Becoming Leader. State: Replica: 7db1844c3d504045a83764e12984d3d1, State: Running, Role: LEADER
I20250625 13:57:40.119544 15315 consensus_queue.cc:237] T 9efac11a67ca46998df938da5a0db53a P 7db1844c3d504045a83764e12984d3d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db1844c3d504045a83764e12984d3d1" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44087 } } peers { permanent_uuid: "bb7f70044dca419a82e9f09444e30ee3" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 39171 } } peers { permanent_uuid: "def62bf4b0e1482583782117fe76f8fa" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 46795 } }
W20250625 13:57:40.126777 15156 tablet.cc:2378] T 9efac11a67ca46998df938da5a0db53a P def62bf4b0e1482583782117fe76f8fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 13:57:40.130609 14831 catalog_manager.cc:5582] T 9efac11a67ca46998df938da5a0db53a P 7db1844c3d504045a83764e12984d3d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7db1844c3d504045a83764e12984d3d1 (127.13.155.193). New cstate: current_term: 1 leader_uuid: "7db1844c3d504045a83764e12984d3d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db1844c3d504045a83764e12984d3d1" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb7f70044dca419a82e9f09444e30ee3" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 39171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "def62bf4b0e1482583782117fe76f8fa" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 46795 } health_report { overall_health: UNKNOWN } } }
I20250625 13:57:40.213363 13935 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250625 13:57:40.418304 13935 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250625 13:57:40.535737 15315 consensus_queue.cc:1035] T 9efac11a67ca46998df938da5a0db53a P 7db1844c3d504045a83764e12984d3d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "def62bf4b0e1482583782117fe76f8fa" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 46795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:57:40.551347 15320 consensus_queue.cc:1035] T 9efac11a67ca46998df938da5a0db53a P 7db1844c3d504045a83764e12984d3d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb7f70044dca419a82e9f09444e30ee3" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 39171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:57:42.429992 14977 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "9efac11a67ca46998df938da5a0db53a"
dest_uuid: "7db1844c3d504045a83764e12984d3d1"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42518
I20250625 13:57:42.430509 14977 raft_consensus.cc:604] T 9efac11a67ca46998df938da5a0db53a P 7db1844c3d504045a83764e12984d3d1 [term 1 LEADER]: Received request to transfer leadership
I20250625 13:57:42.437685 15351 raft_consensus.cc:991] T 9efac11a67ca46998df938da5a0db53a P 7db1844c3d504045a83764e12984d3d1: : Instructing follower bb7f70044dca419a82e9f09444e30ee3 to start an election
I20250625 13:57:42.438189 15334 raft_consensus.cc:1079] T 9efac11a67ca46998df938da5a0db53a P 7db1844c3d504045a83764e12984d3d1 [term 1 LEADER]: Signalling peer bb7f70044dca419a82e9f09444e30ee3 to start an election
I20250625 13:57:42.440318 15243 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "9efac11a67ca46998df938da5a0db53a"
dest_uuid: "bb7f70044dca419a82e9f09444e30ee3"
from {username='slave'} at 127.13.155.193:57935
I20250625 13:57:42.440973 15243 raft_consensus.cc:491] T 9efac11a67ca46998df938da5a0db53a P bb7f70044dca419a82e9f09444e30ee3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250625 13:57:42.441322 15243 raft_consensus.cc:3058] T 9efac11a67ca46998df938da5a0db53a P bb7f70044dca419a82e9f09444e30ee3 [term 1 FOLLOWER]: Advancing to term 2
I20250625 13:57:42.447585 15243 raft_consensus.cc:513] T 9efac11a67ca46998df938da5a0db53a P bb7f70044dca419a82e9f09444e30ee3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db1844c3d504045a83764e12984d3d1" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44087 } } peers { permanent_uuid: "bb7f70044dca419a82e9f09444e30ee3" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 39171 } } peers { permanent_uuid: "def62bf4b0e1482583782117fe76f8fa" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 46795 } }
I20250625 13:57:42.449785 15243 leader_election.cc:290] T 9efac11a67ca46998df938da5a0db53a P bb7f70044dca419a82e9f09444e30ee3 [CANDIDATE]: Term 2 election: Requested vote from peers 7db1844c3d504045a83764e12984d3d1 (127.13.155.193:44087), def62bf4b0e1482583782117fe76f8fa (127.13.155.194:46795)
I20250625 13:57:42.470602 14977 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9efac11a67ca46998df938da5a0db53a" candidate_uuid: "bb7f70044dca419a82e9f09444e30ee3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7db1844c3d504045a83764e12984d3d1"
I20250625 13:57:42.471261 14977 raft_consensus.cc:3053] T 9efac11a67ca46998df938da5a0db53a P 7db1844c3d504045a83764e12984d3d1 [term 1 LEADER]: Stepping down as leader of term 1
I20250625 13:57:42.471577 14977 raft_consensus.cc:738] T 9efac11a67ca46998df938da5a0db53a P 7db1844c3d504045a83764e12984d3d1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7db1844c3d504045a83764e12984d3d1, State: Running, Role: LEADER
I20250625 13:57:42.472182 14977 consensus_queue.cc:260] T 9efac11a67ca46998df938da5a0db53a P 7db1844c3d504045a83764e12984d3d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db1844c3d504045a83764e12984d3d1" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44087 } } peers { permanent_uuid: "bb7f70044dca419a82e9f09444e30ee3" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 39171 } } peers { permanent_uuid: "def62bf4b0e1482583782117fe76f8fa" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 46795 } }
I20250625 13:57:42.473057 15110 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9efac11a67ca46998df938da5a0db53a" candidate_uuid: "bb7f70044dca419a82e9f09444e30ee3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "def62bf4b0e1482583782117fe76f8fa"
I20250625 13:57:42.473258 14977 raft_consensus.cc:3058] T 9efac11a67ca46998df938da5a0db53a P 7db1844c3d504045a83764e12984d3d1 [term 1 FOLLOWER]: Advancing to term 2
I20250625 13:57:42.473599 15110 raft_consensus.cc:3058] T 9efac11a67ca46998df938da5a0db53a P def62bf4b0e1482583782117fe76f8fa [term 1 FOLLOWER]: Advancing to term 2
I20250625 13:57:42.479786 15110 raft_consensus.cc:2466] T 9efac11a67ca46998df938da5a0db53a P def62bf4b0e1482583782117fe76f8fa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb7f70044dca419a82e9f09444e30ee3 in term 2.
I20250625 13:57:42.479794 14977 raft_consensus.cc:2466] T 9efac11a67ca46998df938da5a0db53a P 7db1844c3d504045a83764e12984d3d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb7f70044dca419a82e9f09444e30ee3 in term 2.
I20250625 13:57:42.480899 15177 leader_election.cc:304] T 9efac11a67ca46998df938da5a0db53a P bb7f70044dca419a82e9f09444e30ee3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb7f70044dca419a82e9f09444e30ee3, def62bf4b0e1482583782117fe76f8fa; no voters:
I20250625 13:57:42.482687 15355 raft_consensus.cc:2802] T 9efac11a67ca46998df938da5a0db53a P bb7f70044dca419a82e9f09444e30ee3 [term 2 FOLLOWER]: Leader election won for term 2
I20250625 13:57:42.483919 15355 raft_consensus.cc:695] T 9efac11a67ca46998df938da5a0db53a P bb7f70044dca419a82e9f09444e30ee3 [term 2 LEADER]: Becoming Leader. State: Replica: bb7f70044dca419a82e9f09444e30ee3, State: Running, Role: LEADER
I20250625 13:57:42.484694 15355 consensus_queue.cc:237] T 9efac11a67ca46998df938da5a0db53a P bb7f70044dca419a82e9f09444e30ee3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db1844c3d504045a83764e12984d3d1" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44087 } } peers { permanent_uuid: "bb7f70044dca419a82e9f09444e30ee3" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 39171 } } peers { permanent_uuid: "def62bf4b0e1482583782117fe76f8fa" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 46795 } }
I20250625 13:57:42.491745 14832 catalog_manager.cc:5582] T 9efac11a67ca46998df938da5a0db53a P bb7f70044dca419a82e9f09444e30ee3 reported cstate change: term changed from 1 to 2, leader changed from 7db1844c3d504045a83764e12984d3d1 (127.13.155.193) to bb7f70044dca419a82e9f09444e30ee3 (127.13.155.195). New cstate: current_term: 2 leader_uuid: "bb7f70044dca419a82e9f09444e30ee3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db1844c3d504045a83764e12984d3d1" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb7f70044dca419a82e9f09444e30ee3" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 39171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "def62bf4b0e1482583782117fe76f8fa" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 46795 } health_report { overall_health: UNKNOWN } } }
I20250625 13:57:43.056699 14977 raft_consensus.cc:1273] T 9efac11a67ca46998df938da5a0db53a P 7db1844c3d504045a83764e12984d3d1 [term 2 FOLLOWER]: Refusing update from remote peer bb7f70044dca419a82e9f09444e30ee3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250625 13:57:43.057953 15362 consensus_queue.cc:1035] T 9efac11a67ca46998df938da5a0db53a P bb7f70044dca419a82e9f09444e30ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7db1844c3d504045a83764e12984d3d1" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250625 13:57:43.066821 15110 raft_consensus.cc:1273] T 9efac11a67ca46998df938da5a0db53a P def62bf4b0e1482583782117fe76f8fa [term 2 FOLLOWER]: Refusing update from remote peer bb7f70044dca419a82e9f09444e30ee3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250625 13:57:43.068547 15365 consensus_queue.cc:1035] T 9efac11a67ca46998df938da5a0db53a P bb7f70044dca419a82e9f09444e30ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "def62bf4b0e1482583782117fe76f8fa" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 46795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250625 13:57:45.170344 14977 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "9efac11a67ca46998df938da5a0db53a"
dest_uuid: "7db1844c3d504045a83764e12984d3d1"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52554
I20250625 13:57:45.170986 14977 raft_consensus.cc:604] T 9efac11a67ca46998df938da5a0db53a P 7db1844c3d504045a83764e12984d3d1 [term 2 FOLLOWER]: Received request to transfer leadership
I20250625 13:57:45.171280 14977 raft_consensus.cc:612] T 9efac11a67ca46998df938da5a0db53a P 7db1844c3d504045a83764e12984d3d1 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250625 13:57:46.206553 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 14893
W20250625 13:57:46.230616 15176 connection.cc:537] client connection to 127.13.155.193:44087 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 13:57:46.231092 15176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250625 13:57:46.232127 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 15026
W20250625 13:57:46.236238 15176 consensus_peers.cc:487] T 9efac11a67ca46998df938da5a0db53a P bb7f70044dca419a82e9f09444e30ee3 -> Peer 7db1844c3d504045a83764e12984d3d1 (127.13.155.193:44087): Couldn't send request to peer 7db1844c3d504045a83764e12984d3d1. Status: Network error: Client connection negotiation failed: client connection to 127.13.155.193:44087: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250625 13:57:46.258670 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 15159
I20250625 13:57:46.285843 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 14800
2025-06-25T13:57:46Z chronyd exiting
[ OK ] AdminCliTest.TestGracefulSpecificLeaderStepDown (14508 ms)
[ RUN ] AdminCliTest.TestDescribeTableColumnFlags
I20250625 13:57:46.341696 13935 test_util.cc:276] Using random seed: -731741917
I20250625 13:57:46.345889 13935 ts_itest-base.cc:115] Starting cluster with:
I20250625 13:57:46.346074 13935 ts_itest-base.cc:116] --------------
I20250625 13:57:46.346231 13935 ts_itest-base.cc:117] 3 tablet servers
I20250625 13:57:46.346374 13935 ts_itest-base.cc:118] 3 replicas per TS
I20250625 13:57:46.346661 13935 ts_itest-base.cc:119] --------------
2025-06-25T13:57:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-25T13:57:46Z Disabled control of system clock
I20250625 13:57:46.384117 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.155.254:36053
--webserver_interface=127.13.155.254
--webserver_port=0
--builtin_ntp_servers=127.13.155.212:33993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.155.254:36053 with env {}
W20250625 13:57:46.680083 15402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:57:46.680627 15402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:57:46.681018 15402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:57:46.712154 15402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 13:57:46.712443 15402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:57:46.712649 15402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 13:57:46.712843 15402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 13:57:46.748178 15402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:33993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.155.254:36053
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.155.254:36053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.13.155.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:57:46.749433 15402 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:57:46.751169 15402 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:57:46.765185 15409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:57:46.766366 15408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:57:46.768695 15402 server_base.cc:1048] running on GCE node
W20250625 13:57:46.767412 15411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:57:47.955166 15402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:57:47.957778 15402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:57:47.959138 15402 hybrid_clock.cc:648] HybridClock initialized: now 1750859867959098 us; error 57 us; skew 500 ppm
I20250625 13:57:47.959972 15402 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:57:47.966279 15402 webserver.cc:469] Webserver started at http://127.13.155.254:36219/ using document root <none> and password file <none>
I20250625 13:57:47.967305 15402 fs_manager.cc:362] Metadata directory not provided
I20250625 13:57:47.967505 15402 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:57:47.968009 15402 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:57:47.972476 15402 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "65dd368b05c842f7be36d630d349aff3"
format_stamp: "Formatted at 2025-06-25 13:57:47 on dist-test-slave-k028"
I20250625 13:57:47.973615 15402 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "65dd368b05c842f7be36d630d349aff3"
format_stamp: "Formatted at 2025-06-25 13:57:47 on dist-test-slave-k028"
I20250625 13:57:47.980839 15402 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250625 13:57:47.986629 15418 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:47.987733 15402 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250625 13:57:47.988077 15402 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
uuid: "65dd368b05c842f7be36d630d349aff3"
format_stamp: "Formatted at 2025-06-25 13:57:47 on dist-test-slave-k028"
I20250625 13:57:47.988410 15402 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:57:48.043460 15402 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:57:48.045017 15402 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:57:48.045475 15402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:57:48.115952 15402 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.254:36053
I20250625 13:57:48.116056 15469 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.254:36053 every 8 connection(s)
I20250625 13:57:48.118769 15402 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250625 13:57:48.120078 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 15402
I20250625 13:57:48.120482 13935 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250625 13:57:48.126235 15470 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:57:48.151314 15470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65dd368b05c842f7be36d630d349aff3: Bootstrap starting.
I20250625 13:57:48.157486 15470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65dd368b05c842f7be36d630d349aff3: Neither blocks nor log segments found. Creating new log.
I20250625 13:57:48.159369 15470 log.cc:826] T 00000000000000000000000000000000 P 65dd368b05c842f7be36d630d349aff3: Log is configured to *not* fsync() on all Append() calls
I20250625 13:57:48.163872 15470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65dd368b05c842f7be36d630d349aff3: No bootstrap required, opened a new log
I20250625 13:57:48.181780 15470 raft_consensus.cc:357] T 00000000000000000000000000000000 P 65dd368b05c842f7be36d630d349aff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65dd368b05c842f7be36d630d349aff3" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 36053 } }
I20250625 13:57:48.182489 15470 raft_consensus.cc:383] T 00000000000000000000000000000000 P 65dd368b05c842f7be36d630d349aff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:57:48.182741 15470 raft_consensus.cc:738] T 00000000000000000000000000000000 P 65dd368b05c842f7be36d630d349aff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65dd368b05c842f7be36d630d349aff3, State: Initialized, Role: FOLLOWER
I20250625 13:57:48.183635 15470 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65dd368b05c842f7be36d630d349aff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65dd368b05c842f7be36d630d349aff3" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 36053 } }
I20250625 13:57:48.184286 15470 raft_consensus.cc:397] T 00000000000000000000000000000000 P 65dd368b05c842f7be36d630d349aff3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 13:57:48.184657 15470 raft_consensus.cc:491] T 00000000000000000000000000000000 P 65dd368b05c842f7be36d630d349aff3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 13:57:48.185060 15470 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 65dd368b05c842f7be36d630d349aff3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:57:48.189182 15470 raft_consensus.cc:513] T 00000000000000000000000000000000 P 65dd368b05c842f7be36d630d349aff3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65dd368b05c842f7be36d630d349aff3" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 36053 } }
I20250625 13:57:48.189889 15470 leader_election.cc:304] T 00000000000000000000000000000000 P 65dd368b05c842f7be36d630d349aff3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65dd368b05c842f7be36d630d349aff3; no voters:
I20250625 13:57:48.191529 15470 leader_election.cc:290] T 00000000000000000000000000000000 P 65dd368b05c842f7be36d630d349aff3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 13:57:48.192327 15475 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 65dd368b05c842f7be36d630d349aff3 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:57:48.194545 15475 raft_consensus.cc:695] T 00000000000000000000000000000000 P 65dd368b05c842f7be36d630d349aff3 [term 1 LEADER]: Becoming Leader. State: Replica: 65dd368b05c842f7be36d630d349aff3, State: Running, Role: LEADER
I20250625 13:57:48.195243 15475 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65dd368b05c842f7be36d630d349aff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65dd368b05c842f7be36d630d349aff3" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 36053 } }
I20250625 13:57:48.195894 15470 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65dd368b05c842f7be36d630d349aff3 [sys.catalog]: configured and running, proceeding with master startup.
I20250625 13:57:48.206837 15476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65dd368b05c842f7be36d630d349aff3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65dd368b05c842f7be36d630d349aff3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65dd368b05c842f7be36d630d349aff3" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 36053 } } }
I20250625 13:57:48.206933 15477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65dd368b05c842f7be36d630d349aff3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65dd368b05c842f7be36d630d349aff3. Latest consensus state: current_term: 1 leader_uuid: "65dd368b05c842f7be36d630d349aff3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65dd368b05c842f7be36d630d349aff3" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 36053 } } }
I20250625 13:57:48.208133 15476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65dd368b05c842f7be36d630d349aff3 [sys.catalog]: This master's current role is: LEADER
I20250625 13:57:48.208149 15477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65dd368b05c842f7be36d630d349aff3 [sys.catalog]: This master's current role is: LEADER
I20250625 13:57:48.214926 15485 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 13:57:48.229043 15485 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 13:57:48.245813 15485 catalog_manager.cc:1349] Generated new cluster ID: 018db893602546e0bafcaced5754b04e
I20250625 13:57:48.246166 15485 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 13:57:48.279593 15485 catalog_manager.cc:1372] Generated new certificate authority record
I20250625 13:57:48.281205 15485 catalog_manager.cc:1506] Loading token signing keys...
I20250625 13:57:48.294713 15485 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 65dd368b05c842f7be36d630d349aff3: Generated new TSK 0
I20250625 13:57:48.296779 15485 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 13:57:48.318176 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.155.193:0
--local_ip_for_outbound_sockets=127.13.155.193
--webserver_interface=127.13.155.193
--webserver_port=0
--tserver_master_addrs=127.13.155.254:36053
--builtin_ntp_servers=127.13.155.212:33993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250625 13:57:48.622534 15494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:57:48.623044 15494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:57:48.623534 15494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:57:48.655448 15494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:57:48.656308 15494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.155.193
I20250625 13:57:48.691331 15494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:33993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.155.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.13.155.193
--webserver_port=0
--tserver_master_addrs=127.13.155.254:36053
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.155.193
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:57:48.692649 15494 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:57:48.694291 15494 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:57:48.711429 15501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:57:48.711572 15500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:57:48.714398 15494 server_base.cc:1048] running on GCE node
W20250625 13:57:48.713245 15503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:57:49.911182 15494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:57:49.913942 15494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:57:49.915445 15494 hybrid_clock.cc:648] HybridClock initialized: now 1750859869915384 us; error 74 us; skew 500 ppm
I20250625 13:57:49.916414 15494 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:57:49.928146 15494 webserver.cc:469] Webserver started at http://127.13.155.193:44667/ using document root <none> and password file <none>
I20250625 13:57:49.929087 15494 fs_manager.cc:362] Metadata directory not provided
I20250625 13:57:49.929287 15494 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:57:49.929737 15494 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:57:49.935616 15494 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "ea8da515f6534c9ca2a7a343feaa4347"
format_stamp: "Formatted at 2025-06-25 13:57:49 on dist-test-slave-k028"
I20250625 13:57:49.936736 15494 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "ea8da515f6534c9ca2a7a343feaa4347"
format_stamp: "Formatted at 2025-06-25 13:57:49 on dist-test-slave-k028"
I20250625 13:57:49.944377 15494 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20250625 13:57:49.949865 15510 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:49.950901 15494 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250625 13:57:49.951210 15494 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "ea8da515f6534c9ca2a7a343feaa4347"
format_stamp: "Formatted at 2025-06-25 13:57:49 on dist-test-slave-k028"
I20250625 13:57:49.951526 15494 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:57:50.003222 15494 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:57:50.004655 15494 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:57:50.005098 15494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:57:50.007576 15494 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 13:57:50.011549 15494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 13:57:50.011755 15494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:50.012058 15494 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 13:57:50.012210 15494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:50.153669 15494 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.193:44687
I20250625 13:57:50.153781 15622 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.193:44687 every 8 connection(s)
I20250625 13:57:50.156351 15494 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250625 13:57:50.163652 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 15494
I20250625 13:57:50.164053 13935 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250625 13:57:50.170337 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.155.194:0
--local_ip_for_outbound_sockets=127.13.155.194
--webserver_interface=127.13.155.194
--webserver_port=0
--tserver_master_addrs=127.13.155.254:36053
--builtin_ntp_servers=127.13.155.212:33993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 13:57:50.178805 15623 heartbeater.cc:344] Connected to a master server at 127.13.155.254:36053
I20250625 13:57:50.179356 15623 heartbeater.cc:461] Registering TS with master...
I20250625 13:57:50.180728 15623 heartbeater.cc:507] Master 127.13.155.254:36053 requested a full tablet report, sending...
I20250625 13:57:50.184006 15435 ts_manager.cc:194] Registered new tserver with Master: ea8da515f6534c9ca2a7a343feaa4347 (127.13.155.193:44687)
I20250625 13:57:50.186136 15435 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.155.193:57601
W20250625 13:57:50.476593 15627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:57:50.477094 15627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:57:50.477643 15627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:57:50.509475 15627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:57:50.510406 15627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.155.194
I20250625 13:57:50.546885 15627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:33993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.155.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.13.155.194
--webserver_port=0
--tserver_master_addrs=127.13.155.254:36053
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.155.194
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:57:50.548207 15627 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:57:50.549887 15627 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:57:50.565658 15634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:57:51.190994 15623 heartbeater.cc:499] Master 127.13.155.254:36053 was elected leader, sending a full tablet report...
I20250625 13:57:50.568444 15627 server_base.cc:1048] running on GCE node
W20250625 13:57:50.567353 15636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:57:50.566498 15633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:57:51.759963 15627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:57:51.762315 15627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:57:51.763747 15627 hybrid_clock.cc:648] HybridClock initialized: now 1750859871763707 us; error 62 us; skew 500 ppm
I20250625 13:57:51.764590 15627 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:57:51.771384 15627 webserver.cc:469] Webserver started at http://127.13.155.194:46001/ using document root <none> and password file <none>
I20250625 13:57:51.772389 15627 fs_manager.cc:362] Metadata directory not provided
I20250625 13:57:51.772639 15627 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:57:51.773130 15627 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:57:51.777678 15627 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "243fbb4c47ce414c891291d15cbc4385"
format_stamp: "Formatted at 2025-06-25 13:57:51 on dist-test-slave-k028"
I20250625 13:57:51.778893 15627 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "243fbb4c47ce414c891291d15cbc4385"
format_stamp: "Formatted at 2025-06-25 13:57:51 on dist-test-slave-k028"
I20250625 13:57:51.787007 15627 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250625 13:57:51.794128 15643 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:51.795481 15627 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250625 13:57:51.795846 15627 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "243fbb4c47ce414c891291d15cbc4385"
format_stamp: "Formatted at 2025-06-25 13:57:51 on dist-test-slave-k028"
I20250625 13:57:51.796211 15627 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:57:51.848850 15627 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:57:51.850378 15627 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:57:51.850885 15627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:57:51.853451 15627 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 13:57:51.857715 15627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 13:57:51.857956 15627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:51.858230 15627 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 13:57:51.858399 15627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:51.993287 15627 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.194:45999
I20250625 13:57:51.993399 15755 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.194:45999 every 8 connection(s)
I20250625 13:57:51.995986 15627 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250625 13:57:51.999904 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 15627
I20250625 13:57:52.000445 13935 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250625 13:57:52.007730 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.155.195:0
--local_ip_for_outbound_sockets=127.13.155.195
--webserver_interface=127.13.155.195
--webserver_port=0
--tserver_master_addrs=127.13.155.254:36053
--builtin_ntp_servers=127.13.155.212:33993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 13:57:52.019145 15756 heartbeater.cc:344] Connected to a master server at 127.13.155.254:36053
I20250625 13:57:52.019620 15756 heartbeater.cc:461] Registering TS with master...
I20250625 13:57:52.020737 15756 heartbeater.cc:507] Master 127.13.155.254:36053 requested a full tablet report, sending...
I20250625 13:57:52.023459 15435 ts_manager.cc:194] Registered new tserver with Master: 243fbb4c47ce414c891291d15cbc4385 (127.13.155.194:45999)
I20250625 13:57:52.025458 15435 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.155.194:58597
W20250625 13:57:52.318270 15760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:57:52.318814 15760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:57:52.319270 15760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:57:52.351960 15760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:57:52.352792 15760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.155.195
I20250625 13:57:52.389034 15760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:33993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.155.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.13.155.195
--webserver_port=0
--tserver_master_addrs=127.13.155.254:36053
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.155.195
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:57:52.390316 15760 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:57:52.391991 15760 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:57:52.408471 15767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:57:53.029635 15756 heartbeater.cc:499] Master 127.13.155.254:36053 was elected leader, sending a full tablet report...
W20250625 13:57:52.410506 15770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:57:52.408550 15768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:57:53.613572 15769 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1204 milliseconds
I20250625 13:57:53.613684 15760 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 13:57:53.614935 15760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:57:53.617688 15760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:57:53.619143 15760 hybrid_clock.cc:648] HybridClock initialized: now 1750859873619098 us; error 64 us; skew 500 ppm
I20250625 13:57:53.619917 15760 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:57:53.626782 15760 webserver.cc:469] Webserver started at http://127.13.155.195:46513/ using document root <none> and password file <none>
I20250625 13:57:53.627709 15760 fs_manager.cc:362] Metadata directory not provided
I20250625 13:57:53.627907 15760 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:57:53.628326 15760 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:57:53.633121 15760 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "8f5d2e56dd7d46618bf486a4d1fc8ff1"
format_stamp: "Formatted at 2025-06-25 13:57:53 on dist-test-slave-k028"
I20250625 13:57:53.634210 15760 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "8f5d2e56dd7d46618bf486a4d1fc8ff1"
format_stamp: "Formatted at 2025-06-25 13:57:53 on dist-test-slave-k028"
I20250625 13:57:53.641423 15760 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20250625 13:57:53.647751 15777 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:53.649012 15760 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250625 13:57:53.649346 15760 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "8f5d2e56dd7d46618bf486a4d1fc8ff1"
format_stamp: "Formatted at 2025-06-25 13:57:53 on dist-test-slave-k028"
I20250625 13:57:53.649719 15760 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:57:53.704468 15760 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:57:53.705926 15760 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:57:53.706370 15760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:57:53.708918 15760 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 13:57:53.712927 15760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 13:57:53.713124 15760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:53.713393 15760 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 13:57:53.713546 15760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:53.852243 15760 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.195:41417
I20250625 13:57:53.852363 15889 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.195:41417 every 8 connection(s)
I20250625 13:57:53.854921 15760 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250625 13:57:53.861064 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 15760
I20250625 13:57:53.861685 13935 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250625 13:57:53.876605 15890 heartbeater.cc:344] Connected to a master server at 127.13.155.254:36053
I20250625 13:57:53.877048 15890 heartbeater.cc:461] Registering TS with master...
I20250625 13:57:53.878077 15890 heartbeater.cc:507] Master 127.13.155.254:36053 requested a full tablet report, sending...
I20250625 13:57:53.880566 15434 ts_manager.cc:194] Registered new tserver with Master: 8f5d2e56dd7d46618bf486a4d1fc8ff1 (127.13.155.195:41417)
I20250625 13:57:53.882318 15434 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.155.195:35649
I20250625 13:57:53.882694 13935 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 13:57:53.917914 15434 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52202:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250625 13:57:53.937599 15434 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:57:54.000375 15691 tablet_service.cc:1468] Processing CreateTablet for tablet f7fd7b8c67da4c70a78bc7f0e929d942 (DEFAULT_TABLE table=TestTable [id=2d24f641c8cc4547a834f7a1391a345d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:57:54.002677 15691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7fd7b8c67da4c70a78bc7f0e929d942. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:57:54.013211 15558 tablet_service.cc:1468] Processing CreateTablet for tablet f7fd7b8c67da4c70a78bc7f0e929d942 (DEFAULT_TABLE table=TestTable [id=2d24f641c8cc4547a834f7a1391a345d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:57:54.015414 15558 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7fd7b8c67da4c70a78bc7f0e929d942. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:57:54.020236 15825 tablet_service.cc:1468] Processing CreateTablet for tablet f7fd7b8c67da4c70a78bc7f0e929d942 (DEFAULT_TABLE table=TestTable [id=2d24f641c8cc4547a834f7a1391a345d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:57:54.027607 15825 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7fd7b8c67da4c70a78bc7f0e929d942. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:57:54.046157 15909 tablet_bootstrap.cc:492] T f7fd7b8c67da4c70a78bc7f0e929d942 P 243fbb4c47ce414c891291d15cbc4385: Bootstrap starting.
I20250625 13:57:54.049531 15910 tablet_bootstrap.cc:492] T f7fd7b8c67da4c70a78bc7f0e929d942 P ea8da515f6534c9ca2a7a343feaa4347: Bootstrap starting.
I20250625 13:57:54.054075 15909 tablet_bootstrap.cc:654] T f7fd7b8c67da4c70a78bc7f0e929d942 P 243fbb4c47ce414c891291d15cbc4385: Neither blocks nor log segments found. Creating new log.
I20250625 13:57:54.056710 15909 log.cc:826] T f7fd7b8c67da4c70a78bc7f0e929d942 P 243fbb4c47ce414c891291d15cbc4385: Log is configured to *not* fsync() on all Append() calls
I20250625 13:57:54.057518 15910 tablet_bootstrap.cc:654] T f7fd7b8c67da4c70a78bc7f0e929d942 P ea8da515f6534c9ca2a7a343feaa4347: Neither blocks nor log segments found. Creating new log.
I20250625 13:57:54.060267 15910 log.cc:826] T f7fd7b8c67da4c70a78bc7f0e929d942 P ea8da515f6534c9ca2a7a343feaa4347: Log is configured to *not* fsync() on all Append() calls
I20250625 13:57:54.063663 15909 tablet_bootstrap.cc:492] T f7fd7b8c67da4c70a78bc7f0e929d942 P 243fbb4c47ce414c891291d15cbc4385: No bootstrap required, opened a new log
I20250625 13:57:54.064340 15909 ts_tablet_manager.cc:1397] T f7fd7b8c67da4c70a78bc7f0e929d942 P 243fbb4c47ce414c891291d15cbc4385: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.005s
I20250625 13:57:54.069465 15913 tablet_bootstrap.cc:492] T f7fd7b8c67da4c70a78bc7f0e929d942 P 8f5d2e56dd7d46618bf486a4d1fc8ff1: Bootstrap starting.
I20250625 13:57:54.072412 15910 tablet_bootstrap.cc:492] T f7fd7b8c67da4c70a78bc7f0e929d942 P ea8da515f6534c9ca2a7a343feaa4347: No bootstrap required, opened a new log
I20250625 13:57:54.073032 15910 ts_tablet_manager.cc:1397] T f7fd7b8c67da4c70a78bc7f0e929d942 P ea8da515f6534c9ca2a7a343feaa4347: Time spent bootstrapping tablet: real 0.024s user 0.017s sys 0.004s
I20250625 13:57:54.086244 15913 tablet_bootstrap.cc:654] T f7fd7b8c67da4c70a78bc7f0e929d942 P 8f5d2e56dd7d46618bf486a4d1fc8ff1: Neither blocks nor log segments found. Creating new log.
I20250625 13:57:54.088824 15913 log.cc:826] T f7fd7b8c67da4c70a78bc7f0e929d942 P 8f5d2e56dd7d46618bf486a4d1fc8ff1: Log is configured to *not* fsync() on all Append() calls
I20250625 13:57:54.095048 15909 raft_consensus.cc:357] T f7fd7b8c67da4c70a78bc7f0e929d942 P 243fbb4c47ce414c891291d15cbc4385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8da515f6534c9ca2a7a343feaa4347" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44687 } } peers { permanent_uuid: "243fbb4c47ce414c891291d15cbc4385" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 45999 } } peers { permanent_uuid: "8f5d2e56dd7d46618bf486a4d1fc8ff1" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 41417 } }
I20250625 13:57:54.096340 15909 raft_consensus.cc:383] T f7fd7b8c67da4c70a78bc7f0e929d942 P 243fbb4c47ce414c891291d15cbc4385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:57:54.096767 15909 raft_consensus.cc:738] T f7fd7b8c67da4c70a78bc7f0e929d942 P 243fbb4c47ce414c891291d15cbc4385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 243fbb4c47ce414c891291d15cbc4385, State: Initialized, Role: FOLLOWER
I20250625 13:57:54.098074 15909 consensus_queue.cc:260] T f7fd7b8c67da4c70a78bc7f0e929d942 P 243fbb4c47ce414c891291d15cbc4385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8da515f6534c9ca2a7a343feaa4347" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44687 } } peers { permanent_uuid: "243fbb4c47ce414c891291d15cbc4385" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 45999 } } peers { permanent_uuid: "8f5d2e56dd7d46618bf486a4d1fc8ff1" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 41417 } }
I20250625 13:57:54.102501 15909 ts_tablet_manager.cc:1428] T f7fd7b8c67da4c70a78bc7f0e929d942 P 243fbb4c47ce414c891291d15cbc4385: Time spent starting tablet: real 0.038s user 0.034s sys 0.002s
I20250625 13:57:54.103303 15915 raft_consensus.cc:491] T f7fd7b8c67da4c70a78bc7f0e929d942 P 243fbb4c47ce414c891291d15cbc4385 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:57:54.103725 15910 raft_consensus.cc:357] T f7fd7b8c67da4c70a78bc7f0e929d942 P ea8da515f6534c9ca2a7a343feaa4347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8da515f6534c9ca2a7a343feaa4347" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44687 } } peers { permanent_uuid: "243fbb4c47ce414c891291d15cbc4385" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 45999 } } peers { permanent_uuid: "8f5d2e56dd7d46618bf486a4d1fc8ff1" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 41417 } }
I20250625 13:57:54.104606 15910 raft_consensus.cc:383] T f7fd7b8c67da4c70a78bc7f0e929d942 P ea8da515f6534c9ca2a7a343feaa4347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:57:54.104911 15910 raft_consensus.cc:738] T f7fd7b8c67da4c70a78bc7f0e929d942 P ea8da515f6534c9ca2a7a343feaa4347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea8da515f6534c9ca2a7a343feaa4347, State: Initialized, Role: FOLLOWER
I20250625 13:57:54.105818 15910 consensus_queue.cc:260] T f7fd7b8c67da4c70a78bc7f0e929d942 P ea8da515f6534c9ca2a7a343feaa4347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8da515f6534c9ca2a7a343feaa4347" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44687 } } peers { permanent_uuid: "243fbb4c47ce414c891291d15cbc4385" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 45999 } } peers { permanent_uuid: "8f5d2e56dd7d46618bf486a4d1fc8ff1" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 41417 } }
I20250625 13:57:54.108912 15913 tablet_bootstrap.cc:492] T f7fd7b8c67da4c70a78bc7f0e929d942 P 8f5d2e56dd7d46618bf486a4d1fc8ff1: No bootstrap required, opened a new log
I20250625 13:57:54.110044 15910 ts_tablet_manager.cc:1428] T f7fd7b8c67da4c70a78bc7f0e929d942 P ea8da515f6534c9ca2a7a343feaa4347: Time spent starting tablet: real 0.037s user 0.030s sys 0.004s
I20250625 13:57:54.109841 15915 raft_consensus.cc:513] T f7fd7b8c67da4c70a78bc7f0e929d942 P 243fbb4c47ce414c891291d15cbc4385 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8da515f6534c9ca2a7a343feaa4347" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44687 } } peers { permanent_uuid: "243fbb4c47ce414c891291d15cbc4385" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 45999 } } peers { permanent_uuid: "8f5d2e56dd7d46618bf486a4d1fc8ff1" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 41417 } }
I20250625 13:57:54.111531 15913 ts_tablet_manager.cc:1397] T f7fd7b8c67da4c70a78bc7f0e929d942 P 8f5d2e56dd7d46618bf486a4d1fc8ff1: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.020s
I20250625 13:57:54.127398 15915 leader_election.cc:290] T f7fd7b8c67da4c70a78bc7f0e929d942 P 243fbb4c47ce414c891291d15cbc4385 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea8da515f6534c9ca2a7a343feaa4347 (127.13.155.193:44687), 8f5d2e56dd7d46618bf486a4d1fc8ff1 (127.13.155.195:41417)
I20250625 13:57:54.135982 15578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7fd7b8c67da4c70a78bc7f0e929d942" candidate_uuid: "243fbb4c47ce414c891291d15cbc4385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea8da515f6534c9ca2a7a343feaa4347" is_pre_election: true
I20250625 13:57:54.136888 15578 raft_consensus.cc:2466] T f7fd7b8c67da4c70a78bc7f0e929d942 P ea8da515f6534c9ca2a7a343feaa4347 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 243fbb4c47ce414c891291d15cbc4385 in term 0.
I20250625 13:57:54.138557 15646 leader_election.cc:304] T f7fd7b8c67da4c70a78bc7f0e929d942 P 243fbb4c47ce414c891291d15cbc4385 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 243fbb4c47ce414c891291d15cbc4385, ea8da515f6534c9ca2a7a343feaa4347; no voters:
I20250625 13:57:54.139657 15915 raft_consensus.cc:2802] T f7fd7b8c67da4c70a78bc7f0e929d942 P 243fbb4c47ce414c891291d15cbc4385 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:57:54.140244 15915 raft_consensus.cc:491] T f7fd7b8c67da4c70a78bc7f0e929d942 P 243fbb4c47ce414c891291d15cbc4385 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:57:54.140631 15915 raft_consensus.cc:3058] T f7fd7b8c67da4c70a78bc7f0e929d942 P 243fbb4c47ce414c891291d15cbc4385 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:57:54.141551 15845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7fd7b8c67da4c70a78bc7f0e929d942" candidate_uuid: "243fbb4c47ce414c891291d15cbc4385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f5d2e56dd7d46618bf486a4d1fc8ff1" is_pre_election: true
I20250625 13:57:54.142712 15913 raft_consensus.cc:357] T f7fd7b8c67da4c70a78bc7f0e929d942 P 8f5d2e56dd7d46618bf486a4d1fc8ff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8da515f6534c9ca2a7a343feaa4347" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44687 } } peers { permanent_uuid: "243fbb4c47ce414c891291d15cbc4385" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 45999 } } peers { permanent_uuid: "8f5d2e56dd7d46618bf486a4d1fc8ff1" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 41417 } }
I20250625 13:57:54.143666 15913 raft_consensus.cc:383] T f7fd7b8c67da4c70a78bc7f0e929d942 P 8f5d2e56dd7d46618bf486a4d1fc8ff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250625 13:57:54.143848 15646 leader_election.cc:343] T f7fd7b8c67da4c70a78bc7f0e929d942 P 243fbb4c47ce414c891291d15cbc4385 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f5d2e56dd7d46618bf486a4d1fc8ff1 (127.13.155.195:41417): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:57:54.143985 15913 raft_consensus.cc:738] T f7fd7b8c67da4c70a78bc7f0e929d942 P 8f5d2e56dd7d46618bf486a4d1fc8ff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f5d2e56dd7d46618bf486a4d1fc8ff1, State: Initialized, Role: FOLLOWER
I20250625 13:57:54.144799 15913 consensus_queue.cc:260] T f7fd7b8c67da4c70a78bc7f0e929d942 P 8f5d2e56dd7d46618bf486a4d1fc8ff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8da515f6534c9ca2a7a343feaa4347" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44687 } } peers { permanent_uuid: "243fbb4c47ce414c891291d15cbc4385" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 45999 } } peers { permanent_uuid: "8f5d2e56dd7d46618bf486a4d1fc8ff1" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 41417 } }
I20250625 13:57:54.148027 15915 raft_consensus.cc:513] T f7fd7b8c67da4c70a78bc7f0e929d942 P 243fbb4c47ce414c891291d15cbc4385 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8da515f6534c9ca2a7a343feaa4347" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44687 } } peers { permanent_uuid: "243fbb4c47ce414c891291d15cbc4385" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 45999 } } peers { permanent_uuid: "8f5d2e56dd7d46618bf486a4d1fc8ff1" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 41417 } }
I20250625 13:57:54.151433 15915 leader_election.cc:290] T f7fd7b8c67da4c70a78bc7f0e929d942 P 243fbb4c47ce414c891291d15cbc4385 [CANDIDATE]: Term 1 election: Requested vote from peers ea8da515f6534c9ca2a7a343feaa4347 (127.13.155.193:44687), 8f5d2e56dd7d46618bf486a4d1fc8ff1 (127.13.155.195:41417)
I20250625 13:57:54.153002 15890 heartbeater.cc:499] Master 127.13.155.254:36053 was elected leader, sending a full tablet report...
I20250625 13:57:54.154042 15913 ts_tablet_manager.cc:1428] T f7fd7b8c67da4c70a78bc7f0e929d942 P 8f5d2e56dd7d46618bf486a4d1fc8ff1: Time spent starting tablet: real 0.042s user 0.034s sys 0.005s
I20250625 13:57:54.156219 15578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7fd7b8c67da4c70a78bc7f0e929d942" candidate_uuid: "243fbb4c47ce414c891291d15cbc4385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea8da515f6534c9ca2a7a343feaa4347"
I20250625 13:57:54.157618 15578 raft_consensus.cc:3058] T f7fd7b8c67da4c70a78bc7f0e929d942 P ea8da515f6534c9ca2a7a343feaa4347 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:57:54.159065 15845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7fd7b8c67da4c70a78bc7f0e929d942" candidate_uuid: "243fbb4c47ce414c891291d15cbc4385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f5d2e56dd7d46618bf486a4d1fc8ff1"
I20250625 13:57:54.160286 15845 raft_consensus.cc:3058] T f7fd7b8c67da4c70a78bc7f0e929d942 P 8f5d2e56dd7d46618bf486a4d1fc8ff1 [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:57:54.165678 15624 tablet.cc:2378] T f7fd7b8c67da4c70a78bc7f0e929d942 P ea8da515f6534c9ca2a7a343feaa4347: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 13:57:54.166534 15578 raft_consensus.cc:2466] T f7fd7b8c67da4c70a78bc7f0e929d942 P ea8da515f6534c9ca2a7a343feaa4347 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 243fbb4c47ce414c891291d15cbc4385 in term 1.
I20250625 13:57:54.167627 15845 raft_consensus.cc:2466] T f7fd7b8c67da4c70a78bc7f0e929d942 P 8f5d2e56dd7d46618bf486a4d1fc8ff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 243fbb4c47ce414c891291d15cbc4385 in term 1.
I20250625 13:57:54.168359 15646 leader_election.cc:304] T f7fd7b8c67da4c70a78bc7f0e929d942 P 243fbb4c47ce414c891291d15cbc4385 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 243fbb4c47ce414c891291d15cbc4385, ea8da515f6534c9ca2a7a343feaa4347; no voters:
I20250625 13:57:54.169175 15915 raft_consensus.cc:2802] T f7fd7b8c67da4c70a78bc7f0e929d942 P 243fbb4c47ce414c891291d15cbc4385 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:57:54.169705 15915 raft_consensus.cc:695] T f7fd7b8c67da4c70a78bc7f0e929d942 P 243fbb4c47ce414c891291d15cbc4385 [term 1 LEADER]: Becoming Leader. State: Replica: 243fbb4c47ce414c891291d15cbc4385, State: Running, Role: LEADER
I20250625 13:57:54.170692 15915 consensus_queue.cc:237] T f7fd7b8c67da4c70a78bc7f0e929d942 P 243fbb4c47ce414c891291d15cbc4385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8da515f6534c9ca2a7a343feaa4347" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44687 } } peers { permanent_uuid: "243fbb4c47ce414c891291d15cbc4385" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 45999 } } peers { permanent_uuid: "8f5d2e56dd7d46618bf486a4d1fc8ff1" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 41417 } }
I20250625 13:57:54.185613 15435 catalog_manager.cc:5582] T f7fd7b8c67da4c70a78bc7f0e929d942 P 243fbb4c47ce414c891291d15cbc4385 reported cstate change: term changed from 0 to 1, leader changed from <none> to 243fbb4c47ce414c891291d15cbc4385 (127.13.155.194). New cstate: current_term: 1 leader_uuid: "243fbb4c47ce414c891291d15cbc4385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8da515f6534c9ca2a7a343feaa4347" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "243fbb4c47ce414c891291d15cbc4385" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 45999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f5d2e56dd7d46618bf486a4d1fc8ff1" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 41417 } health_report { overall_health: UNKNOWN } } }
I20250625 13:57:54.246398 13935 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 13:57:54.249825 13935 ts_itest-base.cc:246] Waiting for 1 tablets on tserver ea8da515f6534c9ca2a7a343feaa4347 to finish bootstrapping
W20250625 13:57:54.253151 15757 tablet.cc:2378] T f7fd7b8c67da4c70a78bc7f0e929d942 P 243fbb4c47ce414c891291d15cbc4385: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 13:57:54.263260 13935 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 243fbb4c47ce414c891291d15cbc4385 to finish bootstrapping
I20250625 13:57:54.274053 13935 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 8f5d2e56dd7d46618bf486a4d1fc8ff1 to finish bootstrapping
I20250625 13:57:54.287601 15435 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52202:
name: "TestAnotherTable"
schema {
columns {
name: "foo"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "bar"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
comment: "comment for bar"
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "foo"
}
}
}
W20250625 13:57:54.289156 15435 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAnotherTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:57:54.305964 15558 tablet_service.cc:1468] Processing CreateTablet for tablet 305807b92fd940a7ba4422be7b83c65d (DEFAULT_TABLE table=TestAnotherTable [id=e23bad5187a04d0cbc80127f91b4a402]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250625 13:57:54.306399 15691 tablet_service.cc:1468] Processing CreateTablet for tablet 305807b92fd940a7ba4422be7b83c65d (DEFAULT_TABLE table=TestAnotherTable [id=e23bad5187a04d0cbc80127f91b4a402]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250625 13:57:54.307121 15558 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 305807b92fd940a7ba4422be7b83c65d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:57:54.307511 15691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 305807b92fd940a7ba4422be7b83c65d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:57:54.307269 15825 tablet_service.cc:1468] Processing CreateTablet for tablet 305807b92fd940a7ba4422be7b83c65d (DEFAULT_TABLE table=TestAnotherTable [id=e23bad5187a04d0cbc80127f91b4a402]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250625 13:57:54.308307 15825 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 305807b92fd940a7ba4422be7b83c65d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:57:54.318792 15910 tablet_bootstrap.cc:492] T 305807b92fd940a7ba4422be7b83c65d P ea8da515f6534c9ca2a7a343feaa4347: Bootstrap starting.
I20250625 13:57:54.322160 15909 tablet_bootstrap.cc:492] T 305807b92fd940a7ba4422be7b83c65d P 243fbb4c47ce414c891291d15cbc4385: Bootstrap starting.
I20250625 13:57:54.322418 15913 tablet_bootstrap.cc:492] T 305807b92fd940a7ba4422be7b83c65d P 8f5d2e56dd7d46618bf486a4d1fc8ff1: Bootstrap starting.
I20250625 13:57:54.326849 15910 tablet_bootstrap.cc:654] T 305807b92fd940a7ba4422be7b83c65d P ea8da515f6534c9ca2a7a343feaa4347: Neither blocks nor log segments found. Creating new log.
I20250625 13:57:54.327091 15909 tablet_bootstrap.cc:654] T 305807b92fd940a7ba4422be7b83c65d P 243fbb4c47ce414c891291d15cbc4385: Neither blocks nor log segments found. Creating new log.
I20250625 13:57:54.328730 15913 tablet_bootstrap.cc:654] T 305807b92fd940a7ba4422be7b83c65d P 8f5d2e56dd7d46618bf486a4d1fc8ff1: Neither blocks nor log segments found. Creating new log.
I20250625 13:57:54.334230 15910 tablet_bootstrap.cc:492] T 305807b92fd940a7ba4422be7b83c65d P ea8da515f6534c9ca2a7a343feaa4347: No bootstrap required, opened a new log
I20250625 13:57:54.334537 15909 tablet_bootstrap.cc:492] T 305807b92fd940a7ba4422be7b83c65d P 243fbb4c47ce414c891291d15cbc4385: No bootstrap required, opened a new log
I20250625 13:57:54.334797 15910 ts_tablet_manager.cc:1397] T 305807b92fd940a7ba4422be7b83c65d P ea8da515f6534c9ca2a7a343feaa4347: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.004s
I20250625 13:57:54.335009 15909 ts_tablet_manager.cc:1397] T 305807b92fd940a7ba4422be7b83c65d P 243fbb4c47ce414c891291d15cbc4385: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20250625 13:57:54.335482 15913 tablet_bootstrap.cc:492] T 305807b92fd940a7ba4422be7b83c65d P 8f5d2e56dd7d46618bf486a4d1fc8ff1: No bootstrap required, opened a new log
I20250625 13:57:54.335909 15913 ts_tablet_manager.cc:1397] T 305807b92fd940a7ba4422be7b83c65d P 8f5d2e56dd7d46618bf486a4d1fc8ff1: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20250625 13:57:54.337319 15910 raft_consensus.cc:357] T 305807b92fd940a7ba4422be7b83c65d P ea8da515f6534c9ca2a7a343feaa4347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8da515f6534c9ca2a7a343feaa4347" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44687 } } peers { permanent_uuid: "243fbb4c47ce414c891291d15cbc4385" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 45999 } } peers { permanent_uuid: "8f5d2e56dd7d46618bf486a4d1fc8ff1" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 41417 } }
I20250625 13:57:54.337711 15909 raft_consensus.cc:357] T 305807b92fd940a7ba4422be7b83c65d P 243fbb4c47ce414c891291d15cbc4385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8da515f6534c9ca2a7a343feaa4347" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44687 } } peers { permanent_uuid: "243fbb4c47ce414c891291d15cbc4385" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 45999 } } peers { permanent_uuid: "8f5d2e56dd7d46618bf486a4d1fc8ff1" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 41417 } }
I20250625 13:57:54.338148 15910 raft_consensus.cc:383] T 305807b92fd940a7ba4422be7b83c65d P ea8da515f6534c9ca2a7a343feaa4347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:57:54.338487 15910 raft_consensus.cc:738] T 305807b92fd940a7ba4422be7b83c65d P ea8da515f6534c9ca2a7a343feaa4347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea8da515f6534c9ca2a7a343feaa4347, State: Initialized, Role: FOLLOWER
I20250625 13:57:54.338486 15909 raft_consensus.cc:383] T 305807b92fd940a7ba4422be7b83c65d P 243fbb4c47ce414c891291d15cbc4385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:57:54.338789 15909 raft_consensus.cc:738] T 305807b92fd940a7ba4422be7b83c65d P 243fbb4c47ce414c891291d15cbc4385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 243fbb4c47ce414c891291d15cbc4385, State: Initialized, Role: FOLLOWER
I20250625 13:57:54.338516 15913 raft_consensus.cc:357] T 305807b92fd940a7ba4422be7b83c65d P 8f5d2e56dd7d46618bf486a4d1fc8ff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8da515f6534c9ca2a7a343feaa4347" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44687 } } peers { permanent_uuid: "243fbb4c47ce414c891291d15cbc4385" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 45999 } } peers { permanent_uuid: "8f5d2e56dd7d46618bf486a4d1fc8ff1" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 41417 } }
I20250625 13:57:54.339234 15913 raft_consensus.cc:383] T 305807b92fd940a7ba4422be7b83c65d P 8f5d2e56dd7d46618bf486a4d1fc8ff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:57:54.339152 15910 consensus_queue.cc:260] T 305807b92fd940a7ba4422be7b83c65d P ea8da515f6534c9ca2a7a343feaa4347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8da515f6534c9ca2a7a343feaa4347" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44687 } } peers { permanent_uuid: "243fbb4c47ce414c891291d15cbc4385" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 45999 } } peers { permanent_uuid: "8f5d2e56dd7d46618bf486a4d1fc8ff1" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 41417 } }
I20250625 13:57:54.339507 15913 raft_consensus.cc:738] T 305807b92fd940a7ba4422be7b83c65d P 8f5d2e56dd7d46618bf486a4d1fc8ff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f5d2e56dd7d46618bf486a4d1fc8ff1, State: Initialized, Role: FOLLOWER
I20250625 13:57:54.339498 15909 consensus_queue.cc:260] T 305807b92fd940a7ba4422be7b83c65d P 243fbb4c47ce414c891291d15cbc4385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8da515f6534c9ca2a7a343feaa4347" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44687 } } peers { permanent_uuid: "243fbb4c47ce414c891291d15cbc4385" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 45999 } } peers { permanent_uuid: "8f5d2e56dd7d46618bf486a4d1fc8ff1" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 41417 } }
I20250625 13:57:54.341670 15909 ts_tablet_manager.cc:1428] T 305807b92fd940a7ba4422be7b83c65d P 243fbb4c47ce414c891291d15cbc4385: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20250625 13:57:54.341742 15913 consensus_queue.cc:260] T 305807b92fd940a7ba4422be7b83c65d P 8f5d2e56dd7d46618bf486a4d1fc8ff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8da515f6534c9ca2a7a343feaa4347" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44687 } } peers { permanent_uuid: "243fbb4c47ce414c891291d15cbc4385" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 45999 } } peers { permanent_uuid: "8f5d2e56dd7d46618bf486a4d1fc8ff1" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 41417 } }
I20250625 13:57:54.343480 15910 ts_tablet_manager.cc:1428] T 305807b92fd940a7ba4422be7b83c65d P ea8da515f6534c9ca2a7a343feaa4347: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250625 13:57:54.344107 15913 ts_tablet_manager.cc:1428] T 305807b92fd940a7ba4422be7b83c65d P 8f5d2e56dd7d46618bf486a4d1fc8ff1: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
W20250625 13:57:54.361105 15891 tablet.cc:2378] T 305807b92fd940a7ba4422be7b83c65d P 8f5d2e56dd7d46618bf486a4d1fc8ff1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 13:57:54.438758 15917 raft_consensus.cc:491] T 305807b92fd940a7ba4422be7b83c65d P ea8da515f6534c9ca2a7a343feaa4347 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:57:54.439173 15917 raft_consensus.cc:513] T 305807b92fd940a7ba4422be7b83c65d P ea8da515f6534c9ca2a7a343feaa4347 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8da515f6534c9ca2a7a343feaa4347" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44687 } } peers { permanent_uuid: "243fbb4c47ce414c891291d15cbc4385" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 45999 } } peers { permanent_uuid: "8f5d2e56dd7d46618bf486a4d1fc8ff1" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 41417 } }
I20250625 13:57:54.441478 15917 leader_election.cc:290] T 305807b92fd940a7ba4422be7b83c65d P ea8da515f6534c9ca2a7a343feaa4347 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 243fbb4c47ce414c891291d15cbc4385 (127.13.155.194:45999), 8f5d2e56dd7d46618bf486a4d1fc8ff1 (127.13.155.195:41417)
I20250625 13:57:54.452438 15845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "305807b92fd940a7ba4422be7b83c65d" candidate_uuid: "ea8da515f6534c9ca2a7a343feaa4347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f5d2e56dd7d46618bf486a4d1fc8ff1" is_pre_election: true
I20250625 13:57:54.452677 15711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "305807b92fd940a7ba4422be7b83c65d" candidate_uuid: "ea8da515f6534c9ca2a7a343feaa4347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "243fbb4c47ce414c891291d15cbc4385" is_pre_election: true
I20250625 13:57:54.452919 15845 raft_consensus.cc:2466] T 305807b92fd940a7ba4422be7b83c65d P 8f5d2e56dd7d46618bf486a4d1fc8ff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea8da515f6534c9ca2a7a343feaa4347 in term 0.
I20250625 13:57:54.453317 15711 raft_consensus.cc:2466] T 305807b92fd940a7ba4422be7b83c65d P 243fbb4c47ce414c891291d15cbc4385 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea8da515f6534c9ca2a7a343feaa4347 in term 0.
I20250625 13:57:54.453922 15513 leader_election.cc:304] T 305807b92fd940a7ba4422be7b83c65d P ea8da515f6534c9ca2a7a343feaa4347 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f5d2e56dd7d46618bf486a4d1fc8ff1, ea8da515f6534c9ca2a7a343feaa4347; no voters:
I20250625 13:57:54.454687 15917 raft_consensus.cc:2802] T 305807b92fd940a7ba4422be7b83c65d P ea8da515f6534c9ca2a7a343feaa4347 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:57:54.455000 15917 raft_consensus.cc:491] T 305807b92fd940a7ba4422be7b83c65d P ea8da515f6534c9ca2a7a343feaa4347 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:57:54.455250 15917 raft_consensus.cc:3058] T 305807b92fd940a7ba4422be7b83c65d P ea8da515f6534c9ca2a7a343feaa4347 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:57:54.459481 15917 raft_consensus.cc:513] T 305807b92fd940a7ba4422be7b83c65d P ea8da515f6534c9ca2a7a343feaa4347 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8da515f6534c9ca2a7a343feaa4347" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44687 } } peers { permanent_uuid: "243fbb4c47ce414c891291d15cbc4385" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 45999 } } peers { permanent_uuid: "8f5d2e56dd7d46618bf486a4d1fc8ff1" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 41417 } }
I20250625 13:57:54.460841 15917 leader_election.cc:290] T 305807b92fd940a7ba4422be7b83c65d P ea8da515f6534c9ca2a7a343feaa4347 [CANDIDATE]: Term 1 election: Requested vote from peers 243fbb4c47ce414c891291d15cbc4385 (127.13.155.194:45999), 8f5d2e56dd7d46618bf486a4d1fc8ff1 (127.13.155.195:41417)
I20250625 13:57:54.461702 15711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "305807b92fd940a7ba4422be7b83c65d" candidate_uuid: "ea8da515f6534c9ca2a7a343feaa4347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "243fbb4c47ce414c891291d15cbc4385"
I20250625 13:57:54.461853 15845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "305807b92fd940a7ba4422be7b83c65d" candidate_uuid: "ea8da515f6534c9ca2a7a343feaa4347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f5d2e56dd7d46618bf486a4d1fc8ff1"
I20250625 13:57:54.462113 15711 raft_consensus.cc:3058] T 305807b92fd940a7ba4422be7b83c65d P 243fbb4c47ce414c891291d15cbc4385 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:57:54.462311 15845 raft_consensus.cc:3058] T 305807b92fd940a7ba4422be7b83c65d P 8f5d2e56dd7d46618bf486a4d1fc8ff1 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:57:54.466216 15845 raft_consensus.cc:2466] T 305807b92fd940a7ba4422be7b83c65d P 8f5d2e56dd7d46618bf486a4d1fc8ff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea8da515f6534c9ca2a7a343feaa4347 in term 1.
I20250625 13:57:54.466324 15711 raft_consensus.cc:2466] T 305807b92fd940a7ba4422be7b83c65d P 243fbb4c47ce414c891291d15cbc4385 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea8da515f6534c9ca2a7a343feaa4347 in term 1.
I20250625 13:57:54.467080 15513 leader_election.cc:304] T 305807b92fd940a7ba4422be7b83c65d P ea8da515f6534c9ca2a7a343feaa4347 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f5d2e56dd7d46618bf486a4d1fc8ff1, ea8da515f6534c9ca2a7a343feaa4347; no voters:
I20250625 13:57:54.467856 15917 raft_consensus.cc:2802] T 305807b92fd940a7ba4422be7b83c65d P ea8da515f6534c9ca2a7a343feaa4347 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:57:54.469302 15917 raft_consensus.cc:695] T 305807b92fd940a7ba4422be7b83c65d P ea8da515f6534c9ca2a7a343feaa4347 [term 1 LEADER]: Becoming Leader. State: Replica: ea8da515f6534c9ca2a7a343feaa4347, State: Running, Role: LEADER
I20250625 13:57:54.470019 15917 consensus_queue.cc:237] T 305807b92fd940a7ba4422be7b83c65d P ea8da515f6534c9ca2a7a343feaa4347 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8da515f6534c9ca2a7a343feaa4347" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44687 } } peers { permanent_uuid: "243fbb4c47ce414c891291d15cbc4385" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 45999 } } peers { permanent_uuid: "8f5d2e56dd7d46618bf486a4d1fc8ff1" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 41417 } }
I20250625 13:57:54.480832 15434 catalog_manager.cc:5582] T 305807b92fd940a7ba4422be7b83c65d P ea8da515f6534c9ca2a7a343feaa4347 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea8da515f6534c9ca2a7a343feaa4347 (127.13.155.193). New cstate: current_term: 1 leader_uuid: "ea8da515f6534c9ca2a7a343feaa4347" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8da515f6534c9ca2a7a343feaa4347" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "243fbb4c47ce414c891291d15cbc4385" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 45999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f5d2e56dd7d46618bf486a4d1fc8ff1" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 41417 } health_report { overall_health: UNKNOWN } } }
I20250625 13:57:54.613648 15915 consensus_queue.cc:1035] T f7fd7b8c67da4c70a78bc7f0e929d942 P 243fbb4c47ce414c891291d15cbc4385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea8da515f6534c9ca2a7a343feaa4347" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:57:54.634488 15916 consensus_queue.cc:1035] T f7fd7b8c67da4c70a78bc7f0e929d942 P 243fbb4c47ce414c891291d15cbc4385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f5d2e56dd7d46618bf486a4d1fc8ff1" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 41417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250625 13:57:54.826032 15930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:57:54.826669 15930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:57:54.859112 15930 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250625 13:57:54.904875 15917 consensus_queue.cc:1035] T 305807b92fd940a7ba4422be7b83c65d P ea8da515f6534c9ca2a7a343feaa4347 [LEADER]: Connected to new peer: Peer: permanent_uuid: "243fbb4c47ce414c891291d15cbc4385" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250625 13:57:54.937618 15917 consensus_queue.cc:1035] T 305807b92fd940a7ba4422be7b83c65d P ea8da515f6534c9ca2a7a343feaa4347 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f5d2e56dd7d46618bf486a4d1fc8ff1" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 41417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250625 13:57:56.220002 15930 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.318s user 0.000s sys 0.004s
W20250625 13:57:56.220459 15930 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.319s user 0.000s sys 0.004s
W20250625 13:57:57.612733 15954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:57:57.613400 15954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:57:57.646368 15954 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250625 13:57:58.979950 15954 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.291s user 0.494s sys 0.793s
W20250625 13:57:58.980363 15954 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.291s user 0.494s sys 0.793s
W20250625 13:58:00.359289 15970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:58:00.359894 15970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:58:00.390491 15970 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250625 13:58:01.685225 15970 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.254s user 0.496s sys 0.755s
W20250625 13:58:01.685657 15970 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.254s user 0.496s sys 0.755s
W20250625 13:58:03.093361 15986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:58:03.094044 15986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:58:03.132041 15986 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250625 13:58:04.442509 15986 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.267s user 0.469s sys 0.794s
W20250625 13:58:04.442930 15986 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.268s user 0.469s sys 0.798s
I20250625 13:58:05.523415 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 15494
I20250625 13:58:05.550791 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 15627
I20250625 13:58:05.577466 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 15760
I20250625 13:58:05.602099 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 15402
2025-06-25T13:58:05Z chronyd exiting
[ OK ] AdminCliTest.TestDescribeTableColumnFlags (19320 ms)
[ RUN ] AdminCliTest.TestAuthzResetCacheNotAuthorized
I20250625 13:58:05.662322 13935 test_util.cc:276] Using random seed: -712421289
I20250625 13:58:05.666374 13935 ts_itest-base.cc:115] Starting cluster with:
I20250625 13:58:05.666601 13935 ts_itest-base.cc:116] --------------
I20250625 13:58:05.666723 13935 ts_itest-base.cc:117] 3 tablet servers
I20250625 13:58:05.666831 13935 ts_itest-base.cc:118] 3 replicas per TS
I20250625 13:58:05.666944 13935 ts_itest-base.cc:119] --------------
2025-06-25T13:58:05Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-25T13:58:05Z Disabled control of system clock
I20250625 13:58:05.703178 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.155.254:43607
--webserver_interface=127.13.155.254
--webserver_port=0
--builtin_ntp_servers=127.13.155.212:44737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.155.254:43607
--superuser_acl=no-such-user with env {}
W20250625 13:58:05.997530 16009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:58:05.998077 16009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:58:05.998540 16009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:58:06.029165 16009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 13:58:06.029465 16009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:58:06.029680 16009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 13:58:06.029878 16009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 13:58:06.065775 16009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:44737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.155.254:43607
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.155.254:43607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.13.155.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:58:06.067029 16009 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:58:06.068599 16009 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:58:06.084434 16016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:58:06.084445 16018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:58:06.084592 16015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:58:06.086086 16009 server_base.cc:1048] running on GCE node
I20250625 13:58:07.262418 16009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:58:07.265142 16009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:58:07.266547 16009 hybrid_clock.cc:648] HybridClock initialized: now 1750859887266508 us; error 46 us; skew 500 ppm
I20250625 13:58:07.267359 16009 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:58:07.276903 16009 webserver.cc:469] Webserver started at http://127.13.155.254:38463/ using document root <none> and password file <none>
I20250625 13:58:07.277859 16009 fs_manager.cc:362] Metadata directory not provided
I20250625 13:58:07.278050 16009 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:58:07.278479 16009 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:58:07.282845 16009 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "1a64e4b13bb241f39d60fa08ce9dfe48"
format_stamp: "Formatted at 2025-06-25 13:58:07 on dist-test-slave-k028"
I20250625 13:58:07.283872 16009 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "1a64e4b13bb241f39d60fa08ce9dfe48"
format_stamp: "Formatted at 2025-06-25 13:58:07 on dist-test-slave-k028"
I20250625 13:58:07.290784 16009 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20250625 13:58:07.295974 16025 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:58:07.296953 16009 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250625 13:58:07.297258 16009 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
uuid: "1a64e4b13bb241f39d60fa08ce9dfe48"
format_stamp: "Formatted at 2025-06-25 13:58:07 on dist-test-slave-k028"
I20250625 13:58:07.297564 16009 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:58:07.379895 16009 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:58:07.381377 16009 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:58:07.381829 16009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:58:07.450423 16009 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.254:43607
I20250625 13:58:07.450536 16076 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.254:43607 every 8 connection(s)
I20250625 13:58:07.453159 16009 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250625 13:58:07.458202 16077 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:58:07.461513 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 16009
I20250625 13:58:07.461918 13935 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250625 13:58:07.480342 16077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a64e4b13bb241f39d60fa08ce9dfe48: Bootstrap starting.
I20250625 13:58:07.486227 16077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a64e4b13bb241f39d60fa08ce9dfe48: Neither blocks nor log segments found. Creating new log.
I20250625 13:58:07.487947 16077 log.cc:826] T 00000000000000000000000000000000 P 1a64e4b13bb241f39d60fa08ce9dfe48: Log is configured to *not* fsync() on all Append() calls
I20250625 13:58:07.492388 16077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a64e4b13bb241f39d60fa08ce9dfe48: No bootstrap required, opened a new log
I20250625 13:58:07.509814 16077 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1a64e4b13bb241f39d60fa08ce9dfe48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a64e4b13bb241f39d60fa08ce9dfe48" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 43607 } }
I20250625 13:58:07.510485 16077 raft_consensus.cc:383] T 00000000000000000000000000000000 P 1a64e4b13bb241f39d60fa08ce9dfe48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:58:07.510740 16077 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1a64e4b13bb241f39d60fa08ce9dfe48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a64e4b13bb241f39d60fa08ce9dfe48, State: Initialized, Role: FOLLOWER
I20250625 13:58:07.511443 16077 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a64e4b13bb241f39d60fa08ce9dfe48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a64e4b13bb241f39d60fa08ce9dfe48" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 43607 } }
I20250625 13:58:07.511991 16077 raft_consensus.cc:397] T 00000000000000000000000000000000 P 1a64e4b13bb241f39d60fa08ce9dfe48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 13:58:07.512264 16077 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1a64e4b13bb241f39d60fa08ce9dfe48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 13:58:07.512588 16077 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1a64e4b13bb241f39d60fa08ce9dfe48 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:58:07.516956 16077 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1a64e4b13bb241f39d60fa08ce9dfe48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a64e4b13bb241f39d60fa08ce9dfe48" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 43607 } }
I20250625 13:58:07.517633 16077 leader_election.cc:304] T 00000000000000000000000000000000 P 1a64e4b13bb241f39d60fa08ce9dfe48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a64e4b13bb241f39d60fa08ce9dfe48; no voters:
I20250625 13:58:07.519274 16077 leader_election.cc:290] T 00000000000000000000000000000000 P 1a64e4b13bb241f39d60fa08ce9dfe48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 13:58:07.519985 16082 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1a64e4b13bb241f39d60fa08ce9dfe48 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:58:07.522094 16082 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1a64e4b13bb241f39d60fa08ce9dfe48 [term 1 LEADER]: Becoming Leader. State: Replica: 1a64e4b13bb241f39d60fa08ce9dfe48, State: Running, Role: LEADER
I20250625 13:58:07.522884 16082 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a64e4b13bb241f39d60fa08ce9dfe48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a64e4b13bb241f39d60fa08ce9dfe48" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 43607 } }
I20250625 13:58:07.523841 16077 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a64e4b13bb241f39d60fa08ce9dfe48 [sys.catalog]: configured and running, proceeding with master startup.
I20250625 13:58:07.533735 16084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a64e4b13bb241f39d60fa08ce9dfe48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a64e4b13bb241f39d60fa08ce9dfe48. Latest consensus state: current_term: 1 leader_uuid: "1a64e4b13bb241f39d60fa08ce9dfe48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a64e4b13bb241f39d60fa08ce9dfe48" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 43607 } } }
I20250625 13:58:07.534232 16084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a64e4b13bb241f39d60fa08ce9dfe48 [sys.catalog]: This master's current role is: LEADER
I20250625 13:58:07.534583 16083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a64e4b13bb241f39d60fa08ce9dfe48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a64e4b13bb241f39d60fa08ce9dfe48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a64e4b13bb241f39d60fa08ce9dfe48" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 43607 } } }
I20250625 13:58:07.535260 16083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a64e4b13bb241f39d60fa08ce9dfe48 [sys.catalog]: This master's current role is: LEADER
I20250625 13:58:07.537789 16090 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 13:58:07.549018 16090 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 13:58:07.565819 16090 catalog_manager.cc:1349] Generated new cluster ID: 04dec9e6cf0f407e84285a55595faed9
I20250625 13:58:07.566123 16090 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 13:58:07.601626 16090 catalog_manager.cc:1372] Generated new certificate authority record
I20250625 13:58:07.603143 16090 catalog_manager.cc:1506] Loading token signing keys...
I20250625 13:58:07.620483 16090 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 1a64e4b13bb241f39d60fa08ce9dfe48: Generated new TSK 0
I20250625 13:58:07.621316 16090 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 13:58:07.638964 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.155.193:0
--local_ip_for_outbound_sockets=127.13.155.193
--webserver_interface=127.13.155.193
--webserver_port=0
--tserver_master_addrs=127.13.155.254:43607
--builtin_ntp_servers=127.13.155.212:44737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250625 13:58:07.943693 16101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:58:07.944197 16101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:58:07.944698 16101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:58:07.975772 16101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:58:07.976621 16101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.155.193
I20250625 13:58:08.011566 16101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:44737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.155.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.13.155.193
--webserver_port=0
--tserver_master_addrs=127.13.155.254:43607
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.155.193
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:58:08.012867 16101 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:58:08.014554 16101 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:58:08.032979 16108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:58:08.033636 16107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:58:08.034631 16101 server_base.cc:1048] running on GCE node
W20250625 13:58:08.036368 16110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:58:09.241322 16101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:58:09.244158 16101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:58:09.245669 16101 hybrid_clock.cc:648] HybridClock initialized: now 1750859889245604 us; error 83 us; skew 500 ppm
I20250625 13:58:09.246572 16101 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:58:09.253499 16101 webserver.cc:469] Webserver started at http://127.13.155.193:36079/ using document root <none> and password file <none>
I20250625 13:58:09.254509 16101 fs_manager.cc:362] Metadata directory not provided
I20250625 13:58:09.254753 16101 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:58:09.255227 16101 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:58:09.259784 16101 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "06a171a306f6412c8b92286bf0fbe9d3"
format_stamp: "Formatted at 2025-06-25 13:58:09 on dist-test-slave-k028"
I20250625 13:58:09.260924 16101 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "06a171a306f6412c8b92286bf0fbe9d3"
format_stamp: "Formatted at 2025-06-25 13:58:09 on dist-test-slave-k028"
I20250625 13:58:09.268585 16101 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.001s sys 0.008s
I20250625 13:58:09.274880 16117 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:58:09.276032 16101 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250625 13:58:09.276362 16101 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "06a171a306f6412c8b92286bf0fbe9d3"
format_stamp: "Formatted at 2025-06-25 13:58:09 on dist-test-slave-k028"
I20250625 13:58:09.276687 16101 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:58:09.341080 16101 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:58:09.342574 16101 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:58:09.343020 16101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:58:09.345482 16101 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 13:58:09.349596 16101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 13:58:09.349800 16101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 13:58:09.350080 16101 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 13:58:09.350243 16101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 13:58:09.486991 16101 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.193:45573
I20250625 13:58:09.487093 16229 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.193:45573 every 8 connection(s)
I20250625 13:58:09.489609 16101 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250625 13:58:09.494439 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 16101
I20250625 13:58:09.494958 13935 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250625 13:58:09.501884 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.155.194:0
--local_ip_for_outbound_sockets=127.13.155.194
--webserver_interface=127.13.155.194
--webserver_port=0
--tserver_master_addrs=127.13.155.254:43607
--builtin_ntp_servers=127.13.155.212:44737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 13:58:09.514047 16230 heartbeater.cc:344] Connected to a master server at 127.13.155.254:43607
I20250625 13:58:09.514629 16230 heartbeater.cc:461] Registering TS with master...
I20250625 13:58:09.516018 16230 heartbeater.cc:507] Master 127.13.155.254:43607 requested a full tablet report, sending...
I20250625 13:58:09.519336 16042 ts_manager.cc:194] Registered new tserver with Master: 06a171a306f6412c8b92286bf0fbe9d3 (127.13.155.193:45573)
I20250625 13:58:09.522590 16042 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.155.193:54813
W20250625 13:58:09.812852 16234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:58:09.813311 16234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:58:09.813746 16234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:58:09.844938 16234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:58:09.845779 16234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.155.194
I20250625 13:58:09.879736 16234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:44737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.155.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.13.155.194
--webserver_port=0
--tserver_master_addrs=127.13.155.254:43607
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.155.194
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:58:09.880973 16234 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:58:09.882543 16234 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:58:09.898448 16241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:58:10.527577 16230 heartbeater.cc:499] Master 127.13.155.254:43607 was elected leader, sending a full tablet report...
W20250625 13:58:09.899085 16240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:58:09.900492 16234 server_base.cc:1048] running on GCE node
W20250625 13:58:09.900848 16243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:58:11.054634 16234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:58:11.056859 16234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:58:11.058207 16234 hybrid_clock.cc:648] HybridClock initialized: now 1750859891058168 us; error 55 us; skew 500 ppm
I20250625 13:58:11.059015 16234 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:58:11.065088 16234 webserver.cc:469] Webserver started at http://127.13.155.194:32887/ using document root <none> and password file <none>
I20250625 13:58:11.066009 16234 fs_manager.cc:362] Metadata directory not provided
I20250625 13:58:11.066229 16234 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:58:11.066718 16234 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:58:11.071173 16234 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "76edee9a23e54665999832d1e8382112"
format_stamp: "Formatted at 2025-06-25 13:58:11 on dist-test-slave-k028"
I20250625 13:58:11.072221 16234 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "76edee9a23e54665999832d1e8382112"
format_stamp: "Formatted at 2025-06-25 13:58:11 on dist-test-slave-k028"
I20250625 13:58:11.079102 16234 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250625 13:58:11.084436 16250 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:58:11.085383 16234 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.002s
I20250625 13:58:11.085698 16234 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "76edee9a23e54665999832d1e8382112"
format_stamp: "Formatted at 2025-06-25 13:58:11 on dist-test-slave-k028"
I20250625 13:58:11.086019 16234 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:58:11.130265 16234 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:58:11.131711 16234 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:58:11.132133 16234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:58:11.134539 16234 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 13:58:11.138495 16234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 13:58:11.138722 16234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250625 13:58:11.138959 16234 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 13:58:11.139127 16234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 13:58:11.272457 16234 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.194:35629
I20250625 13:58:11.272564 16362 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.194:35629 every 8 connection(s)
I20250625 13:58:11.275146 16234 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250625 13:58:11.284067 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 16234
I20250625 13:58:11.284446 13935 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250625 13:58:11.290768 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.155.195:0
--local_ip_for_outbound_sockets=127.13.155.195
--webserver_interface=127.13.155.195
--webserver_port=0
--tserver_master_addrs=127.13.155.254:43607
--builtin_ntp_servers=127.13.155.212:44737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 13:58:11.296638 16363 heartbeater.cc:344] Connected to a master server at 127.13.155.254:43607
I20250625 13:58:11.297049 16363 heartbeater.cc:461] Registering TS with master...
I20250625 13:58:11.298271 16363 heartbeater.cc:507] Master 127.13.155.254:43607 requested a full tablet report, sending...
I20250625 13:58:11.300537 16042 ts_manager.cc:194] Registered new tserver with Master: 76edee9a23e54665999832d1e8382112 (127.13.155.194:35629)
I20250625 13:58:11.301725 16042 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.155.194:46667
W20250625 13:58:11.587867 16367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:58:11.588378 16367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:58:11.588886 16367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:58:11.620802 16367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:58:11.621644 16367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.155.195
I20250625 13:58:11.657630 16367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:44737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.155.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.13.155.195
--webserver_port=0
--tserver_master_addrs=127.13.155.254:43607
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.155.195
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:58:11.659078 16367 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:58:11.660804 16367 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:58:11.678071 16376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:58:12.304641 16363 heartbeater.cc:499] Master 127.13.155.254:43607 was elected leader, sending a full tablet report...
W20250625 13:58:11.678059 16374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:58:11.679914 16367 server_base.cc:1048] running on GCE node
W20250625 13:58:11.678206 16373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:58:12.849061 16367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:58:12.851996 16367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:58:12.853454 16367 hybrid_clock.cc:648] HybridClock initialized: now 1750859892853413 us; error 59 us; skew 500 ppm
I20250625 13:58:12.854292 16367 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:58:12.866151 16367 webserver.cc:469] Webserver started at http://127.13.155.195:36201/ using document root <none> and password file <none>
I20250625 13:58:12.867167 16367 fs_manager.cc:362] Metadata directory not provided
I20250625 13:58:12.867377 16367 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:58:12.867892 16367 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:58:12.872583 16367 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "c57b7e65f9924137afe59614e9fbc757"
format_stamp: "Formatted at 2025-06-25 13:58:12 on dist-test-slave-k028"
I20250625 13:58:12.873709 16367 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "c57b7e65f9924137afe59614e9fbc757"
format_stamp: "Formatted at 2025-06-25 13:58:12 on dist-test-slave-k028"
I20250625 13:58:12.881106 16367 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250625 13:58:12.886813 16384 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:58:12.887872 16367 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250625 13:58:12.888208 16367 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "c57b7e65f9924137afe59614e9fbc757"
format_stamp: "Formatted at 2025-06-25 13:58:12 on dist-test-slave-k028"
I20250625 13:58:12.888562 16367 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:58:12.952653 16367 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:58:12.954151 16367 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:58:12.954617 16367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:58:12.957111 16367 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 13:58:12.961279 16367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 13:58:12.961501 16367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 13:58:12.961760 16367 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 13:58:12.961941 16367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 13:58:13.097714 16367 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.195:36179
I20250625 13:58:13.097841 16496 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.195:36179 every 8 connection(s)
I20250625 13:58:13.100346 16367 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250625 13:58:13.106667 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 16367
I20250625 13:58:13.107203 13935 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250625 13:58:13.122540 16497 heartbeater.cc:344] Connected to a master server at 127.13.155.254:43607
I20250625 13:58:13.123041 16497 heartbeater.cc:461] Registering TS with master...
I20250625 13:58:13.124053 16497 heartbeater.cc:507] Master 127.13.155.254:43607 requested a full tablet report, sending...
I20250625 13:58:13.126577 16042 ts_manager.cc:194] Registered new tserver with Master: c57b7e65f9924137afe59614e9fbc757 (127.13.155.195:36179)
I20250625 13:58:13.127746 13935 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 13:58:13.128280 16042 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.155.195:47203
I20250625 13:58:13.163714 16042 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57954:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250625 13:58:13.182688 16042 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:58:13.233868 16165 tablet_service.cc:1468] Processing CreateTablet for tablet c6bd24aa88dd469da35b8d30e1835ebe (DEFAULT_TABLE table=TestTable [id=4021a2372c114b578202effd292ab3c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:58:13.235488 16165 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6bd24aa88dd469da35b8d30e1835ebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:58:13.235071 16432 tablet_service.cc:1468] Processing CreateTablet for tablet c6bd24aa88dd469da35b8d30e1835ebe (DEFAULT_TABLE table=TestTable [id=4021a2372c114b578202effd292ab3c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:58:13.236966 16432 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6bd24aa88dd469da35b8d30e1835ebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:58:13.246977 16298 tablet_service.cc:1468] Processing CreateTablet for tablet c6bd24aa88dd469da35b8d30e1835ebe (DEFAULT_TABLE table=TestTable [id=4021a2372c114b578202effd292ab3c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:58:13.250540 16298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6bd24aa88dd469da35b8d30e1835ebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:58:13.263190 16516 tablet_bootstrap.cc:492] T c6bd24aa88dd469da35b8d30e1835ebe P c57b7e65f9924137afe59614e9fbc757: Bootstrap starting.
I20250625 13:58:13.265566 16517 tablet_bootstrap.cc:492] T c6bd24aa88dd469da35b8d30e1835ebe P 06a171a306f6412c8b92286bf0fbe9d3: Bootstrap starting.
I20250625 13:58:13.273682 16517 tablet_bootstrap.cc:654] T c6bd24aa88dd469da35b8d30e1835ebe P 06a171a306f6412c8b92286bf0fbe9d3: Neither blocks nor log segments found. Creating new log.
I20250625 13:58:13.274859 16516 tablet_bootstrap.cc:654] T c6bd24aa88dd469da35b8d30e1835ebe P c57b7e65f9924137afe59614e9fbc757: Neither blocks nor log segments found. Creating new log.
I20250625 13:58:13.276351 16517 log.cc:826] T c6bd24aa88dd469da35b8d30e1835ebe P 06a171a306f6412c8b92286bf0fbe9d3: Log is configured to *not* fsync() on all Append() calls
I20250625 13:58:13.277603 16516 log.cc:826] T c6bd24aa88dd469da35b8d30e1835ebe P c57b7e65f9924137afe59614e9fbc757: Log is configured to *not* fsync() on all Append() calls
I20250625 13:58:13.289429 16517 tablet_bootstrap.cc:492] T c6bd24aa88dd469da35b8d30e1835ebe P 06a171a306f6412c8b92286bf0fbe9d3: No bootstrap required, opened a new log
I20250625 13:58:13.289991 16517 ts_tablet_manager.cc:1397] T c6bd24aa88dd469da35b8d30e1835ebe P 06a171a306f6412c8b92286bf0fbe9d3: Time spent bootstrapping tablet: real 0.025s user 0.013s sys 0.009s
I20250625 13:58:13.292200 16516 tablet_bootstrap.cc:492] T c6bd24aa88dd469da35b8d30e1835ebe P c57b7e65f9924137afe59614e9fbc757: No bootstrap required, opened a new log
I20250625 13:58:13.292827 16516 ts_tablet_manager.cc:1397] T c6bd24aa88dd469da35b8d30e1835ebe P c57b7e65f9924137afe59614e9fbc757: Time spent bootstrapping tablet: real 0.030s user 0.015s sys 0.007s
I20250625 13:58:13.295791 16518 tablet_bootstrap.cc:492] T c6bd24aa88dd469da35b8d30e1835ebe P 76edee9a23e54665999832d1e8382112: Bootstrap starting.
I20250625 13:58:13.303722 16518 tablet_bootstrap.cc:654] T c6bd24aa88dd469da35b8d30e1835ebe P 76edee9a23e54665999832d1e8382112: Neither blocks nor log segments found. Creating new log.
I20250625 13:58:13.306195 16518 log.cc:826] T c6bd24aa88dd469da35b8d30e1835ebe P 76edee9a23e54665999832d1e8382112: Log is configured to *not* fsync() on all Append() calls
I20250625 13:58:13.318957 16518 tablet_bootstrap.cc:492] T c6bd24aa88dd469da35b8d30e1835ebe P 76edee9a23e54665999832d1e8382112: No bootstrap required, opened a new log
I20250625 13:58:13.319497 16518 ts_tablet_manager.cc:1397] T c6bd24aa88dd469da35b8d30e1835ebe P 76edee9a23e54665999832d1e8382112: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.010s
I20250625 13:58:13.321058 16517 raft_consensus.cc:357] T c6bd24aa88dd469da35b8d30e1835ebe P 06a171a306f6412c8b92286bf0fbe9d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57b7e65f9924137afe59614e9fbc757" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36179 } } peers { permanent_uuid: "06a171a306f6412c8b92286bf0fbe9d3" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 45573 } } peers { permanent_uuid: "76edee9a23e54665999832d1e8382112" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35629 } }
I20250625 13:58:13.321996 16517 raft_consensus.cc:383] T c6bd24aa88dd469da35b8d30e1835ebe P 06a171a306f6412c8b92286bf0fbe9d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:58:13.322301 16517 raft_consensus.cc:738] T c6bd24aa88dd469da35b8d30e1835ebe P 06a171a306f6412c8b92286bf0fbe9d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06a171a306f6412c8b92286bf0fbe9d3, State: Initialized, Role: FOLLOWER
I20250625 13:58:13.323561 16517 consensus_queue.cc:260] T c6bd24aa88dd469da35b8d30e1835ebe P 06a171a306f6412c8b92286bf0fbe9d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57b7e65f9924137afe59614e9fbc757" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36179 } } peers { permanent_uuid: "06a171a306f6412c8b92286bf0fbe9d3" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 45573 } } peers { permanent_uuid: "76edee9a23e54665999832d1e8382112" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35629 } }
I20250625 13:58:13.327286 16516 raft_consensus.cc:357] T c6bd24aa88dd469da35b8d30e1835ebe P c57b7e65f9924137afe59614e9fbc757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57b7e65f9924137afe59614e9fbc757" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36179 } } peers { permanent_uuid: "06a171a306f6412c8b92286bf0fbe9d3" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 45573 } } peers { permanent_uuid: "76edee9a23e54665999832d1e8382112" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35629 } }
I20250625 13:58:13.328420 16516 raft_consensus.cc:383] T c6bd24aa88dd469da35b8d30e1835ebe P c57b7e65f9924137afe59614e9fbc757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:58:13.328848 16516 raft_consensus.cc:738] T c6bd24aa88dd469da35b8d30e1835ebe P c57b7e65f9924137afe59614e9fbc757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c57b7e65f9924137afe59614e9fbc757, State: Initialized, Role: FOLLOWER
I20250625 13:58:13.330235 16516 consensus_queue.cc:260] T c6bd24aa88dd469da35b8d30e1835ebe P c57b7e65f9924137afe59614e9fbc757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57b7e65f9924137afe59614e9fbc757" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36179 } } peers { permanent_uuid: "06a171a306f6412c8b92286bf0fbe9d3" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 45573 } } peers { permanent_uuid: "76edee9a23e54665999832d1e8382112" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35629 } }
I20250625 13:58:13.333916 16517 ts_tablet_manager.cc:1428] T c6bd24aa88dd469da35b8d30e1835ebe P 06a171a306f6412c8b92286bf0fbe9d3: Time spent starting tablet: real 0.044s user 0.023s sys 0.013s
I20250625 13:58:13.335803 16497 heartbeater.cc:499] Master 127.13.155.254:43607 was elected leader, sending a full tablet report...
I20250625 13:58:13.337278 16516 ts_tablet_manager.cc:1428] T c6bd24aa88dd469da35b8d30e1835ebe P c57b7e65f9924137afe59614e9fbc757: Time spent starting tablet: real 0.044s user 0.032s sys 0.004s
I20250625 13:58:13.345745 16518 raft_consensus.cc:357] T c6bd24aa88dd469da35b8d30e1835ebe P 76edee9a23e54665999832d1e8382112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57b7e65f9924137afe59614e9fbc757" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36179 } } peers { permanent_uuid: "06a171a306f6412c8b92286bf0fbe9d3" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 45573 } } peers { permanent_uuid: "76edee9a23e54665999832d1e8382112" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35629 } }
I20250625 13:58:13.346721 16518 raft_consensus.cc:383] T c6bd24aa88dd469da35b8d30e1835ebe P 76edee9a23e54665999832d1e8382112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:58:13.347133 16518 raft_consensus.cc:738] T c6bd24aa88dd469da35b8d30e1835ebe P 76edee9a23e54665999832d1e8382112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76edee9a23e54665999832d1e8382112, State: Initialized, Role: FOLLOWER
I20250625 13:58:13.347872 16518 consensus_queue.cc:260] T c6bd24aa88dd469da35b8d30e1835ebe P 76edee9a23e54665999832d1e8382112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57b7e65f9924137afe59614e9fbc757" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36179 } } peers { permanent_uuid: "06a171a306f6412c8b92286bf0fbe9d3" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 45573 } } peers { permanent_uuid: "76edee9a23e54665999832d1e8382112" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35629 } }
I20250625 13:58:13.350986 16518 ts_tablet_manager.cc:1428] T c6bd24aa88dd469da35b8d30e1835ebe P 76edee9a23e54665999832d1e8382112: Time spent starting tablet: real 0.031s user 0.030s sys 0.001s
W20250625 13:58:13.354944 16498 tablet.cc:2378] T c6bd24aa88dd469da35b8d30e1835ebe P c57b7e65f9924137afe59614e9fbc757: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 13:58:13.432534 16523 raft_consensus.cc:491] T c6bd24aa88dd469da35b8d30e1835ebe P c57b7e65f9924137afe59614e9fbc757 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:58:13.432997 16523 raft_consensus.cc:513] T c6bd24aa88dd469da35b8d30e1835ebe P c57b7e65f9924137afe59614e9fbc757 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57b7e65f9924137afe59614e9fbc757" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36179 } } peers { permanent_uuid: "06a171a306f6412c8b92286bf0fbe9d3" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 45573 } } peers { permanent_uuid: "76edee9a23e54665999832d1e8382112" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35629 } }
I20250625 13:58:13.435312 16523 leader_election.cc:290] T c6bd24aa88dd469da35b8d30e1835ebe P c57b7e65f9924137afe59614e9fbc757 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06a171a306f6412c8b92286bf0fbe9d3 (127.13.155.193:45573), 76edee9a23e54665999832d1e8382112 (127.13.155.194:35629)
I20250625 13:58:13.444748 16185 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6bd24aa88dd469da35b8d30e1835ebe" candidate_uuid: "c57b7e65f9924137afe59614e9fbc757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06a171a306f6412c8b92286bf0fbe9d3" is_pre_election: true
I20250625 13:58:13.445559 16185 raft_consensus.cc:2466] T c6bd24aa88dd469da35b8d30e1835ebe P 06a171a306f6412c8b92286bf0fbe9d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c57b7e65f9924137afe59614e9fbc757 in term 0.
I20250625 13:58:13.446627 16318 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6bd24aa88dd469da35b8d30e1835ebe" candidate_uuid: "c57b7e65f9924137afe59614e9fbc757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76edee9a23e54665999832d1e8382112" is_pre_election: true
I20250625 13:58:13.446764 16387 leader_election.cc:304] T c6bd24aa88dd469da35b8d30e1835ebe P c57b7e65f9924137afe59614e9fbc757 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06a171a306f6412c8b92286bf0fbe9d3, c57b7e65f9924137afe59614e9fbc757; no voters:
I20250625 13:58:13.447294 16318 raft_consensus.cc:2466] T c6bd24aa88dd469da35b8d30e1835ebe P 76edee9a23e54665999832d1e8382112 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c57b7e65f9924137afe59614e9fbc757 in term 0.
I20250625 13:58:13.447513 16523 raft_consensus.cc:2802] T c6bd24aa88dd469da35b8d30e1835ebe P c57b7e65f9924137afe59614e9fbc757 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:58:13.447856 16523 raft_consensus.cc:491] T c6bd24aa88dd469da35b8d30e1835ebe P c57b7e65f9924137afe59614e9fbc757 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:58:13.448192 16523 raft_consensus.cc:3058] T c6bd24aa88dd469da35b8d30e1835ebe P c57b7e65f9924137afe59614e9fbc757 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:58:13.453197 16523 raft_consensus.cc:513] T c6bd24aa88dd469da35b8d30e1835ebe P c57b7e65f9924137afe59614e9fbc757 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57b7e65f9924137afe59614e9fbc757" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36179 } } peers { permanent_uuid: "06a171a306f6412c8b92286bf0fbe9d3" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 45573 } } peers { permanent_uuid: "76edee9a23e54665999832d1e8382112" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35629 } }
I20250625 13:58:13.454666 16523 leader_election.cc:290] T c6bd24aa88dd469da35b8d30e1835ebe P c57b7e65f9924137afe59614e9fbc757 [CANDIDATE]: Term 1 election: Requested vote from peers 06a171a306f6412c8b92286bf0fbe9d3 (127.13.155.193:45573), 76edee9a23e54665999832d1e8382112 (127.13.155.194:35629)
I20250625 13:58:13.455541 16185 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6bd24aa88dd469da35b8d30e1835ebe" candidate_uuid: "c57b7e65f9924137afe59614e9fbc757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06a171a306f6412c8b92286bf0fbe9d3"
I20250625 13:58:13.455789 16318 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6bd24aa88dd469da35b8d30e1835ebe" candidate_uuid: "c57b7e65f9924137afe59614e9fbc757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76edee9a23e54665999832d1e8382112"
I20250625 13:58:13.455987 16185 raft_consensus.cc:3058] T c6bd24aa88dd469da35b8d30e1835ebe P 06a171a306f6412c8b92286bf0fbe9d3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:58:13.456331 16318 raft_consensus.cc:3058] T c6bd24aa88dd469da35b8d30e1835ebe P 76edee9a23e54665999832d1e8382112 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:58:13.460469 16185 raft_consensus.cc:2466] T c6bd24aa88dd469da35b8d30e1835ebe P 06a171a306f6412c8b92286bf0fbe9d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c57b7e65f9924137afe59614e9fbc757 in term 1.
I20250625 13:58:13.461381 16387 leader_election.cc:304] T c6bd24aa88dd469da35b8d30e1835ebe P c57b7e65f9924137afe59614e9fbc757 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06a171a306f6412c8b92286bf0fbe9d3, c57b7e65f9924137afe59614e9fbc757; no voters:
I20250625 13:58:13.462016 16523 raft_consensus.cc:2802] T c6bd24aa88dd469da35b8d30e1835ebe P c57b7e65f9924137afe59614e9fbc757 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:58:13.462989 16318 raft_consensus.cc:2466] T c6bd24aa88dd469da35b8d30e1835ebe P 76edee9a23e54665999832d1e8382112 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c57b7e65f9924137afe59614e9fbc757 in term 1.
I20250625 13:58:13.464082 16523 raft_consensus.cc:695] T c6bd24aa88dd469da35b8d30e1835ebe P c57b7e65f9924137afe59614e9fbc757 [term 1 LEADER]: Becoming Leader. State: Replica: c57b7e65f9924137afe59614e9fbc757, State: Running, Role: LEADER
I20250625 13:58:13.464965 16523 consensus_queue.cc:237] T c6bd24aa88dd469da35b8d30e1835ebe P c57b7e65f9924137afe59614e9fbc757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57b7e65f9924137afe59614e9fbc757" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36179 } } peers { permanent_uuid: "06a171a306f6412c8b92286bf0fbe9d3" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 45573 } } peers { permanent_uuid: "76edee9a23e54665999832d1e8382112" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35629 } }
I20250625 13:58:13.476563 16041 catalog_manager.cc:5582] T c6bd24aa88dd469da35b8d30e1835ebe P c57b7e65f9924137afe59614e9fbc757 reported cstate change: term changed from 0 to 1, leader changed from <none> to c57b7e65f9924137afe59614e9fbc757 (127.13.155.195). New cstate: current_term: 1 leader_uuid: "c57b7e65f9924137afe59614e9fbc757" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57b7e65f9924137afe59614e9fbc757" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06a171a306f6412c8b92286bf0fbe9d3" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 45573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76edee9a23e54665999832d1e8382112" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35629 } health_report { overall_health: UNKNOWN } } }
W20250625 13:58:13.498710 16231 tablet.cc:2378] T c6bd24aa88dd469da35b8d30e1835ebe P 06a171a306f6412c8b92286bf0fbe9d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250625 13:58:13.532101 16364 tablet.cc:2378] T c6bd24aa88dd469da35b8d30e1835ebe P 76edee9a23e54665999832d1e8382112: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 13:58:13.536132 13935 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 13:58:13.539574 13935 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 06a171a306f6412c8b92286bf0fbe9d3 to finish bootstrapping
I20250625 13:58:13.552626 13935 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 76edee9a23e54665999832d1e8382112 to finish bootstrapping
I20250625 13:58:13.563738 13935 ts_itest-base.cc:246] Waiting for 1 tablets on tserver c57b7e65f9924137afe59614e9fbc757 to finish bootstrapping
I20250625 13:58:13.964740 16523 consensus_queue.cc:1035] T c6bd24aa88dd469da35b8d30e1835ebe P c57b7e65f9924137afe59614e9fbc757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06a171a306f6412c8b92286bf0fbe9d3" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 45573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250625 13:58:13.978808 16523 consensus_queue.cc:1035] T c6bd24aa88dd469da35b8d30e1835ebe P c57b7e65f9924137afe59614e9fbc757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76edee9a23e54665999832d1e8382112" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 35629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250625 13:58:15.232673 16040 server_base.cc:1130] Unauthorized access attempt to method kudu.master.MasterService.RefreshAuthzCache from {username='slave'} at 127.0.0.1:51620
I20250625 13:58:16.266575 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 16101
I20250625 13:58:16.300696 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 16234
I20250625 13:58:16.325261 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 16367
I20250625 13:58:16.351704 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 16009
2025-06-25T13:58:16Z chronyd exiting
[ OK ] AdminCliTest.TestAuthzResetCacheNotAuthorized (10740 ms)
[ RUN ] AdminCliTest.TestRebuildTables
I20250625 13:58:16.403278 13935 test_util.cc:276] Using random seed: -701680334
I20250625 13:58:16.407429 13935 ts_itest-base.cc:115] Starting cluster with:
I20250625 13:58:16.407598 13935 ts_itest-base.cc:116] --------------
I20250625 13:58:16.407716 13935 ts_itest-base.cc:117] 3 tablet servers
I20250625 13:58:16.407827 13935 ts_itest-base.cc:118] 3 replicas per TS
I20250625 13:58:16.407943 13935 ts_itest-base.cc:119] --------------
2025-06-25T13:58:16Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-25T13:58:16Z Disabled control of system clock
I20250625 13:58:16.445833 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.155.254:45625
--webserver_interface=127.13.155.254
--webserver_port=0
--builtin_ntp_servers=127.13.155.212:39675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.155.254:45625 with env {}
W20250625 13:58:16.745849 16566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:58:16.746410 16566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:58:16.746928 16566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:58:16.779203 16566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 13:58:16.779517 16566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:58:16.779726 16566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 13:58:16.779920 16566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 13:58:16.816304 16566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:39675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.155.254:45625
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.155.254:45625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.13.155.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:58:16.817642 16566 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:58:16.819474 16566 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:58:16.833850 16573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:58:16.834196 16572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:58:16.836491 16566 server_base.cc:1048] running on GCE node
W20250625 13:58:16.835738 16575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:58:18.018421 16566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:58:18.021631 16566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:58:18.023068 16566 hybrid_clock.cc:648] HybridClock initialized: now 1750859898023008 us; error 76 us; skew 500 ppm
I20250625 13:58:18.023851 16566 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:58:18.035856 16566 webserver.cc:469] Webserver started at http://127.13.155.254:38733/ using document root <none> and password file <none>
I20250625 13:58:18.036958 16566 fs_manager.cc:362] Metadata directory not provided
I20250625 13:58:18.037201 16566 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:58:18.037642 16566 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:58:18.042160 16566 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "322e3b3a9cca41578d3358d15bbd13cb"
format_stamp: "Formatted at 2025-06-25 13:58:18 on dist-test-slave-k028"
I20250625 13:58:18.043318 16566 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "322e3b3a9cca41578d3358d15bbd13cb"
format_stamp: "Formatted at 2025-06-25 13:58:18 on dist-test-slave-k028"
I20250625 13:58:18.050685 16566 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20250625 13:58:18.056510 16582 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:58:18.057670 16566 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250625 13:58:18.058030 16566 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
uuid: "322e3b3a9cca41578d3358d15bbd13cb"
format_stamp: "Formatted at 2025-06-25 13:58:18 on dist-test-slave-k028"
I20250625 13:58:18.058377 16566 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:58:18.115504 16566 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:58:18.116991 16566 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:58:18.117475 16566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:58:18.186172 16566 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.254:45625
I20250625 13:58:18.186244 16633 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.254:45625 every 8 connection(s)
I20250625 13:58:18.188916 16566 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250625 13:58:18.189517 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 16566
I20250625 13:58:18.189899 13935 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250625 13:58:18.195730 16634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:58:18.219484 16634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb: Bootstrap starting.
I20250625 13:58:18.225247 16634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb: Neither blocks nor log segments found. Creating new log.
I20250625 13:58:18.227232 16634 log.cc:826] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb: Log is configured to *not* fsync() on all Append() calls
I20250625 13:58:18.231564 16634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb: No bootstrap required, opened a new log
I20250625 13:58:18.248770 16634 raft_consensus.cc:357] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322e3b3a9cca41578d3358d15bbd13cb" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 45625 } }
I20250625 13:58:18.249445 16634 raft_consensus.cc:383] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:58:18.249706 16634 raft_consensus.cc:738] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 322e3b3a9cca41578d3358d15bbd13cb, State: Initialized, Role: FOLLOWER
I20250625 13:58:18.250451 16634 consensus_queue.cc:260] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322e3b3a9cca41578d3358d15bbd13cb" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 45625 } }
I20250625 13:58:18.250969 16634 raft_consensus.cc:397] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 13:58:18.251250 16634 raft_consensus.cc:491] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 13:58:18.251554 16634 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:58:18.256917 16634 raft_consensus.cc:513] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322e3b3a9cca41578d3358d15bbd13cb" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 45625 } }
I20250625 13:58:18.258006 16634 leader_election.cc:304] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 322e3b3a9cca41578d3358d15bbd13cb; no voters:
I20250625 13:58:18.259696 16634 leader_election.cc:290] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 13:58:18.260434 16639 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:58:18.262579 16639 raft_consensus.cc:695] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [term 1 LEADER]: Becoming Leader. State: Replica: 322e3b3a9cca41578d3358d15bbd13cb, State: Running, Role: LEADER
I20250625 13:58:18.263540 16639 consensus_queue.cc:237] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322e3b3a9cca41578d3358d15bbd13cb" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 45625 } }
I20250625 13:58:18.264331 16634 sys_catalog.cc:564] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [sys.catalog]: configured and running, proceeding with master startup.
I20250625 13:58:18.276976 16640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "322e3b3a9cca41578d3358d15bbd13cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322e3b3a9cca41578d3358d15bbd13cb" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 45625 } } }
I20250625 13:58:18.277876 16640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [sys.catalog]: This master's current role is: LEADER
I20250625 13:58:18.277727 16641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 322e3b3a9cca41578d3358d15bbd13cb. Latest consensus state: current_term: 1 leader_uuid: "322e3b3a9cca41578d3358d15bbd13cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322e3b3a9cca41578d3358d15bbd13cb" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 45625 } } }
I20250625 13:58:18.278324 16641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [sys.catalog]: This master's current role is: LEADER
I20250625 13:58:18.280702 16649 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 13:58:18.291913 16649 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 13:58:18.307413 16649 catalog_manager.cc:1349] Generated new cluster ID: cba9adb64c954b8993397fb3176e4ce6
I20250625 13:58:18.307667 16649 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 13:58:18.331080 16649 catalog_manager.cc:1372] Generated new certificate authority record
I20250625 13:58:18.332528 16649 catalog_manager.cc:1506] Loading token signing keys...
I20250625 13:58:18.346356 16649 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb: Generated new TSK 0
I20250625 13:58:18.347286 16649 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 13:58:18.369702 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.155.193:0
--local_ip_for_outbound_sockets=127.13.155.193
--webserver_interface=127.13.155.193
--webserver_port=0
--tserver_master_addrs=127.13.155.254:45625
--builtin_ntp_servers=127.13.155.212:39675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250625 13:58:18.667649 16658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:58:18.668151 16658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:58:18.668648 16658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:58:18.699913 16658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:58:18.700851 16658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.155.193
I20250625 13:58:18.735853 16658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:39675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.155.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.13.155.193
--webserver_port=0
--tserver_master_addrs=127.13.155.254:45625
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.155.193
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:58:18.737200 16658 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:58:18.738837 16658 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:58:18.755941 16667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:58:18.756634 16664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:58:18.756757 16665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:58:19.934217 16666 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250625 13:58:19.934208 16658 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 13:58:19.939325 16658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:58:19.942018 16658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:58:19.943485 16658 hybrid_clock.cc:648] HybridClock initialized: now 1750859899943431 us; error 48 us; skew 500 ppm
I20250625 13:58:19.944527 16658 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:58:19.951906 16658 webserver.cc:469] Webserver started at http://127.13.155.193:46795/ using document root <none> and password file <none>
I20250625 13:58:19.953075 16658 fs_manager.cc:362] Metadata directory not provided
I20250625 13:58:19.953338 16658 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:58:19.953879 16658 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:58:19.960218 16658 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "4fc2202863b14ddcb7c61b763182bad8"
format_stamp: "Formatted at 2025-06-25 13:58:19 on dist-test-slave-k028"
I20250625 13:58:19.961647 16658 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "4fc2202863b14ddcb7c61b763182bad8"
format_stamp: "Formatted at 2025-06-25 13:58:19 on dist-test-slave-k028"
I20250625 13:58:19.970594 16658 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.011s sys 0.001s
I20250625 13:58:19.977641 16674 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:58:19.978716 16658 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250625 13:58:19.979048 16658 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "4fc2202863b14ddcb7c61b763182bad8"
format_stamp: "Formatted at 2025-06-25 13:58:19 on dist-test-slave-k028"
I20250625 13:58:19.979372 16658 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:58:20.043872 16658 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:58:20.045363 16658 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:58:20.045782 16658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:58:20.048269 16658 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 13:58:20.052429 16658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 13:58:20.052615 16658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 13:58:20.052883 16658 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 13:58:20.053033 16658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 13:58:20.187381 16658 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.193:41377
I20250625 13:58:20.187458 16786 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.193:41377 every 8 connection(s)
I20250625 13:58:20.190940 16658 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250625 13:58:20.192327 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 16658
I20250625 13:58:20.192855 13935 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250625 13:58:20.202311 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.155.194:0
--local_ip_for_outbound_sockets=127.13.155.194
--webserver_interface=127.13.155.194
--webserver_port=0
--tserver_master_addrs=127.13.155.254:45625
--builtin_ntp_servers=127.13.155.212:39675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 13:58:20.214753 16787 heartbeater.cc:344] Connected to a master server at 127.13.155.254:45625
I20250625 13:58:20.215175 16787 heartbeater.cc:461] Registering TS with master...
I20250625 13:58:20.216195 16787 heartbeater.cc:507] Master 127.13.155.254:45625 requested a full tablet report, sending...
I20250625 13:58:20.219002 16599 ts_manager.cc:194] Registered new tserver with Master: 4fc2202863b14ddcb7c61b763182bad8 (127.13.155.193:41377)
I20250625 13:58:20.220914 16599 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.155.193:36709
W20250625 13:58:20.495311 16791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:58:20.495815 16791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:58:20.496296 16791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:58:20.526837 16791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:58:20.527714 16791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.155.194
I20250625 13:58:20.561515 16791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:39675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.155.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.13.155.194
--webserver_port=0
--tserver_master_addrs=127.13.155.254:45625
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.155.194
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:58:20.562911 16791 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:58:20.564497 16791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:58:20.580019 16798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:58:21.224381 16787 heartbeater.cc:499] Master 127.13.155.254:45625 was elected leader, sending a full tablet report...
W20250625 13:58:20.580389 16797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:58:20.581058 16800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:58:21.730141 16799 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250625 13:58:21.730273 16791 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 13:58:21.734541 16791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:58:21.737280 16791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:58:21.738700 16791 hybrid_clock.cc:648] HybridClock initialized: now 1750859901738664 us; error 61 us; skew 500 ppm
I20250625 13:58:21.739500 16791 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:58:21.746086 16791 webserver.cc:469] Webserver started at http://127.13.155.194:34305/ using document root <none> and password file <none>
I20250625 13:58:21.747030 16791 fs_manager.cc:362] Metadata directory not provided
I20250625 13:58:21.747238 16791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:58:21.747694 16791 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:58:21.752089 16791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "3db2009bfdf246fdbd62df05635ea607"
format_stamp: "Formatted at 2025-06-25 13:58:21 on dist-test-slave-k028"
I20250625 13:58:21.753137 16791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "3db2009bfdf246fdbd62df05635ea607"
format_stamp: "Formatted at 2025-06-25 13:58:21 on dist-test-slave-k028"
I20250625 13:58:21.759819 16791 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.000s sys 0.008s
I20250625 13:58:21.765126 16807 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:58:21.766106 16791 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250625 13:58:21.766417 16791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "3db2009bfdf246fdbd62df05635ea607"
format_stamp: "Formatted at 2025-06-25 13:58:21 on dist-test-slave-k028"
I20250625 13:58:21.766788 16791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:58:21.816958 16791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:58:21.818411 16791 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:58:21.818897 16791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:58:21.821362 16791 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 13:58:21.825309 16791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 13:58:21.825500 16791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 13:58:21.825783 16791 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 13:58:21.825928 16791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 13:58:21.957839 16791 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.194:44817
I20250625 13:58:21.957947 16919 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.194:44817 every 8 connection(s)
I20250625 13:58:21.960397 16791 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250625 13:58:21.962951 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 16791
I20250625 13:58:21.963429 13935 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250625 13:58:21.970604 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.155.195:0
--local_ip_for_outbound_sockets=127.13.155.195
--webserver_interface=127.13.155.195
--webserver_port=0
--tserver_master_addrs=127.13.155.254:45625
--builtin_ntp_servers=127.13.155.212:39675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 13:58:21.982347 16920 heartbeater.cc:344] Connected to a master server at 127.13.155.254:45625
I20250625 13:58:21.982806 16920 heartbeater.cc:461] Registering TS with master...
I20250625 13:58:21.983769 16920 heartbeater.cc:507] Master 127.13.155.254:45625 requested a full tablet report, sending...
I20250625 13:58:21.986006 16599 ts_manager.cc:194] Registered new tserver with Master: 3db2009bfdf246fdbd62df05635ea607 (127.13.155.194:44817)
I20250625 13:58:21.988013 16599 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.155.194:43295
W20250625 13:58:22.269680 16924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:58:22.270370 16924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:58:22.270961 16924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:58:22.303999 16924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:58:22.304900 16924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.155.195
I20250625 13:58:22.339511 16924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:39675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.155.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.13.155.195
--webserver_port=0
--tserver_master_addrs=127.13.155.254:45625
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.155.195
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:58:22.340873 16924 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:58:22.342509 16924 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:58:22.357326 16930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:58:22.991426 16920 heartbeater.cc:499] Master 127.13.155.254:45625 was elected leader, sending a full tablet report...
W20250625 13:58:23.759334 16929 debug-util.cc:398] Leaking SignalData structure 0x7b08000184e0 after lost signal to thread 16924
W20250625 13:58:23.761102 16924 thread.cc:640] OpenStack (cloud detector) Time spent creating pthread: real 1.407s user 0.483s sys 0.900s
W20250625 13:58:22.357316 16931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:58:23.761641 16924 thread.cc:606] OpenStack (cloud detector) Time spent starting thread: real 1.407s user 0.484s sys 0.901s
W20250625 13:58:23.763516 16933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:58:23.765738 16932 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1411 milliseconds
I20250625 13:58:23.765834 16924 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 13:58:23.766971 16924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:58:23.768949 16924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:58:23.770298 16924 hybrid_clock.cc:648] HybridClock initialized: now 1750859903770245 us; error 43 us; skew 500 ppm
I20250625 13:58:23.771118 16924 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:58:23.788003 16924 webserver.cc:469] Webserver started at http://127.13.155.195:40695/ using document root <none> and password file <none>
I20250625 13:58:23.789232 16924 fs_manager.cc:362] Metadata directory not provided
I20250625 13:58:23.789579 16924 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:58:23.790181 16924 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:58:23.795212 16924 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "0442350ae42f400fa4d5eca497bc9141"
format_stamp: "Formatted at 2025-06-25 13:58:23 on dist-test-slave-k028"
I20250625 13:58:23.796262 16924 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "0442350ae42f400fa4d5eca497bc9141"
format_stamp: "Formatted at 2025-06-25 13:58:23 on dist-test-slave-k028"
I20250625 13:58:23.803269 16924 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20250625 13:58:23.808802 16941 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:58:23.809808 16924 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.001s
I20250625 13:58:23.810101 16924 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "0442350ae42f400fa4d5eca497bc9141"
format_stamp: "Formatted at 2025-06-25 13:58:23 on dist-test-slave-k028"
I20250625 13:58:23.810400 16924 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:58:23.857054 16924 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:58:23.858511 16924 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:58:23.858924 16924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:58:23.861291 16924 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 13:58:23.865298 16924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 13:58:23.865491 16924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 13:58:23.865731 16924 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 13:58:23.865895 16924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 13:58:24.000767 16924 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.195:36029
I20250625 13:58:24.000876 17053 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.195:36029 every 8 connection(s)
I20250625 13:58:24.003540 16924 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250625 13:58:24.012508 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 16924
I20250625 13:58:24.013274 13935 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250625 13:58:24.025633 17054 heartbeater.cc:344] Connected to a master server at 127.13.155.254:45625
I20250625 13:58:24.026116 17054 heartbeater.cc:461] Registering TS with master...
I20250625 13:58:24.027204 17054 heartbeater.cc:507] Master 127.13.155.254:45625 requested a full tablet report, sending...
I20250625 13:58:24.029456 16599 ts_manager.cc:194] Registered new tserver with Master: 0442350ae42f400fa4d5eca497bc9141 (127.13.155.195:36029)
I20250625 13:58:24.030790 16599 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.155.195:40193
I20250625 13:58:24.034226 13935 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 13:58:24.069444 13935 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 13:58:24.069775 13935 test_util.cc:276] Using random seed: -694013826
I20250625 13:58:24.110025 16599 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49292:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:58:24.151154 16855 tablet_service.cc:1468] Processing CreateTablet for tablet 6f5556481b084e90931ef5a1f435126e (DEFAULT_TABLE table=TestTable [id=45592c6a5f254db4a06f07071d4ad9fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:58:24.152631 16855 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f5556481b084e90931ef5a1f435126e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:58:24.172969 17074 tablet_bootstrap.cc:492] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607: Bootstrap starting.
I20250625 13:58:24.178689 17074 tablet_bootstrap.cc:654] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607: Neither blocks nor log segments found. Creating new log.
I20250625 13:58:24.180455 17074 log.cc:826] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607: Log is configured to *not* fsync() on all Append() calls
I20250625 13:58:24.185276 17074 tablet_bootstrap.cc:492] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607: No bootstrap required, opened a new log
I20250625 13:58:24.185753 17074 ts_tablet_manager.cc:1397] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.008s
I20250625 13:58:24.205288 17074 raft_consensus.cc:357] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } }
I20250625 13:58:24.205919 17074 raft_consensus.cc:383] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:58:24.206161 17074 raft_consensus.cc:738] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3db2009bfdf246fdbd62df05635ea607, State: Initialized, Role: FOLLOWER
I20250625 13:58:24.206925 17074 consensus_queue.cc:260] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } }
I20250625 13:58:24.207484 17074 raft_consensus.cc:397] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 13:58:24.207767 17074 raft_consensus.cc:491] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 13:58:24.208086 17074 raft_consensus.cc:3058] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:58:24.212993 17074 raft_consensus.cc:513] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } }
I20250625 13:58:24.213814 17074 leader_election.cc:304] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3db2009bfdf246fdbd62df05635ea607; no voters:
I20250625 13:58:24.216145 17074 leader_election.cc:290] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 13:58:24.216574 17076 raft_consensus.cc:2802] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:58:24.221346 17074 ts_tablet_manager.cc:1428] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607: Time spent starting tablet: real 0.035s user 0.031s sys 0.004s
I20250625 13:58:24.221385 17076 raft_consensus.cc:695] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [term 1 LEADER]: Becoming Leader. State: Replica: 3db2009bfdf246fdbd62df05635ea607, State: Running, Role: LEADER
I20250625 13:58:24.222316 17076 consensus_queue.cc:237] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } }
I20250625 13:58:24.236742 16599 catalog_manager.cc:5582] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3db2009bfdf246fdbd62df05635ea607 (127.13.155.194). New cstate: current_term: 1 leader_uuid: "3db2009bfdf246fdbd62df05635ea607" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } health_report { overall_health: HEALTHY } } }
W20250625 13:58:24.328173 16783 debug-util.cc:398] Leaking SignalData structure 0x7b08000b2180 after lost signal to thread 16659
W20250625 13:58:24.329464 16783 debug-util.cc:398] Leaking SignalData structure 0x7b08000b2420 after lost signal to thread 16786
I20250625 13:58:24.437594 13935 test_util.cc:276] Using random seed: -693646018
I20250625 13:58:24.466904 16596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49296:
name: "TestTable1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:58:24.501320 16989 tablet_service.cc:1468] Processing CreateTablet for tablet 14807c1bcb064549b794601cf576aa4b (DEFAULT_TABLE table=TestTable1 [id=7a43ed5ee37b4439a7449b3cbff58fbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:58:24.503196 16989 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14807c1bcb064549b794601cf576aa4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:58:24.525218 17095 tablet_bootstrap.cc:492] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141: Bootstrap starting.
I20250625 13:58:24.531145 17095 tablet_bootstrap.cc:654] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141: Neither blocks nor log segments found. Creating new log.
I20250625 13:58:24.532887 17095 log.cc:826] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141: Log is configured to *not* fsync() on all Append() calls
I20250625 13:58:24.537429 17095 tablet_bootstrap.cc:492] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141: No bootstrap required, opened a new log
I20250625 13:58:24.537863 17095 ts_tablet_manager.cc:1397] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20250625 13:58:24.555707 17095 raft_consensus.cc:357] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } }
I20250625 13:58:24.556318 17095 raft_consensus.cc:383] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:58:24.556562 17095 raft_consensus.cc:738] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0442350ae42f400fa4d5eca497bc9141, State: Initialized, Role: FOLLOWER
I20250625 13:58:24.557335 17095 consensus_queue.cc:260] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } }
I20250625 13:58:24.557889 17095 raft_consensus.cc:397] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 13:58:24.558168 17095 raft_consensus.cc:491] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 13:58:24.558544 17095 raft_consensus.cc:3058] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:58:24.562757 17095 raft_consensus.cc:513] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } }
I20250625 13:58:24.563565 17095 leader_election.cc:304] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0442350ae42f400fa4d5eca497bc9141; no voters:
I20250625 13:58:24.566897 17095 leader_election.cc:290] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 13:58:24.567577 17097 raft_consensus.cc:2802] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:58:24.570336 17054 heartbeater.cc:499] Master 127.13.155.254:45625 was elected leader, sending a full tablet report...
I20250625 13:58:24.570631 17095 ts_tablet_manager.cc:1428] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141: Time spent starting tablet: real 0.033s user 0.026s sys 0.008s
I20250625 13:58:24.572007 17097 raft_consensus.cc:695] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [term 1 LEADER]: Becoming Leader. State: Replica: 0442350ae42f400fa4d5eca497bc9141, State: Running, Role: LEADER
I20250625 13:58:24.572944 17097 consensus_queue.cc:237] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } }
I20250625 13:58:24.585580 16596 catalog_manager.cc:5582] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0442350ae42f400fa4d5eca497bc9141 (127.13.155.195). New cstate: current_term: 1 leader_uuid: "0442350ae42f400fa4d5eca497bc9141" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } health_report { overall_health: HEALTHY } } }
I20250625 13:58:24.827008 13935 test_util.cc:276] Using random seed: -693256603
I20250625 13:58:24.852770 16593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33376:
name: "TestTable2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:58:24.890270 16722 tablet_service.cc:1468] Processing CreateTablet for tablet 20344a6b08f946e69ea5a45322f37a8a (DEFAULT_TABLE table=TestTable2 [id=ba36d7afb6184074bada1151e4115b10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:58:24.892634 16722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20344a6b08f946e69ea5a45322f37a8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:58:24.914489 17116 tablet_bootstrap.cc:492] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8: Bootstrap starting.
I20250625 13:58:24.920486 17116 tablet_bootstrap.cc:654] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8: Neither blocks nor log segments found. Creating new log.
I20250625 13:58:24.922580 17116 log.cc:826] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8: Log is configured to *not* fsync() on all Append() calls
I20250625 13:58:24.927811 17116 tablet_bootstrap.cc:492] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8: No bootstrap required, opened a new log
I20250625 13:58:24.928309 17116 ts_tablet_manager.cc:1397] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.009s
I20250625 13:58:24.946735 17116 raft_consensus.cc:357] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } }
I20250625 13:58:24.947556 17116 raft_consensus.cc:383] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:58:24.947906 17116 raft_consensus.cc:738] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fc2202863b14ddcb7c61b763182bad8, State: Initialized, Role: FOLLOWER
I20250625 13:58:24.948731 17116 consensus_queue.cc:260] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } }
I20250625 13:58:24.949321 17116 raft_consensus.cc:397] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 13:58:24.949589 17116 raft_consensus.cc:491] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 13:58:24.949888 17116 raft_consensus.cc:3058] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:58:24.954123 17116 raft_consensus.cc:513] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } }
I20250625 13:58:24.954891 17116 leader_election.cc:304] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fc2202863b14ddcb7c61b763182bad8; no voters:
I20250625 13:58:24.957230 17116 leader_election.cc:290] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 13:58:24.957703 17118 raft_consensus.cc:2802] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:58:24.961375 17116 ts_tablet_manager.cc:1428] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8: Time spent starting tablet: real 0.033s user 0.025s sys 0.007s
I20250625 13:58:24.971096 17118 raft_consensus.cc:695] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8 [term 1 LEADER]: Becoming Leader. State: Replica: 4fc2202863b14ddcb7c61b763182bad8, State: Running, Role: LEADER
I20250625 13:58:24.972110 17118 consensus_queue.cc:237] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } }
I20250625 13:58:24.983727 16593 catalog_manager.cc:5582] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fc2202863b14ddcb7c61b763182bad8 (127.13.155.193). New cstate: current_term: 1 leader_uuid: "4fc2202863b14ddcb7c61b763182bad8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } health_report { overall_health: HEALTHY } } }
I20250625 13:58:25.304563 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 16566
W20250625 13:58:25.603358 17054 heartbeater.cc:646] Failed to heartbeat to 127.13.155.254:45625 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.155.254:45625: connect: Connection refused (error 111)
W20250625 13:58:26.026247 16787 heartbeater.cc:646] Failed to heartbeat to 127.13.155.254:45625 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.155.254:45625: connect: Connection refused (error 111)
W20250625 13:58:26.274971 16920 heartbeater.cc:646] Failed to heartbeat to 127.13.155.254:45625 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.155.254:45625: connect: Connection refused (error 111)
I20250625 13:58:30.347431 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 16658
I20250625 13:58:30.371553 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 16791
I20250625 13:58:30.399721 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 16924
I20250625 13:58:30.426398 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.155.254:45625
--webserver_interface=127.13.155.254
--webserver_port=38733
--builtin_ntp_servers=127.13.155.212:39675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.155.254:45625 with env {}
W20250625 13:58:30.721462 17195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:58:30.722023 17195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:58:30.722424 17195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:58:30.753578 17195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 13:58:30.753868 17195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:58:30.754070 17195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 13:58:30.754262 17195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 13:58:30.789285 17195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:39675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.155.254:45625
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.155.254:45625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.13.155.254
--webserver_port=38733
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:58:30.790671 17195 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:58:30.792271 17195 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:58:30.808292 17201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:58:30.808316 17202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:58:30.810593 17204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:58:30.810679 17195 server_base.cc:1048] running on GCE node
I20250625 13:58:31.997210 17195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:58:31.999857 17195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:58:32.001214 17195 hybrid_clock.cc:648] HybridClock initialized: now 1750859912001174 us; error 59 us; skew 500 ppm
I20250625 13:58:32.002014 17195 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:58:32.012626 17195 webserver.cc:469] Webserver started at http://127.13.155.254:38733/ using document root <none> and password file <none>
I20250625 13:58:32.013593 17195 fs_manager.cc:362] Metadata directory not provided
I20250625 13:58:32.013835 17195 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:58:32.021677 17195 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.001s
I20250625 13:58:32.027820 17211 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:58:32.028831 17195 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250625 13:58:32.029150 17195 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
uuid: "322e3b3a9cca41578d3358d15bbd13cb"
format_stamp: "Formatted at 2025-06-25 13:58:18 on dist-test-slave-k028"
I20250625 13:58:32.031103 17195 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:58:32.090009 17195 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:58:32.091509 17195 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:58:32.091958 17195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:58:32.161356 17195 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.254:45625
I20250625 13:58:32.161451 17262 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.254:45625 every 8 connection(s)
I20250625 13:58:32.164145 17195 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250625 13:58:32.168572 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 17195
I20250625 13:58:32.169938 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.155.193:41377
--local_ip_for_outbound_sockets=127.13.155.193
--tserver_master_addrs=127.13.155.254:45625
--webserver_port=46795
--webserver_interface=127.13.155.193
--builtin_ntp_servers=127.13.155.212:39675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 13:58:32.174116 17263 sys_catalog.cc:263] Verifying existing consensus state
I20250625 13:58:32.179397 17263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb: Bootstrap starting.
I20250625 13:58:32.195472 17263 log.cc:826] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb: Log is configured to *not* fsync() on all Append() calls
I20250625 13:58:32.261806 17263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=13 ignored=0} mutations{seen=10 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 13:58:32.262966 17263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb: Bootstrap complete.
I20250625 13:58:32.296964 17263 raft_consensus.cc:357] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322e3b3a9cca41578d3358d15bbd13cb" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 45625 } }
I20250625 13:58:32.300802 17263 raft_consensus.cc:738] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 322e3b3a9cca41578d3358d15bbd13cb, State: Initialized, Role: FOLLOWER
I20250625 13:58:32.302009 17263 consensus_queue.cc:260] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322e3b3a9cca41578d3358d15bbd13cb" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 45625 } }
I20250625 13:58:32.302923 17263 raft_consensus.cc:397] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 13:58:32.303289 17263 raft_consensus.cc:491] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 13:58:32.303726 17263 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [term 2 FOLLOWER]: Advancing to term 3
I20250625 13:58:32.314970 17263 raft_consensus.cc:513] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322e3b3a9cca41578d3358d15bbd13cb" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 45625 } }
I20250625 13:58:32.315984 17263 leader_election.cc:304] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 322e3b3a9cca41578d3358d15bbd13cb; no voters:
I20250625 13:58:32.318146 17263 leader_election.cc:290] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [CANDIDATE]: Term 3 election: Requested vote from peers
I20250625 13:58:32.318696 17267 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [term 3 FOLLOWER]: Leader election won for term 3
I20250625 13:58:32.322708 17267 raft_consensus.cc:695] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [term 3 LEADER]: Becoming Leader. State: Replica: 322e3b3a9cca41578d3358d15bbd13cb, State: Running, Role: LEADER
I20250625 13:58:32.323686 17263 sys_catalog.cc:564] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [sys.catalog]: configured and running, proceeding with master startup.
I20250625 13:58:32.323838 17267 consensus_queue.cc:237] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322e3b3a9cca41578d3358d15bbd13cb" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 45625 } }
I20250625 13:58:32.338030 17268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "322e3b3a9cca41578d3358d15bbd13cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322e3b3a9cca41578d3358d15bbd13cb" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 45625 } } }
I20250625 13:58:32.338871 17268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [sys.catalog]: This master's current role is: LEADER
I20250625 13:58:32.340812 17269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 322e3b3a9cca41578d3358d15bbd13cb. Latest consensus state: current_term: 3 leader_uuid: "322e3b3a9cca41578d3358d15bbd13cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322e3b3a9cca41578d3358d15bbd13cb" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 45625 } } }
I20250625 13:58:32.341496 17269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [sys.catalog]: This master's current role is: LEADER
I20250625 13:58:32.355439 17275 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 13:58:32.370486 17275 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=39b413cde03c404b8f95a92511c6accf]
I20250625 13:58:32.372493 17275 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=ba36d7afb6184074bada1151e4115b10]
I20250625 13:58:32.374394 17275 catalog_manager.cc:671] Loaded metadata for table TestTable [id=e66ce998eddf427e91c07938e0e93bf3]
I20250625 13:58:32.382952 17275 tablet_loader.cc:96] loaded metadata for tablet 14807c1bcb064549b794601cf576aa4b (table TestTable1 [id=39b413cde03c404b8f95a92511c6accf])
I20250625 13:58:32.384521 17275 tablet_loader.cc:96] loaded metadata for tablet 20344a6b08f946e69ea5a45322f37a8a (table TestTable2 [id=ba36d7afb6184074bada1151e4115b10])
I20250625 13:58:32.386022 17275 tablet_loader.cc:96] loaded metadata for tablet 6f5556481b084e90931ef5a1f435126e (table TestTable [id=e66ce998eddf427e91c07938e0e93bf3])
I20250625 13:58:32.387702 17275 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 13:58:32.393903 17275 catalog_manager.cc:1261] Loaded cluster ID: cba9adb64c954b8993397fb3176e4ce6
I20250625 13:58:32.394372 17275 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 13:58:32.404129 17275 catalog_manager.cc:1506] Loading token signing keys...
I20250625 13:58:32.410404 17275 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb: Loaded TSK: 0
I20250625 13:58:32.412322 17275 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250625 13:58:32.567435 17265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:58:32.567981 17265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:58:32.568499 17265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:58:32.601454 17265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:58:32.602388 17265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.155.193
I20250625 13:58:32.639174 17265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:39675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.155.193:41377
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.13.155.193
--webserver_port=46795
--tserver_master_addrs=127.13.155.254:45625
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.155.193
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:58:32.640604 17265 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:58:32.642406 17265 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:58:32.661423 17292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:58:32.663748 17294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:58:32.662139 17291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:58:33.921634 17293 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1258 milliseconds
I20250625 13:58:33.921736 17265 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 13:58:33.922998 17265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:58:33.925776 17265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:58:33.927232 17265 hybrid_clock.cc:648] HybridClock initialized: now 1750859913927177 us; error 75 us; skew 500 ppm
I20250625 13:58:33.928030 17265 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:58:33.935241 17265 webserver.cc:469] Webserver started at http://127.13.155.193:46795/ using document root <none> and password file <none>
I20250625 13:58:33.936159 17265 fs_manager.cc:362] Metadata directory not provided
I20250625 13:58:33.936398 17265 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:58:33.944355 17265 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20250625 13:58:33.949556 17301 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:58:33.950731 17265 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20250625 13:58:33.951053 17265 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "4fc2202863b14ddcb7c61b763182bad8"
format_stamp: "Formatted at 2025-06-25 13:58:19 on dist-test-slave-k028"
I20250625 13:58:33.952944 17265 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:58:34.004281 17265 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:58:34.005872 17265 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:58:34.006335 17265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:58:34.009404 17265 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 13:58:34.015861 17308 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250625 13:58:34.024039 17265 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250625 13:58:34.024269 17265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.002s
I20250625 13:58:34.024571 17265 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250625 13:58:34.029354 17265 ts_tablet_manager.cc:610] Registered 1 tablets
I20250625 13:58:34.029727 17265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.004s sys 0.001s
I20250625 13:58:34.030030 17308 tablet_bootstrap.cc:492] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8: Bootstrap starting.
I20250625 13:58:34.112069 17308 log.cc:826] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8: Log is configured to *not* fsync() on all Append() calls
I20250625 13:58:34.237803 17265 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.193:41377
I20250625 13:58:34.238016 17415 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.193:41377 every 8 connection(s)
I20250625 13:58:34.241680 17265 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250625 13:58:34.244603 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 17265
I20250625 13:58:34.246713 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.155.194:44817
--local_ip_for_outbound_sockets=127.13.155.194
--tserver_master_addrs=127.13.155.254:45625
--webserver_port=34305
--webserver_interface=127.13.155.194
--builtin_ntp_servers=127.13.155.212:39675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 13:58:34.247427 17308 tablet_bootstrap.cc:492] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 13:58:34.248718 17308 tablet_bootstrap.cc:492] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8: Bootstrap complete.
I20250625 13:58:34.251160 17308 ts_tablet_manager.cc:1397] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8: Time spent bootstrapping tablet: real 0.222s user 0.165s sys 0.051s
I20250625 13:58:34.281342 17308 raft_consensus.cc:357] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } }
I20250625 13:58:34.284817 17308 raft_consensus.cc:738] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fc2202863b14ddcb7c61b763182bad8, State: Initialized, Role: FOLLOWER
I20250625 13:58:34.286091 17308 consensus_queue.cc:260] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } }
I20250625 13:58:34.286985 17308 raft_consensus.cc:397] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 13:58:34.287405 17308 raft_consensus.cc:491] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 13:58:34.287885 17308 raft_consensus.cc:3058] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8 [term 1 FOLLOWER]: Advancing to term 2
I20250625 13:58:34.298422 17308 raft_consensus.cc:513] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } }
I20250625 13:58:34.299537 17308 leader_election.cc:304] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fc2202863b14ddcb7c61b763182bad8; no voters:
I20250625 13:58:34.300701 17416 heartbeater.cc:344] Connected to a master server at 127.13.155.254:45625
I20250625 13:58:34.301199 17416 heartbeater.cc:461] Registering TS with master...
I20250625 13:58:34.302390 17416 heartbeater.cc:507] Master 127.13.155.254:45625 requested a full tablet report, sending...
I20250625 13:58:34.307337 17228 ts_manager.cc:194] Registered new tserver with Master: 4fc2202863b14ddcb7c61b763182bad8 (127.13.155.193:41377)
I20250625 13:58:34.308164 17421 raft_consensus.cc:2802] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8 [term 2 FOLLOWER]: Leader election won for term 2
I20250625 13:58:34.309850 17308 leader_election.cc:290] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250625 13:58:34.314687 17308 ts_tablet_manager.cc:1428] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8: Time spent starting tablet: real 0.063s user 0.033s sys 0.019s
I20250625 13:58:34.315341 17228 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.155.193:44577
I20250625 13:58:34.316004 17421 raft_consensus.cc:695] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8 [term 2 LEADER]: Becoming Leader. State: Replica: 4fc2202863b14ddcb7c61b763182bad8, State: Running, Role: LEADER
I20250625 13:58:34.316908 17421 consensus_queue.cc:237] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } }
I20250625 13:58:34.322270 17416 heartbeater.cc:499] Master 127.13.155.254:45625 was elected leader, sending a full tablet report...
I20250625 13:58:34.332883 17228 catalog_manager.cc:5582] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "4fc2202863b14ddcb7c61b763182bad8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } health_report { overall_health: HEALTHY } } }
W20250625 13:58:34.631268 17420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:58:34.631824 17420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:58:34.632315 17420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:58:34.663800 17420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:58:34.664789 17420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.155.194
I20250625 13:58:34.700439 17420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:39675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.155.194:44817
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.13.155.194
--webserver_port=34305
--tserver_master_addrs=127.13.155.254:45625
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.155.194
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:58:34.701772 17420 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:58:34.703456 17420 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:58:34.720913 17436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:58:34.721150 17435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:58:34.721150 17438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:58:35.894152 17437 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1174 milliseconds
I20250625 13:58:35.894254 17420 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 13:58:35.899643 17420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:58:35.902062 17420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:58:35.903466 17420 hybrid_clock.cc:648] HybridClock initialized: now 1750859915903414 us; error 70 us; skew 500 ppm
I20250625 13:58:35.904379 17420 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:58:35.910889 17420 webserver.cc:469] Webserver started at http://127.13.155.194:34305/ using document root <none> and password file <none>
I20250625 13:58:35.911814 17420 fs_manager.cc:362] Metadata directory not provided
I20250625 13:58:35.912053 17420 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:58:35.919870 17420 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20250625 13:58:35.924532 17445 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:58:35.925618 17420 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250625 13:58:35.925933 17420 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "3db2009bfdf246fdbd62df05635ea607"
format_stamp: "Formatted at 2025-06-25 13:58:21 on dist-test-slave-k028"
I20250625 13:58:35.927834 17420 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:58:35.982543 17420 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:58:35.984082 17420 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:58:35.984809 17420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:58:35.987574 17420 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 13:58:35.993225 17452 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250625 13:58:36.000826 17420 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250625 13:58:36.001081 17420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.001s sys 0.001s
I20250625 13:58:36.001348 17420 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250625 13:58:36.006052 17420 ts_tablet_manager.cc:610] Registered 1 tablets
I20250625 13:58:36.006263 17420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.000s sys 0.002s
I20250625 13:58:36.006700 17452 tablet_bootstrap.cc:492] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607: Bootstrap starting.
I20250625 13:58:36.066272 17452 log.cc:826] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607: Log is configured to *not* fsync() on all Append() calls
I20250625 13:58:36.166163 17452 tablet_bootstrap.cc:492] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 13:58:36.167244 17452 tablet_bootstrap.cc:492] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607: Bootstrap complete.
I20250625 13:58:36.169041 17452 ts_tablet_manager.cc:1397] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607: Time spent bootstrapping tablet: real 0.163s user 0.116s sys 0.043s
I20250625 13:58:36.188124 17420 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.194:44817
I20250625 13:58:36.188300 17559 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.194:44817 every 8 connection(s)
I20250625 13:58:36.190893 17420 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250625 13:58:36.190140 17452 raft_consensus.cc:357] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } }
I20250625 13:58:36.192307 17452 raft_consensus.cc:738] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3db2009bfdf246fdbd62df05635ea607, State: Initialized, Role: FOLLOWER
I20250625 13:58:36.193150 17452 consensus_queue.cc:260] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } }
I20250625 13:58:36.193642 17452 raft_consensus.cc:397] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 13:58:36.193928 17452 raft_consensus.cc:491] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 13:58:36.194506 17452 raft_consensus.cc:3058] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [term 1 FOLLOWER]: Advancing to term 2
I20250625 13:58:36.195636 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 17420
I20250625 13:58:36.196971 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.155.195:36029
--local_ip_for_outbound_sockets=127.13.155.195
--tserver_master_addrs=127.13.155.254:45625
--webserver_port=40695
--webserver_interface=127.13.155.195
--builtin_ntp_servers=127.13.155.212:39675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 13:58:36.201165 17452 raft_consensus.cc:513] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } }
I20250625 13:58:36.202418 17452 leader_election.cc:304] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3db2009bfdf246fdbd62df05635ea607; no voters:
I20250625 13:58:36.219062 17452 leader_election.cc:290] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250625 13:58:36.219796 17564 raft_consensus.cc:2802] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [term 2 FOLLOWER]: Leader election won for term 2
I20250625 13:58:36.228924 17452 ts_tablet_manager.cc:1428] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607: Time spent starting tablet: real 0.060s user 0.026s sys 0.031s
I20250625 13:58:36.230283 17564 raft_consensus.cc:695] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [term 2 LEADER]: Becoming Leader. State: Replica: 3db2009bfdf246fdbd62df05635ea607, State: Running, Role: LEADER
I20250625 13:58:36.231362 17564 consensus_queue.cc:237] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } }
I20250625 13:58:36.235539 17560 heartbeater.cc:344] Connected to a master server at 127.13.155.254:45625
I20250625 13:58:36.236085 17560 heartbeater.cc:461] Registering TS with master...
I20250625 13:58:36.237255 17560 heartbeater.cc:507] Master 127.13.155.254:45625 requested a full tablet report, sending...
I20250625 13:58:36.241325 17228 ts_manager.cc:194] Registered new tserver with Master: 3db2009bfdf246fdbd62df05635ea607 (127.13.155.194:44817)
I20250625 13:58:36.242753 17228 catalog_manager.cc:5582] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 reported cstate change: term changed from 0 to 2, leader changed from <none> to 3db2009bfdf246fdbd62df05635ea607 (127.13.155.194), VOTER 3db2009bfdf246fdbd62df05635ea607 (127.13.155.194) added. New cstate: current_term: 2 leader_uuid: "3db2009bfdf246fdbd62df05635ea607" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } health_report { overall_health: HEALTHY } } }
I20250625 13:58:36.262405 17228 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.155.194:43359
I20250625 13:58:36.270968 17560 heartbeater.cc:499] Master 127.13.155.254:45625 was elected leader, sending a full tablet report...
I20250625 13:58:36.282397 17515 consensus_queue.cc:237] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: NON_VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: true } }
I20250625 13:58:36.285671 17567 raft_consensus.cc:2953] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index -1 to 9, NON_VOTER 4fc2202863b14ddcb7c61b763182bad8 (127.13.155.193) added. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: NON_VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: true } } }
I20250625 13:58:36.295359 17214 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6f5556481b084e90931ef5a1f435126e with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250625 13:58:36.298698 17228 catalog_manager.cc:5582] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 reported cstate change: config changed from index -1 to 9, NON_VOTER 4fc2202863b14ddcb7c61b763182bad8 (127.13.155.193) added. New cstate: current_term: 2 leader_uuid: "3db2009bfdf246fdbd62df05635ea607" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: NON_VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250625 13:58:36.300408 17447 consensus_peers.cc:487] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 -> Peer 4fc2202863b14ddcb7c61b763182bad8 (127.13.155.193:41377): Couldn't send request to peer 4fc2202863b14ddcb7c61b763182bad8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f5556481b084e90931ef5a1f435126e. This is attempt 1: this message will repeat every 5th retry.
W20250625 13:58:36.306807 17228 catalog_manager.cc:5260] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6f5556481b084e90931ef5a1f435126e with cas_config_opid_index 9: no extra replica candidate found for tablet 6f5556481b084e90931ef5a1f435126e (table TestTable [id=e66ce998eddf427e91c07938e0e93bf3]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (2) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
W20250625 13:58:36.531826 17565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:58:36.532338 17565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:58:36.532871 17565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:58:36.564260 17565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:58:36.565160 17565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.155.195
I20250625 13:58:36.599820 17565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:39675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.155.195:36029
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.13.155.195
--webserver_port=40695
--tserver_master_addrs=127.13.155.254:45625
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.155.195
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:58:36.601097 17565 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:58:36.602723 17565 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:58:36.619578 17585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:58:36.691813 17588 ts_tablet_manager.cc:927] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8: Initiating tablet copy from peer 3db2009bfdf246fdbd62df05635ea607 (127.13.155.194:44817)
I20250625 13:58:36.694262 17588 tablet_copy_client.cc:323] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8: tablet copy: Beginning tablet copy session from remote peer at address 127.13.155.194:44817
I20250625 13:58:36.708546 17535 tablet_copy_service.cc:140] P 3db2009bfdf246fdbd62df05635ea607: Received BeginTabletCopySession request for tablet 6f5556481b084e90931ef5a1f435126e from peer 4fc2202863b14ddcb7c61b763182bad8 ({username='slave'} at 127.13.155.193:52777)
I20250625 13:58:36.709038 17535 tablet_copy_service.cc:161] P 3db2009bfdf246fdbd62df05635ea607: Beginning new tablet copy session on tablet 6f5556481b084e90931ef5a1f435126e from peer 4fc2202863b14ddcb7c61b763182bad8 at {username='slave'} at 127.13.155.193:52777: session id = 4fc2202863b14ddcb7c61b763182bad8-6f5556481b084e90931ef5a1f435126e
I20250625 13:58:36.713732 17535 tablet_copy_source_session.cc:215] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607: Tablet Copy: opened 0 blocks and 1 log segments
I20250625 13:58:36.717095 17588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f5556481b084e90931ef5a1f435126e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:58:36.727914 17588 tablet_copy_client.cc:806] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8: tablet copy: Starting download of 0 data blocks...
I20250625 13:58:36.728469 17588 tablet_copy_client.cc:670] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8: tablet copy: Starting download of 1 WAL segments...
I20250625 13:58:36.732560 17588 tablet_copy_client.cc:538] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250625 13:58:36.738013 17588 tablet_bootstrap.cc:492] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8: Bootstrap starting.
I20250625 13:58:36.819782 17588 tablet_bootstrap.cc:492] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 13:58:36.820423 17588 tablet_bootstrap.cc:492] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8: Bootstrap complete.
I20250625 13:58:36.820889 17588 ts_tablet_manager.cc:1397] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8: Time spent bootstrapping tablet: real 0.083s user 0.079s sys 0.004s
I20250625 13:58:36.822506 17588 raft_consensus.cc:357] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: NON_VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: true } }
I20250625 13:58:36.822948 17588 raft_consensus.cc:738] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 4fc2202863b14ddcb7c61b763182bad8, State: Initialized, Role: LEARNER
I20250625 13:58:36.823354 17588 consensus_queue.cc:260] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: NON_VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: true } }
I20250625 13:58:36.826154 17588 ts_tablet_manager.cc:1428] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8: Time spent starting tablet: real 0.005s user 0.003s sys 0.004s
I20250625 13:58:36.827615 17535 tablet_copy_service.cc:342] P 3db2009bfdf246fdbd62df05635ea607: Request end of tablet copy session 4fc2202863b14ddcb7c61b763182bad8-6f5556481b084e90931ef5a1f435126e received from {username='slave'} at 127.13.155.193:52777
I20250625 13:58:36.828013 17535 tablet_copy_service.cc:434] P 3db2009bfdf246fdbd62df05635ea607: ending tablet copy session 4fc2202863b14ddcb7c61b763182bad8-6f5556481b084e90931ef5a1f435126e on tablet 6f5556481b084e90931ef5a1f435126e with peer 4fc2202863b14ddcb7c61b763182bad8
I20250625 13:58:37.270875 17371 raft_consensus.cc:1215] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20250625 13:58:37.694334 17595 raft_consensus.cc:1062] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607: attempting to promote NON_VOTER 4fc2202863b14ddcb7c61b763182bad8 to VOTER
I20250625 13:58:37.696197 17595 consensus_queue.cc:237] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } }
I20250625 13:58:37.701676 17371 raft_consensus.cc:1273] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8 [term 2 LEARNER]: Refusing update from remote peer 3db2009bfdf246fdbd62df05635ea607: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250625 13:58:37.702978 17595 consensus_queue.cc:1035] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250625 13:58:37.709928 17594 raft_consensus.cc:2953] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 4fc2202863b14ddcb7c61b763182bad8 (127.13.155.193) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } } }
I20250625 13:58:37.711589 17371 raft_consensus.cc:2953] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 4fc2202863b14ddcb7c61b763182bad8 (127.13.155.193) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } } }
I20250625 13:58:37.721863 17227 catalog_manager.cc:5582] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 reported cstate change: config changed from index 9 to 10, 4fc2202863b14ddcb7c61b763182bad8 (127.13.155.193) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "3db2009bfdf246fdbd62df05635ea607" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250625 13:58:36.619621 17583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:58:36.619912 17582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:58:37.831144 17584 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250625 13:58:37.831257 17565 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 13:58:37.835834 17565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:58:37.837980 17565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:58:37.839304 17565 hybrid_clock.cc:648] HybridClock initialized: now 1750859917839266 us; error 57 us; skew 500 ppm
I20250625 13:58:37.840375 17565 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:58:37.850430 17565 webserver.cc:469] Webserver started at http://127.13.155.195:40695/ using document root <none> and password file <none>
I20250625 13:58:37.851408 17565 fs_manager.cc:362] Metadata directory not provided
I20250625 13:58:37.851653 17565 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:58:37.859377 17565 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20250625 13:58:37.863854 17606 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:58:37.865058 17565 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250625 13:58:37.865392 17565 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "0442350ae42f400fa4d5eca497bc9141"
format_stamp: "Formatted at 2025-06-25 13:58:23 on dist-test-slave-k028"
I20250625 13:58:37.867340 17565 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:58:37.932586 17565 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:58:37.934149 17565 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:58:37.934616 17565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:58:37.937332 17565 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 13:58:37.943002 17613 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250625 13:58:37.950680 17565 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250625 13:58:37.950973 17565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.002s
I20250625 13:58:37.951236 17565 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250625 13:58:37.955885 17565 ts_tablet_manager.cc:610] Registered 1 tablets
I20250625 13:58:37.956084 17565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.002s sys 0.001s
I20250625 13:58:37.956457 17613 tablet_bootstrap.cc:492] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141: Bootstrap starting.
I20250625 13:58:38.011473 17613 log.cc:826] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141: Log is configured to *not* fsync() on all Append() calls
I20250625 13:58:38.094889 17613 tablet_bootstrap.cc:492] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 13:58:38.095990 17613 tablet_bootstrap.cc:492] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141: Bootstrap complete.
I20250625 13:58:38.097304 17613 ts_tablet_manager.cc:1397] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141: Time spent bootstrapping tablet: real 0.141s user 0.110s sys 0.028s
I20250625 13:58:38.113673 17613 raft_consensus.cc:357] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } }
I20250625 13:58:38.115998 17613 raft_consensus.cc:738] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0442350ae42f400fa4d5eca497bc9141, State: Initialized, Role: FOLLOWER
I20250625 13:58:38.116861 17613 consensus_queue.cc:260] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } }
I20250625 13:58:38.117347 17613 raft_consensus.cc:397] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 13:58:38.117615 17613 raft_consensus.cc:491] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 13:58:38.117897 17613 raft_consensus.cc:3058] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [term 1 FOLLOWER]: Advancing to term 2
I20250625 13:58:38.124701 17613 raft_consensus.cc:513] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } }
I20250625 13:58:38.125463 17613 leader_election.cc:304] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0442350ae42f400fa4d5eca497bc9141; no voters:
I20250625 13:58:38.127853 17613 leader_election.cc:290] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250625 13:58:38.128254 17715 raft_consensus.cc:2802] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [term 2 FOLLOWER]: Leader election won for term 2
I20250625 13:58:38.133697 17613 ts_tablet_manager.cc:1428] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141: Time spent starting tablet: real 0.036s user 0.026s sys 0.009s
I20250625 13:58:38.134141 17715 raft_consensus.cc:695] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [term 2 LEADER]: Becoming Leader. State: Replica: 0442350ae42f400fa4d5eca497bc9141, State: Running, Role: LEADER
I20250625 13:58:38.135188 17715 consensus_queue.cc:237] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } }
I20250625 13:58:38.138993 17565 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.195:36029
I20250625 13:58:38.139328 17723 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.195:36029 every 8 connection(s)
I20250625 13:58:38.142139 17565 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250625 13:58:38.143312 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 17565
I20250625 13:58:38.178097 17726 heartbeater.cc:344] Connected to a master server at 127.13.155.254:45625
I20250625 13:58:38.178540 17726 heartbeater.cc:461] Registering TS with master...
I20250625 13:58:38.179495 17726 heartbeater.cc:507] Master 127.13.155.254:45625 requested a full tablet report, sending...
I20250625 13:58:38.182710 17227 ts_manager.cc:194] Registered new tserver with Master: 0442350ae42f400fa4d5eca497bc9141 (127.13.155.195:36029)
I20250625 13:58:38.183866 17227 catalog_manager.cc:5582] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 reported cstate change: term changed from 0 to 2, leader changed from <none> to 0442350ae42f400fa4d5eca497bc9141 (127.13.155.195), VOTER 0442350ae42f400fa4d5eca497bc9141 (127.13.155.195) added. New cstate: current_term: 2 leader_uuid: "0442350ae42f400fa4d5eca497bc9141" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } health_report { overall_health: HEALTHY } } }
I20250625 13:58:38.191334 13935 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 13:58:38.196704 13935 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
W20250625 13:58:38.201565 13935 ts_itest-base.cc:209] found only 2 out of 3 replicas of tablet 6f5556481b084e90931ef5a1f435126e: tablet_id: "6f5556481b084e90931ef5a1f435126e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER } interned_replicas { ts_info_idx: 1 role: FOLLOWER }
I20250625 13:58:38.204265 17227 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.155.195:51257
I20250625 13:58:38.211040 17726 heartbeater.cc:499] Master 127.13.155.254:45625 was elected leader, sending a full tablet report...
I20250625 13:58:38.221395 17676 consensus_queue.cc:237] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } } peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: NON_VOTER last_known_addr { host: "127.13.155.194" port: 44817 } attrs { promote: true } }
I20250625 13:58:38.224751 17718 raft_consensus.cc:2953] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index -1 to 8, NON_VOTER 3db2009bfdf246fdbd62df05635ea607 (127.13.155.194) added. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } } peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: NON_VOTER last_known_addr { host: "127.13.155.194" port: 44817 } attrs { promote: true } } }
I20250625 13:58:38.232805 17214 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 14807c1bcb064549b794601cf576aa4b with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250625 13:58:38.235591 17609 consensus_peers.cc:487] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 -> Peer 3db2009bfdf246fdbd62df05635ea607 (127.13.155.194:44817): Couldn't send request to peer 3db2009bfdf246fdbd62df05635ea607. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 14807c1bcb064549b794601cf576aa4b. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:58:38.235705 17227 catalog_manager.cc:5582] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 reported cstate change: config changed from index -1 to 8, NON_VOTER 3db2009bfdf246fdbd62df05635ea607 (127.13.155.194) added. New cstate: current_term: 2 leader_uuid: "0442350ae42f400fa4d5eca497bc9141" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: NON_VOTER last_known_addr { host: "127.13.155.194" port: 44817 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250625 13:58:38.245185 17676 consensus_queue.cc:237] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } } peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: NON_VOTER last_known_addr { host: "127.13.155.194" port: 44817 } attrs { promote: true } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: NON_VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: true } }
I20250625 13:58:38.248656 17715 raft_consensus.cc:2953] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 4fc2202863b14ddcb7c61b763182bad8 (127.13.155.193) added. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } } peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: NON_VOTER last_known_addr { host: "127.13.155.194" port: 44817 } attrs { promote: true } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: NON_VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: true } } }
W20250625 13:58:38.250131 17609 consensus_peers.cc:487] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 -> Peer 3db2009bfdf246fdbd62df05635ea607 (127.13.155.194:44817): Couldn't send request to peer 3db2009bfdf246fdbd62df05635ea607. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 14807c1bcb064549b794601cf576aa4b. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:58:38.258226 17214 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 14807c1bcb064549b794601cf576aa4b with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250625 13:58:38.259656 17608 consensus_peers.cc:487] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 -> Peer 4fc2202863b14ddcb7c61b763182bad8 (127.13.155.193:41377): Couldn't send request to peer 4fc2202863b14ddcb7c61b763182bad8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 14807c1bcb064549b794601cf576aa4b. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:58:38.261551 17227 catalog_manager.cc:5582] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 reported cstate change: config changed from index 8 to 9, NON_VOTER 4fc2202863b14ddcb7c61b763182bad8 (127.13.155.193) added. New cstate: current_term: 2 leader_uuid: "0442350ae42f400fa4d5eca497bc9141" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: NON_VOTER last_known_addr { host: "127.13.155.194" port: 44817 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: NON_VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250625 13:58:38.349607 17515 consensus_queue.cc:237] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } } peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: NON_VOTER last_known_addr { host: "127.13.155.195" port: 36029 } attrs { promote: true } }
I20250625 13:58:38.354735 17371 raft_consensus.cc:1273] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8 [term 2 FOLLOWER]: Refusing update from remote peer 3db2009bfdf246fdbd62df05635ea607: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250625 13:58:38.356043 17736 consensus_queue.cc:1035] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250625 13:58:38.361516 17737 raft_consensus.cc:2953] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 0442350ae42f400fa4d5eca497bc9141 (127.13.155.195) added. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } } peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: NON_VOTER last_known_addr { host: "127.13.155.195" port: 36029 } attrs { promote: true } } }
I20250625 13:58:38.363312 17371 raft_consensus.cc:2953] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 0442350ae42f400fa4d5eca497bc9141 (127.13.155.195) added. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } } peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: NON_VOTER last_known_addr { host: "127.13.155.195" port: 36029 } attrs { promote: true } } }
W20250625 13:58:38.365010 17448 consensus_peers.cc:487] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 -> Peer 0442350ae42f400fa4d5eca497bc9141 (127.13.155.195:36029): Couldn't send request to peer 0442350ae42f400fa4d5eca497bc9141. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f5556481b084e90931ef5a1f435126e. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:58:38.370240 17214 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6f5556481b084e90931ef5a1f435126e with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 6)
I20250625 13:58:38.376407 17227 catalog_manager.cc:5582] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8 reported cstate change: config changed from index 10 to 11, NON_VOTER 0442350ae42f400fa4d5eca497bc9141 (127.13.155.195) added. New cstate: current_term: 2 leader_uuid: "3db2009bfdf246fdbd62df05635ea607" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } } peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: NON_VOTER last_known_addr { host: "127.13.155.195" port: 36029 } attrs { promote: true } } }
I20250625 13:58:38.606559 17213 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6f5556481b084e90931ef5a1f435126e with cas_config_opid_index 9: aborting the task: latest config opid_index 11; task opid_index 9
I20250625 13:58:38.661626 17741 ts_tablet_manager.cc:927] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607: Initiating tablet copy from peer 0442350ae42f400fa4d5eca497bc9141 (127.13.155.195:36029)
I20250625 13:58:38.663738 17741 tablet_copy_client.cc:323] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607: tablet copy: Beginning tablet copy session from remote peer at address 127.13.155.195:36029
I20250625 13:58:38.665380 17696 tablet_copy_service.cc:140] P 0442350ae42f400fa4d5eca497bc9141: Received BeginTabletCopySession request for tablet 14807c1bcb064549b794601cf576aa4b from peer 3db2009bfdf246fdbd62df05635ea607 ({username='slave'} at 127.13.155.194:51313)
I20250625 13:58:38.665784 17696 tablet_copy_service.cc:161] P 0442350ae42f400fa4d5eca497bc9141: Beginning new tablet copy session on tablet 14807c1bcb064549b794601cf576aa4b from peer 3db2009bfdf246fdbd62df05635ea607 at {username='slave'} at 127.13.155.194:51313: session id = 3db2009bfdf246fdbd62df05635ea607-14807c1bcb064549b794601cf576aa4b
I20250625 13:58:38.670171 17696 tablet_copy_source_session.cc:215] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141: Tablet Copy: opened 0 blocks and 1 log segments
I20250625 13:58:38.673488 17741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14807c1bcb064549b794601cf576aa4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:58:38.683961 17741 tablet_copy_client.cc:806] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607: tablet copy: Starting download of 0 data blocks...
I20250625 13:58:38.684473 17741 tablet_copy_client.cc:670] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607: tablet copy: Starting download of 1 WAL segments...
I20250625 13:58:38.688267 17741 tablet_copy_client.cc:538] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250625 13:58:38.693827 17741 tablet_bootstrap.cc:492] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607: Bootstrap starting.
I20250625 13:58:38.750411 17744 ts_tablet_manager.cc:927] T 14807c1bcb064549b794601cf576aa4b P 4fc2202863b14ddcb7c61b763182bad8: Initiating tablet copy from peer 0442350ae42f400fa4d5eca497bc9141 (127.13.155.195:36029)
I20250625 13:58:38.751799 17744 tablet_copy_client.cc:323] T 14807c1bcb064549b794601cf576aa4b P 4fc2202863b14ddcb7c61b763182bad8: tablet copy: Beginning tablet copy session from remote peer at address 127.13.155.195:36029
I20250625 13:58:38.762210 17696 tablet_copy_service.cc:140] P 0442350ae42f400fa4d5eca497bc9141: Received BeginTabletCopySession request for tablet 14807c1bcb064549b794601cf576aa4b from peer 4fc2202863b14ddcb7c61b763182bad8 ({username='slave'} at 127.13.155.193:60009)
I20250625 13:58:38.762646 17696 tablet_copy_service.cc:161] P 0442350ae42f400fa4d5eca497bc9141: Beginning new tablet copy session on tablet 14807c1bcb064549b794601cf576aa4b from peer 4fc2202863b14ddcb7c61b763182bad8 at {username='slave'} at 127.13.155.193:60009: session id = 4fc2202863b14ddcb7c61b763182bad8-14807c1bcb064549b794601cf576aa4b
I20250625 13:58:38.766834 17696 tablet_copy_source_session.cc:215] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141: Tablet Copy: opened 0 blocks and 1 log segments
I20250625 13:58:38.769012 17744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14807c1bcb064549b794601cf576aa4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:58:38.776648 17741 tablet_bootstrap.cc:492] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 13:58:38.777246 17741 tablet_bootstrap.cc:492] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607: Bootstrap complete.
I20250625 13:58:38.777666 17741 ts_tablet_manager.cc:1397] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607: Time spent bootstrapping tablet: real 0.084s user 0.076s sys 0.008s
I20250625 13:58:38.778318 17744 tablet_copy_client.cc:806] T 14807c1bcb064549b794601cf576aa4b P 4fc2202863b14ddcb7c61b763182bad8: tablet copy: Starting download of 0 data blocks...
I20250625 13:58:38.778818 17744 tablet_copy_client.cc:670] T 14807c1bcb064549b794601cf576aa4b P 4fc2202863b14ddcb7c61b763182bad8: tablet copy: Starting download of 1 WAL segments...
I20250625 13:58:38.779378 17741 raft_consensus.cc:357] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } } peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: NON_VOTER last_known_addr { host: "127.13.155.194" port: 44817 } attrs { promote: true } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: NON_VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: true } }
I20250625 13:58:38.779958 17741 raft_consensus.cc:738] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 3db2009bfdf246fdbd62df05635ea607, State: Initialized, Role: LEARNER
I20250625 13:58:38.780434 17741 consensus_queue.cc:260] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } } peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: NON_VOTER last_known_addr { host: "127.13.155.194" port: 44817 } attrs { promote: true } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: NON_VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: true } }
I20250625 13:58:38.782311 17741 ts_tablet_manager.cc:1428] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250625 13:58:38.782421 17744 tablet_copy_client.cc:538] T 14807c1bcb064549b794601cf576aa4b P 4fc2202863b14ddcb7c61b763182bad8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250625 13:58:38.783856 17696 tablet_copy_service.cc:342] P 0442350ae42f400fa4d5eca497bc9141: Request end of tablet copy session 3db2009bfdf246fdbd62df05635ea607-14807c1bcb064549b794601cf576aa4b received from {username='slave'} at 127.13.155.194:51313
I20250625 13:58:38.784303 17696 tablet_copy_service.cc:434] P 0442350ae42f400fa4d5eca497bc9141: ending tablet copy session 3db2009bfdf246fdbd62df05635ea607-14807c1bcb064549b794601cf576aa4b on tablet 14807c1bcb064549b794601cf576aa4b with peer 3db2009bfdf246fdbd62df05635ea607
I20250625 13:58:38.789919 17744 tablet_bootstrap.cc:492] T 14807c1bcb064549b794601cf576aa4b P 4fc2202863b14ddcb7c61b763182bad8: Bootstrap starting.
I20250625 13:58:38.860038 17744 tablet_bootstrap.cc:492] T 14807c1bcb064549b794601cf576aa4b P 4fc2202863b14ddcb7c61b763182bad8: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 13:58:38.860675 17744 tablet_bootstrap.cc:492] T 14807c1bcb064549b794601cf576aa4b P 4fc2202863b14ddcb7c61b763182bad8: Bootstrap complete.
I20250625 13:58:38.861147 17744 ts_tablet_manager.cc:1397] T 14807c1bcb064549b794601cf576aa4b P 4fc2202863b14ddcb7c61b763182bad8: Time spent bootstrapping tablet: real 0.071s user 0.059s sys 0.013s
I20250625 13:58:38.862727 17744 raft_consensus.cc:357] T 14807c1bcb064549b794601cf576aa4b P 4fc2202863b14ddcb7c61b763182bad8 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } } peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: NON_VOTER last_known_addr { host: "127.13.155.194" port: 44817 } attrs { promote: true } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: NON_VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: true } }
I20250625 13:58:38.863224 17744 raft_consensus.cc:738] T 14807c1bcb064549b794601cf576aa4b P 4fc2202863b14ddcb7c61b763182bad8 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 4fc2202863b14ddcb7c61b763182bad8, State: Initialized, Role: LEARNER
I20250625 13:58:38.863621 17744 consensus_queue.cc:260] T 14807c1bcb064549b794601cf576aa4b P 4fc2202863b14ddcb7c61b763182bad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } } peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: NON_VOTER last_known_addr { host: "127.13.155.194" port: 44817 } attrs { promote: true } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: NON_VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: true } }
I20250625 13:58:38.865161 17744 ts_tablet_manager.cc:1428] T 14807c1bcb064549b794601cf576aa4b P 4fc2202863b14ddcb7c61b763182bad8: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250625 13:58:38.866624 17696 tablet_copy_service.cc:342] P 0442350ae42f400fa4d5eca497bc9141: Request end of tablet copy session 4fc2202863b14ddcb7c61b763182bad8-14807c1bcb064549b794601cf576aa4b received from {username='slave'} at 127.13.155.193:60009
I20250625 13:58:38.867095 17696 tablet_copy_service.cc:434] P 0442350ae42f400fa4d5eca497bc9141: ending tablet copy session 4fc2202863b14ddcb7c61b763182bad8-14807c1bcb064549b794601cf576aa4b on tablet 14807c1bcb064549b794601cf576aa4b with peer 4fc2202863b14ddcb7c61b763182bad8
I20250625 13:58:38.870589 17748 ts_tablet_manager.cc:927] T 6f5556481b084e90931ef5a1f435126e P 0442350ae42f400fa4d5eca497bc9141: Initiating tablet copy from peer 3db2009bfdf246fdbd62df05635ea607 (127.13.155.194:44817)
I20250625 13:58:38.872466 17748 tablet_copy_client.cc:323] T 6f5556481b084e90931ef5a1f435126e P 0442350ae42f400fa4d5eca497bc9141: tablet copy: Beginning tablet copy session from remote peer at address 127.13.155.194:44817
I20250625 13:58:38.874148 17535 tablet_copy_service.cc:140] P 3db2009bfdf246fdbd62df05635ea607: Received BeginTabletCopySession request for tablet 6f5556481b084e90931ef5a1f435126e from peer 0442350ae42f400fa4d5eca497bc9141 ({username='slave'} at 127.13.155.195:48381)
I20250625 13:58:38.874583 17535 tablet_copy_service.cc:161] P 3db2009bfdf246fdbd62df05635ea607: Beginning new tablet copy session on tablet 6f5556481b084e90931ef5a1f435126e from peer 0442350ae42f400fa4d5eca497bc9141 at {username='slave'} at 127.13.155.195:48381: session id = 0442350ae42f400fa4d5eca497bc9141-6f5556481b084e90931ef5a1f435126e
I20250625 13:58:38.878866 17535 tablet_copy_source_session.cc:215] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607: Tablet Copy: opened 0 blocks and 1 log segments
I20250625 13:58:38.881460 17748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f5556481b084e90931ef5a1f435126e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:58:38.894088 17748 tablet_copy_client.cc:806] T 6f5556481b084e90931ef5a1f435126e P 0442350ae42f400fa4d5eca497bc9141: tablet copy: Starting download of 0 data blocks...
I20250625 13:58:38.894621 17748 tablet_copy_client.cc:670] T 6f5556481b084e90931ef5a1f435126e P 0442350ae42f400fa4d5eca497bc9141: tablet copy: Starting download of 1 WAL segments...
I20250625 13:58:38.897946 17748 tablet_copy_client.cc:538] T 6f5556481b084e90931ef5a1f435126e P 0442350ae42f400fa4d5eca497bc9141: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250625 13:58:38.903781 17748 tablet_bootstrap.cc:492] T 6f5556481b084e90931ef5a1f435126e P 0442350ae42f400fa4d5eca497bc9141: Bootstrap starting.
I20250625 13:58:38.985888 17748 tablet_bootstrap.cc:492] T 6f5556481b084e90931ef5a1f435126e P 0442350ae42f400fa4d5eca497bc9141: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 13:58:38.986598 17748 tablet_bootstrap.cc:492] T 6f5556481b084e90931ef5a1f435126e P 0442350ae42f400fa4d5eca497bc9141: Bootstrap complete.
I20250625 13:58:38.987062 17748 ts_tablet_manager.cc:1397] T 6f5556481b084e90931ef5a1f435126e P 0442350ae42f400fa4d5eca497bc9141: Time spent bootstrapping tablet: real 0.083s user 0.071s sys 0.013s
I20250625 13:58:38.988719 17748 raft_consensus.cc:357] T 6f5556481b084e90931ef5a1f435126e P 0442350ae42f400fa4d5eca497bc9141 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } } peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: NON_VOTER last_known_addr { host: "127.13.155.195" port: 36029 } attrs { promote: true } }
I20250625 13:58:38.989174 17748 raft_consensus.cc:738] T 6f5556481b084e90931ef5a1f435126e P 0442350ae42f400fa4d5eca497bc9141 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 0442350ae42f400fa4d5eca497bc9141, State: Initialized, Role: LEARNER
I20250625 13:58:38.989605 17748 consensus_queue.cc:260] T 6f5556481b084e90931ef5a1f435126e P 0442350ae42f400fa4d5eca497bc9141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } } peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: NON_VOTER last_known_addr { host: "127.13.155.195" port: 36029 } attrs { promote: true } }
I20250625 13:58:38.991130 17748 ts_tablet_manager.cc:1428] T 6f5556481b084e90931ef5a1f435126e P 0442350ae42f400fa4d5eca497bc9141: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250625 13:58:38.992691 17535 tablet_copy_service.cc:342] P 3db2009bfdf246fdbd62df05635ea607: Request end of tablet copy session 0442350ae42f400fa4d5eca497bc9141-6f5556481b084e90931ef5a1f435126e received from {username='slave'} at 127.13.155.195:48381
I20250625 13:58:38.993046 17535 tablet_copy_service.cc:434] P 3db2009bfdf246fdbd62df05635ea607: ending tablet copy session 0442350ae42f400fa4d5eca497bc9141-6f5556481b084e90931ef5a1f435126e on tablet 6f5556481b084e90931ef5a1f435126e with peer 0442350ae42f400fa4d5eca497bc9141
I20250625 13:58:39.206761 13935 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 4fc2202863b14ddcb7c61b763182bad8 to finish bootstrapping
I20250625 13:58:39.233314 13935 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 3db2009bfdf246fdbd62df05635ea607 to finish bootstrapping
I20250625 13:58:39.249115 13935 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 0442350ae42f400fa4d5eca497bc9141 to finish bootstrapping
I20250625 13:58:39.249303 17371 raft_consensus.cc:1215] T 14807c1bcb064549b794601cf576aa4b P 4fc2202863b14ddcb7c61b763182bad8 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20250625 13:58:39.278486 17515 raft_consensus.cc:1215] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20250625 13:58:39.374485 17676 raft_consensus.cc:1215] T 6f5556481b084e90931ef5a1f435126e P 0442350ae42f400fa4d5eca497bc9141 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
I20250625 13:58:39.521476 17656 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250625 13:58:39.524312 17495 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250625 13:58:39.526656 17351 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250625 13:58:39.636315 17715 raft_consensus.cc:1062] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141: attempting to promote NON_VOTER 4fc2202863b14ddcb7c61b763182bad8 to VOTER
I20250625 13:58:39.638794 17715 consensus_queue.cc:237] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } } peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: NON_VOTER last_known_addr { host: "127.13.155.194" port: 44817 } attrs { promote: true } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } }
I20250625 13:58:39.665769 17371 raft_consensus.cc:1273] T 14807c1bcb064549b794601cf576aa4b P 4fc2202863b14ddcb7c61b763182bad8 [term 2 LEARNER]: Refusing update from remote peer 0442350ae42f400fa4d5eca497bc9141: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250625 13:58:39.667621 17762 consensus_queue.cc:1035] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20250625 13:58:39.676404 17515 raft_consensus.cc:1273] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607 [term 2 LEARNER]: Refusing update from remote peer 0442350ae42f400fa4d5eca497bc9141: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250625 13:58:39.677920 17762 consensus_queue.cc:1035] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: NON_VOTER last_known_addr { host: "127.13.155.194" port: 44817 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250625 13:58:39.688802 17754 raft_consensus.cc:2953] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 4fc2202863b14ddcb7c61b763182bad8 (127.13.155.193) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } } peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: NON_VOTER last_known_addr { host: "127.13.155.194" port: 44817 } attrs { promote: true } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } } }
I20250625 13:58:39.691243 17370 raft_consensus.cc:2953] T 14807c1bcb064549b794601cf576aa4b P 4fc2202863b14ddcb7c61b763182bad8 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 4fc2202863b14ddcb7c61b763182bad8 (127.13.155.193) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } } peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: NON_VOTER last_known_addr { host: "127.13.155.194" port: 44817 } attrs { promote: true } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } } }
I20250625 13:58:39.695748 17515 raft_consensus.cc:2953] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607 [term 2 LEARNER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 4fc2202863b14ddcb7c61b763182bad8 (127.13.155.193) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } } peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: NON_VOTER last_known_addr { host: "127.13.155.194" port: 44817 } attrs { promote: true } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } } }
I20250625 13:58:39.698427 17754 raft_consensus.cc:1062] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141: attempting to promote NON_VOTER 3db2009bfdf246fdbd62df05635ea607 to VOTER
I20250625 13:58:39.700582 17754 consensus_queue.cc:237] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } } peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } attrs { promote: false } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } }
I20250625 13:58:39.711678 17227 catalog_manager.cc:5582] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 reported cstate change: config changed from index 9 to 10, 4fc2202863b14ddcb7c61b763182bad8 (127.13.155.193) changed from NON_VOTER to VOTER, now has a pending config: VOTER 0442350ae42f400fa4d5eca497bc9141 (127.13.155.195), VOTER 3db2009bfdf246fdbd62df05635ea607 (127.13.155.194), VOTER 4fc2202863b14ddcb7c61b763182bad8 (127.13.155.193). New cstate: current_term: 2 leader_uuid: "0442350ae42f400fa4d5eca497bc9141" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: NON_VOTER last_known_addr { host: "127.13.155.194" port: 44817 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } } peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } attrs { promote: false } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } } }
I20250625 13:58:39.718403 17370 raft_consensus.cc:1273] T 14807c1bcb064549b794601cf576aa4b P 4fc2202863b14ddcb7c61b763182bad8 [term 2 FOLLOWER]: Refusing update from remote peer 0442350ae42f400fa4d5eca497bc9141: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250625 13:58:39.721262 17715 consensus_queue.cc:1035] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250625 13:58:39.734335 17514 raft_consensus.cc:1273] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607 [term 2 LEARNER]: Refusing update from remote peer 0442350ae42f400fa4d5eca497bc9141: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250625 13:58:39.742276 17370 raft_consensus.cc:2953] T 14807c1bcb064549b794601cf576aa4b P 4fc2202863b14ddcb7c61b763182bad8 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 3db2009bfdf246fdbd62df05635ea607 (127.13.155.194) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } } peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } attrs { promote: false } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } } }
I20250625 13:58:39.754379 17715 consensus_queue.cc:1035] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250625 13:58:39.765359 17762 raft_consensus.cc:2953] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 3db2009bfdf246fdbd62df05635ea607 (127.13.155.194) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } } peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } attrs { promote: false } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } } }
I20250625 13:58:39.770507 17515 raft_consensus.cc:2953] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 3db2009bfdf246fdbd62df05635ea607 (127.13.155.194) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } } peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } attrs { promote: false } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } } }
I20250625 13:58:39.798081 17226 catalog_manager.cc:5582] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 reported cstate change: config changed from index 10 to 11, 3db2009bfdf246fdbd62df05635ea607 (127.13.155.194) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER 0442350ae42f400fa4d5eca497bc9141 (127.13.155.195), VOTER 3db2009bfdf246fdbd62df05635ea607 (127.13.155.194), VOTER 4fc2202863b14ddcb7c61b763182bad8 (127.13.155.193). New cstate: current_term: 2 leader_uuid: "0442350ae42f400fa4d5eca497bc9141" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
322e3b3a9cca41578d3358d15bbd13cb | 127.13.155.254:45625 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.13.155.212:39675 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
0442350ae42f400fa4d5eca497bc9141 | 127.13.155.195:36029 | HEALTHY | <none> | 1 | 0
3db2009bfdf246fdbd62df05635ea607 | 127.13.155.194:44817 | HEALTHY | <none> | 1 | 0
4fc2202863b14ddcb7c61b763182bad8 | 127.13.155.193:41377 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.13.155.193 | experimental | 127.13.155.193:41377
local_ip_for_outbound_sockets | 127.13.155.194 | experimental | 127.13.155.194:44817
local_ip_for_outbound_sockets | 127.13.155.195 | experimental | 127.13.155.195:36029
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.13.155.193:41377
server_dump_info_path | /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.13.155.194:44817
server_dump_info_path | /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.13.155.195:36029
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.13.155.212:39675 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable1 | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable2 | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 3
Tablets | 3
Replicas | 7
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250625 13:58:39.822045 13935 log_verifier.cc:126] Checking tablet 14807c1bcb064549b794601cf576aa4b
I20250625 13:58:39.910423 13935 log_verifier.cc:177] Verified matching terms for 11 ops in tablet 14807c1bcb064549b794601cf576aa4b
I20250625 13:58:39.910769 13935 log_verifier.cc:126] Checking tablet 20344a6b08f946e69ea5a45322f37a8a
I20250625 13:58:39.937631 13935 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 20344a6b08f946e69ea5a45322f37a8a
I20250625 13:58:39.937920 13935 log_verifier.cc:126] Checking tablet 6f5556481b084e90931ef5a1f435126e
I20250625 13:58:39.958791 17764 raft_consensus.cc:1062] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607: attempting to promote NON_VOTER 0442350ae42f400fa4d5eca497bc9141 to VOTER
I20250625 13:58:39.960192 17764 consensus_queue.cc:237] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } } peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } attrs { promote: false } }
I20250625 13:58:39.965899 17370 raft_consensus.cc:1273] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8 [term 2 FOLLOWER]: Refusing update from remote peer 3db2009bfdf246fdbd62df05635ea607: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250625 13:58:39.966390 17676 raft_consensus.cc:1273] T 6f5556481b084e90931ef5a1f435126e P 0442350ae42f400fa4d5eca497bc9141 [term 2 LEARNER]: Refusing update from remote peer 3db2009bfdf246fdbd62df05635ea607: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250625 13:58:39.967197 17794 consensus_queue.cc:1035] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20250625 13:58:39.968040 17764 consensus_queue.cc:1035] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250625 13:58:39.977006 17794 raft_consensus.cc:2953] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 0442350ae42f400fa4d5eca497bc9141 (127.13.155.195) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } } peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } attrs { promote: false } } }
I20250625 13:58:39.978714 17676 raft_consensus.cc:2953] T 6f5556481b084e90931ef5a1f435126e P 0442350ae42f400fa4d5eca497bc9141 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 0442350ae42f400fa4d5eca497bc9141 (127.13.155.195) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } } peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } attrs { promote: false } } }
I20250625 13:58:39.980631 17371 raft_consensus.cc:2953] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 0442350ae42f400fa4d5eca497bc9141 (127.13.155.195) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } } peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } attrs { promote: false } } }
I20250625 13:58:39.990705 17228 catalog_manager.cc:5582] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 reported cstate change: config changed from index 11 to 12, 0442350ae42f400fa4d5eca497bc9141 (127.13.155.195) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "3db2009bfdf246fdbd62df05635ea607" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250625 13:58:40.041720 13935 log_verifier.cc:177] Verified matching terms for 12 ops in tablet 6f5556481b084e90931ef5a1f435126e
I20250625 13:58:40.042160 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 17195
I20250625 13:58:40.072937 13935 minidump.cc:252] Setting minidump size limit to 20M
I20250625 13:58:40.074136 13935 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:58:40.075173 13935 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:58:40.085369 17798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:58:40.085397 17799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:58:40.086839 13935 server_base.cc:1048] running on GCE node
W20250625 13:58:40.087476 17801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:58:40.184991 13935 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250625 13:58:40.185292 13935 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250625 13:58:40.185493 13935 hybrid_clock.cc:648] HybridClock initialized: now 1750859920185473 us; error 0 us; skew 500 ppm
I20250625 13:58:40.186381 13935 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:58:40.190148 13935 webserver.cc:469] Webserver started at http://0.0.0.0:41185/ using document root <none> and password file <none>
I20250625 13:58:40.191270 13935 fs_manager.cc:362] Metadata directory not provided
I20250625 13:58:40.191537 13935 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:58:40.198410 13935 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20250625 13:58:40.203048 17806 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:58:40.204085 13935 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250625 13:58:40.204473 13935 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
uuid: "322e3b3a9cca41578d3358d15bbd13cb"
format_stamp: "Formatted at 2025-06-25 13:58:18 on dist-test-slave-k028"
I20250625 13:58:40.206820 13935 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:58:40.235585 13935 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:58:40.237411 13935 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:58:40.237941 13935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:58:40.250289 13935 sys_catalog.cc:263] Verifying existing consensus state
W20250625 13:58:40.254125 13935 sys_catalog.cc:243] For a single master config, on-disk Raft master: 127.13.155.254:45625 exists but no master address supplied!
I20250625 13:58:40.256264 13935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb: Bootstrap starting.
I20250625 13:58:40.298995 13935 log.cc:826] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb: Log is configured to *not* fsync() on all Append() calls
I20250625 13:58:40.361783 13935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=13 ignored=0} mutations{seen=21 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 13:58:40.362622 13935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb: Bootstrap complete.
I20250625 13:58:40.375429 13935 raft_consensus.cc:357] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322e3b3a9cca41578d3358d15bbd13cb" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 45625 } }
I20250625 13:58:40.375974 13935 raft_consensus.cc:738] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 322e3b3a9cca41578d3358d15bbd13cb, State: Initialized, Role: FOLLOWER
I20250625 13:58:40.376766 13935 consensus_queue.cc:260] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322e3b3a9cca41578d3358d15bbd13cb" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 45625 } }
I20250625 13:58:40.377252 13935 raft_consensus.cc:397] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [term 3 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 13:58:40.377514 13935 raft_consensus.cc:491] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [term 3 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 13:58:40.377818 13935 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [term 3 FOLLOWER]: Advancing to term 4
I20250625 13:58:40.383152 13935 raft_consensus.cc:513] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322e3b3a9cca41578d3358d15bbd13cb" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 45625 } }
I20250625 13:58:40.383827 13935 leader_election.cc:304] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 322e3b3a9cca41578d3358d15bbd13cb; no voters:
I20250625 13:58:40.384972 13935 leader_election.cc:290] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [CANDIDATE]: Term 4 election: Requested vote from peers
I20250625 13:58:40.385246 17813 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [term 4 FOLLOWER]: Leader election won for term 4
I20250625 13:58:40.386587 17813 raft_consensus.cc:695] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [term 4 LEADER]: Becoming Leader. State: Replica: 322e3b3a9cca41578d3358d15bbd13cb, State: Running, Role: LEADER
I20250625 13:58:40.387324 17813 consensus_queue.cc:237] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322e3b3a9cca41578d3358d15bbd13cb" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 45625 } }
I20250625 13:58:40.393764 17814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 leader_uuid: "322e3b3a9cca41578d3358d15bbd13cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322e3b3a9cca41578d3358d15bbd13cb" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 45625 } } }
I20250625 13:58:40.394498 17814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [sys.catalog]: This master's current role is: LEADER
I20250625 13:58:40.395067 17815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 322e3b3a9cca41578d3358d15bbd13cb. Latest consensus state: current_term: 4 leader_uuid: "322e3b3a9cca41578d3358d15bbd13cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322e3b3a9cca41578d3358d15bbd13cb" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 45625 } } }
I20250625 13:58:40.395490 17815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [sys.catalog]: This master's current role is: LEADER
I20250625 13:58:40.421969 13935 tablet_replica.cc:331] stopping tablet replica
I20250625 13:58:40.422709 13935 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [term 4 LEADER]: Raft consensus shutting down.
I20250625 13:58:40.423271 13935 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [term 4 FOLLOWER]: Raft consensus is shut down!
I20250625 13:58:40.426415 13935 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250625 13:58:40.427047 13935 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250625 13:58:40.452342 13935 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
W20250625 13:58:41.012845 17726 heartbeater.cc:646] Failed to heartbeat to 127.13.155.254:45625 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.155.254:45625: connect: Connection refused (error 111)
W20250625 13:58:41.036662 17560 heartbeater.cc:646] Failed to heartbeat to 127.13.155.254:45625 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.155.254:45625: connect: Connection refused (error 111)
W20250625 13:58:41.041203 17416 heartbeater.cc:646] Failed to heartbeat to 127.13.155.254:45625 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.155.254:45625: connect: Connection refused (error 111)
I20250625 13:58:45.718317 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 17265
I20250625 13:58:45.745522 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 17420
I20250625 13:58:45.771515 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 17565
I20250625 13:58:45.804499 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.155.254:45625
--webserver_interface=127.13.155.254
--webserver_port=38733
--builtin_ntp_servers=127.13.155.212:39675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.155.254:45625 with env {}
W20250625 13:58:46.106326 17888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:58:46.106956 17888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:58:46.107435 17888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:58:46.139084 17888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 13:58:46.139407 17888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:58:46.139668 17888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 13:58:46.139902 17888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 13:58:46.175119 17888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:39675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.155.254:45625
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.155.254:45625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.13.155.254
--webserver_port=38733
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:58:46.176472 17888 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:58:46.178110 17888 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:58:46.192458 17894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:58:46.192651 17895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:58:46.194041 17888 server_base.cc:1048] running on GCE node
W20250625 13:58:46.193737 17897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:58:47.371503 17888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:58:47.374610 17888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:58:47.376045 17888 hybrid_clock.cc:648] HybridClock initialized: now 1750859927376012 us; error 47 us; skew 500 ppm
I20250625 13:58:47.376853 17888 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:58:47.383579 17888 webserver.cc:469] Webserver started at http://127.13.155.254:38733/ using document root <none> and password file <none>
I20250625 13:58:47.384505 17888 fs_manager.cc:362] Metadata directory not provided
I20250625 13:58:47.384711 17888 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:58:47.392834 17888 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.002s sys 0.003s
I20250625 13:58:47.397446 17904 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:58:47.398586 17888 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250625 13:58:47.398927 17888 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
uuid: "322e3b3a9cca41578d3358d15bbd13cb"
format_stamp: "Formatted at 2025-06-25 13:58:18 on dist-test-slave-k028"
I20250625 13:58:47.400856 17888 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:58:47.463662 17888 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:58:47.465168 17888 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:58:47.465626 17888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:58:47.537216 17888 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.254:45625
I20250625 13:58:47.537293 17955 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.254:45625 every 8 connection(s)
I20250625 13:58:47.540155 17888 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250625 13:58:47.543557 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 17888
I20250625 13:58:47.545529 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.155.193:41377
--local_ip_for_outbound_sockets=127.13.155.193
--tserver_master_addrs=127.13.155.254:45625
--webserver_port=46795
--webserver_interface=127.13.155.193
--builtin_ntp_servers=127.13.155.212:39675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 13:58:47.556377 17956 sys_catalog.cc:263] Verifying existing consensus state
I20250625 13:58:47.561448 17956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb: Bootstrap starting.
I20250625 13:58:47.571983 17956 log.cc:826] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb: Log is configured to *not* fsync() on all Append() calls
I20250625 13:58:47.650612 17956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=15 ignored=0} mutations{seen=23 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 13:58:47.651410 17956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb: Bootstrap complete.
I20250625 13:58:47.669927 17956 raft_consensus.cc:357] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322e3b3a9cca41578d3358d15bbd13cb" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 45625 } }
I20250625 13:58:47.671916 17956 raft_consensus.cc:738] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 322e3b3a9cca41578d3358d15bbd13cb, State: Initialized, Role: FOLLOWER
I20250625 13:58:47.672654 17956 consensus_queue.cc:260] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322e3b3a9cca41578d3358d15bbd13cb" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 45625 } }
I20250625 13:58:47.673096 17956 raft_consensus.cc:397] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [term 5 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 13:58:47.673363 17956 raft_consensus.cc:491] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [term 5 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 13:58:47.673632 17956 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [term 5 FOLLOWER]: Advancing to term 6
I20250625 13:58:47.678619 17956 raft_consensus.cc:513] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322e3b3a9cca41578d3358d15bbd13cb" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 45625 } }
I20250625 13:58:47.679253 17956 leader_election.cc:304] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 322e3b3a9cca41578d3358d15bbd13cb; no voters:
I20250625 13:58:47.681384 17956 leader_election.cc:290] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [CANDIDATE]: Term 6 election: Requested vote from peers
I20250625 13:58:47.681946 17960 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [term 6 FOLLOWER]: Leader election won for term 6
I20250625 13:58:47.684808 17960 raft_consensus.cc:695] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [term 6 LEADER]: Becoming Leader. State: Replica: 322e3b3a9cca41578d3358d15bbd13cb, State: Running, Role: LEADER
I20250625 13:58:47.685802 17956 sys_catalog.cc:564] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [sys.catalog]: configured and running, proceeding with master startup.
I20250625 13:58:47.685618 17960 consensus_queue.cc:237] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 6, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322e3b3a9cca41578d3358d15bbd13cb" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 45625 } }
I20250625 13:58:47.696732 17962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 322e3b3a9cca41578d3358d15bbd13cb. Latest consensus state: current_term: 6 leader_uuid: "322e3b3a9cca41578d3358d15bbd13cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322e3b3a9cca41578d3358d15bbd13cb" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 45625 } } }
I20250625 13:58:47.697587 17962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [sys.catalog]: This master's current role is: LEADER
I20250625 13:58:47.696611 17961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 leader_uuid: "322e3b3a9cca41578d3358d15bbd13cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322e3b3a9cca41578d3358d15bbd13cb" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 45625 } } }
I20250625 13:58:47.698863 17961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb [sys.catalog]: This master's current role is: LEADER
I20250625 13:58:47.711489 17968 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 13:58:47.728277 17968 catalog_manager.cc:671] Loaded metadata for table TestTable [id=35a14a3a82ec47b0b866bf2fc544a710]
I20250625 13:58:47.729951 17968 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=39b413cde03c404b8f95a92511c6accf]
I20250625 13:58:47.731607 17968 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=ba36d7afb6184074bada1151e4115b10]
I20250625 13:58:47.740731 17968 tablet_loader.cc:96] loaded metadata for tablet 14807c1bcb064549b794601cf576aa4b (table TestTable1 [id=39b413cde03c404b8f95a92511c6accf])
I20250625 13:58:47.742097 17968 tablet_loader.cc:96] loaded metadata for tablet 20344a6b08f946e69ea5a45322f37a8a (table TestTable2 [id=ba36d7afb6184074bada1151e4115b10])
I20250625 13:58:47.743325 17968 tablet_loader.cc:96] loaded metadata for tablet 6f5556481b084e90931ef5a1f435126e (table TestTable [id=35a14a3a82ec47b0b866bf2fc544a710])
I20250625 13:58:47.744755 17968 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 13:58:47.752192 17968 catalog_manager.cc:1261] Loaded cluster ID: cba9adb64c954b8993397fb3176e4ce6
I20250625 13:58:47.752597 17968 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 13:58:47.761966 17968 catalog_manager.cc:1506] Loading token signing keys...
I20250625 13:58:47.765739 17968 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 322e3b3a9cca41578d3358d15bbd13cb: Loaded TSK: 0
I20250625 13:58:47.766981 17968 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250625 13:58:47.939517 17958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:58:47.940018 17958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:58:47.940528 17958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:58:47.971897 17958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:58:47.972857 17958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.155.193
I20250625 13:58:48.008948 17958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:39675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.155.193:41377
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.13.155.193
--webserver_port=46795
--tserver_master_addrs=127.13.155.254:45625
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.155.193
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:58:48.010305 17958 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:58:48.012046 17958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:58:48.030068 17984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:58:48.033226 17986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:58:48.034018 17958 server_base.cc:1048] running on GCE node
W20250625 13:58:48.033649 17983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:58:49.227834 17958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:58:49.230813 17958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:58:49.232280 17958 hybrid_clock.cc:648] HybridClock initialized: now 1750859929232231 us; error 62 us; skew 500 ppm
I20250625 13:58:49.233392 17958 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:58:49.251549 17958 webserver.cc:469] Webserver started at http://127.13.155.193:46795/ using document root <none> and password file <none>
I20250625 13:58:49.252892 17958 fs_manager.cc:362] Metadata directory not provided
I20250625 13:58:49.253212 17958 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:58:49.264180 17958 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.004s sys 0.004s
I20250625 13:58:49.270627 17993 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:58:49.271931 17958 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250625 13:58:49.272373 17958 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "4fc2202863b14ddcb7c61b763182bad8"
format_stamp: "Formatted at 2025-06-25 13:58:19 on dist-test-slave-k028"
I20250625 13:58:49.275252 17958 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:58:49.343361 17958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:58:49.344913 17958 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:58:49.345379 17958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:58:49.348137 17958 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 13:58:49.354310 18000 ts_tablet_manager.cc:542] Loading tablet metadata (0/3 complete)
I20250625 13:58:49.378708 17958 ts_tablet_manager.cc:579] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20250625 13:58:49.378986 17958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.026s user 0.002s sys 0.000s
I20250625 13:58:49.379287 17958 ts_tablet_manager.cc:594] Registering tablets (0/3 complete)
I20250625 13:58:49.384757 18000 tablet_bootstrap.cc:492] T 14807c1bcb064549b794601cf576aa4b P 4fc2202863b14ddcb7c61b763182bad8: Bootstrap starting.
I20250625 13:58:49.390628 17958 ts_tablet_manager.cc:610] Registered 3 tablets
I20250625 13:58:49.390859 17958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.012s user 0.012s sys 0.000s
I20250625 13:58:49.466802 18000 log.cc:826] T 14807c1bcb064549b794601cf576aa4b P 4fc2202863b14ddcb7c61b763182bad8: Log is configured to *not* fsync() on all Append() calls
I20250625 13:58:49.589682 17958 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.193:41377
I20250625 13:58:49.589874 18107 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.193:41377 every 8 connection(s)
I20250625 13:58:49.593482 17958 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250625 13:58:49.603240 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 17958
I20250625 13:58:49.605027 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.155.194:44817
--local_ip_for_outbound_sockets=127.13.155.194
--tserver_master_addrs=127.13.155.254:45625
--webserver_port=34305
--webserver_interface=127.13.155.194
--builtin_ntp_servers=127.13.155.212:39675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 13:58:49.621416 18000 tablet_bootstrap.cc:492] T 14807c1bcb064549b794601cf576aa4b P 4fc2202863b14ddcb7c61b763182bad8: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 13:58:49.622578 18000 tablet_bootstrap.cc:492] T 14807c1bcb064549b794601cf576aa4b P 4fc2202863b14ddcb7c61b763182bad8: Bootstrap complete.
I20250625 13:58:49.624433 18000 ts_tablet_manager.cc:1397] T 14807c1bcb064549b794601cf576aa4b P 4fc2202863b14ddcb7c61b763182bad8: Time spent bootstrapping tablet: real 0.240s user 0.169s sys 0.045s
I20250625 13:58:49.644630 18108 heartbeater.cc:344] Connected to a master server at 127.13.155.254:45625
I20250625 13:58:49.645206 18108 heartbeater.cc:461] Registering TS with master...
I20250625 13:58:49.642220 18000 raft_consensus.cc:357] T 14807c1bcb064549b794601cf576aa4b P 4fc2202863b14ddcb7c61b763182bad8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } } peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } attrs { promote: false } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } }
I20250625 13:58:49.645524 18000 raft_consensus.cc:738] T 14807c1bcb064549b794601cf576aa4b P 4fc2202863b14ddcb7c61b763182bad8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fc2202863b14ddcb7c61b763182bad8, State: Initialized, Role: FOLLOWER
I20250625 13:58:49.646750 18108 heartbeater.cc:507] Master 127.13.155.254:45625 requested a full tablet report, sending...
I20250625 13:58:49.646651 18000 consensus_queue.cc:260] T 14807c1bcb064549b794601cf576aa4b P 4fc2202863b14ddcb7c61b763182bad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } } peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } attrs { promote: false } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } }
I20250625 13:58:49.652547 17921 ts_manager.cc:194] Registered new tserver with Master: 4fc2202863b14ddcb7c61b763182bad8 (127.13.155.193:41377)
I20250625 13:58:49.657748 17921 catalog_manager.cc:5582] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8 reported cstate change: config changed from index -1 to 12, term changed from 0 to 2, VOTER 0442350ae42f400fa4d5eca497bc9141 (127.13.155.195) added, VOTER 3db2009bfdf246fdbd62df05635ea607 (127.13.155.194) added, VOTER 4fc2202863b14ddcb7c61b763182bad8 (127.13.155.193) added. New cstate: current_term: 2 committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } } peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } attrs { promote: false } } }
I20250625 13:58:49.672428 18000 ts_tablet_manager.cc:1428] T 14807c1bcb064549b794601cf576aa4b P 4fc2202863b14ddcb7c61b763182bad8: Time spent starting tablet: real 0.048s user 0.027s sys 0.019s
I20250625 13:58:49.673303 18000 tablet_bootstrap.cc:492] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8: Bootstrap starting.
I20250625 13:58:49.714227 17921 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.155.193:50503
I20250625 13:58:49.718974 18108 heartbeater.cc:499] Master 127.13.155.254:45625 was elected leader, sending a full tablet report...
I20250625 13:58:49.802932 18000 tablet_bootstrap.cc:492] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 13:58:49.803797 18000 tablet_bootstrap.cc:492] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8: Bootstrap complete.
I20250625 13:58:49.805387 18000 ts_tablet_manager.cc:1397] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8: Time spent bootstrapping tablet: real 0.132s user 0.116s sys 0.008s
I20250625 13:58:49.807548 18000 raft_consensus.cc:357] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } }
I20250625 13:58:49.808048 18000 raft_consensus.cc:738] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fc2202863b14ddcb7c61b763182bad8, State: Initialized, Role: FOLLOWER
I20250625 13:58:49.808593 18000 consensus_queue.cc:260] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } }
I20250625 13:58:49.809114 18000 raft_consensus.cc:397] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 13:58:49.809423 18000 raft_consensus.cc:491] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 13:58:49.809814 18000 raft_consensus.cc:3058] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8 [term 2 FOLLOWER]: Advancing to term 3
I20250625 13:58:49.817152 18000 raft_consensus.cc:513] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } }
I20250625 13:58:49.817821 18000 leader_election.cc:304] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fc2202863b14ddcb7c61b763182bad8; no voters:
I20250625 13:58:49.818322 18000 leader_election.cc:290] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250625 13:58:49.818624 18113 raft_consensus.cc:2802] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8 [term 3 FOLLOWER]: Leader election won for term 3
I20250625 13:58:49.821183 18113 raft_consensus.cc:695] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8 [term 3 LEADER]: Becoming Leader. State: Replica: 4fc2202863b14ddcb7c61b763182bad8, State: Running, Role: LEADER
I20250625 13:58:49.821524 18000 ts_tablet_manager.cc:1428] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8: Time spent starting tablet: real 0.016s user 0.017s sys 0.001s
I20250625 13:58:49.822150 18000 tablet_bootstrap.cc:492] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8: Bootstrap starting.
I20250625 13:58:49.822083 18113 consensus_queue.cc:237] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } }
I20250625 13:58:49.837023 17921 catalog_manager.cc:5582] T 20344a6b08f946e69ea5a45322f37a8a P 4fc2202863b14ddcb7c61b763182bad8 reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "4fc2202863b14ddcb7c61b763182bad8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } health_report { overall_health: HEALTHY } } }
I20250625 13:58:49.930868 18000 tablet_bootstrap.cc:492] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 13:58:49.931538 18000 tablet_bootstrap.cc:492] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8: Bootstrap complete.
I20250625 13:58:49.932653 18000 ts_tablet_manager.cc:1397] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8: Time spent bootstrapping tablet: real 0.111s user 0.091s sys 0.016s
I20250625 13:58:49.934103 18000 raft_consensus.cc:357] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } } peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } attrs { promote: false } }
I20250625 13:58:49.934736 18000 raft_consensus.cc:738] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fc2202863b14ddcb7c61b763182bad8, State: Initialized, Role: FOLLOWER
I20250625 13:58:49.935356 18000 consensus_queue.cc:260] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } } peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } attrs { promote: false } }
I20250625 13:58:49.936985 18000 ts_tablet_manager.cc:1428] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20250625 13:58:49.971413 18112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:58:49.971925 18112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:58:49.972440 18112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:58:50.004336 18112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:58:50.005338 18112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.155.194
I20250625 13:58:50.041106 18112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:39675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.155.194:44817
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.13.155.194
--webserver_port=34305
--tserver_master_addrs=127.13.155.254:45625
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.155.194
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:58:50.042428 18112 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:58:50.044152 18112 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:58:50.059746 18129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:58:50.060417 18130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:58:50.060536 18132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:58:50.062186 18112 server_base.cc:1048] running on GCE node
I20250625 13:58:51.226064 18112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:58:51.228430 18112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:58:51.229818 18112 hybrid_clock.cc:648] HybridClock initialized: now 1750859931229776 us; error 53 us; skew 500 ppm
I20250625 13:58:51.230706 18112 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:58:51.237494 18112 webserver.cc:469] Webserver started at http://127.13.155.194:34305/ using document root <none> and password file <none>
I20250625 13:58:51.238420 18112 fs_manager.cc:362] Metadata directory not provided
I20250625 13:58:51.238673 18112 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:58:51.246521 18112 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.001s
I20250625 13:58:51.251215 18139 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:58:51.252224 18112 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250625 13:58:51.252566 18112 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "3db2009bfdf246fdbd62df05635ea607"
format_stamp: "Formatted at 2025-06-25 13:58:21 on dist-test-slave-k028"
I20250625 13:58:51.254554 18112 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:58:51.301044 18112 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:58:51.302515 18112 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:58:51.302954 18112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:58:51.305469 18112 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 13:58:51.311163 18146 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250625 13:58:51.322934 18112 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250625 13:58:51.323222 18112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.000s sys 0.002s
I20250625 13:58:51.323498 18112 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250625 13:58:51.328608 18146 tablet_bootstrap.cc:492] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607: Bootstrap starting.
I20250625 13:58:51.331216 18112 ts_tablet_manager.cc:610] Registered 2 tablets
I20250625 13:58:51.331419 18112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.008s user 0.003s sys 0.005s
I20250625 13:58:51.380929 18169 raft_consensus.cc:491] T 14807c1bcb064549b794601cf576aa4b P 4fc2202863b14ddcb7c61b763182bad8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:58:51.381697 18146 log.cc:826] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607: Log is configured to *not* fsync() on all Append() calls
I20250625 13:58:51.381644 18169 raft_consensus.cc:513] T 14807c1bcb064549b794601cf576aa4b P 4fc2202863b14ddcb7c61b763182bad8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } } peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } attrs { promote: false } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } }
I20250625 13:58:51.385859 18169 leader_election.cc:290] T 14807c1bcb064549b794601cf576aa4b P 4fc2202863b14ddcb7c61b763182bad8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0442350ae42f400fa4d5eca497bc9141 (127.13.155.195:36029), 3db2009bfdf246fdbd62df05635ea607 (127.13.155.194:44817)
W20250625 13:58:51.391989 17996 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.155.195:36029: connect: Connection refused (error 111)
W20250625 13:58:51.398200 17996 leader_election.cc:336] T 14807c1bcb064549b794601cf576aa4b P 4fc2202863b14ddcb7c61b763182bad8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0442350ae42f400fa4d5eca497bc9141 (127.13.155.195:36029): Network error: Client connection negotiation failed: client connection to 127.13.155.195:36029: connect: Connection refused (error 111)
W20250625 13:58:51.399247 17996 leader_election.cc:336] T 14807c1bcb064549b794601cf576aa4b P 4fc2202863b14ddcb7c61b763182bad8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3db2009bfdf246fdbd62df05635ea607 (127.13.155.194:44817): Network error: Client connection negotiation failed: client connection to 127.13.155.194:44817: connect: Connection refused (error 111)
I20250625 13:58:51.399672 17996 leader_election.cc:304] T 14807c1bcb064549b794601cf576aa4b P 4fc2202863b14ddcb7c61b763182bad8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fc2202863b14ddcb7c61b763182bad8; no voters: 0442350ae42f400fa4d5eca497bc9141, 3db2009bfdf246fdbd62df05635ea607
I20250625 13:58:51.400640 18169 raft_consensus.cc:2747] T 14807c1bcb064549b794601cf576aa4b P 4fc2202863b14ddcb7c61b763182bad8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250625 13:58:51.476543 18146 tablet_bootstrap.cc:492] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 13:58:51.477362 18146 tablet_bootstrap.cc:492] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607: Bootstrap complete.
I20250625 13:58:51.478767 18146 ts_tablet_manager.cc:1397] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607: Time spent bootstrapping tablet: real 0.151s user 0.123s sys 0.024s
I20250625 13:58:51.494439 18146 raft_consensus.cc:357] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } } peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } attrs { promote: false } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } }
I20250625 13:58:51.497731 18146 raft_consensus.cc:738] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3db2009bfdf246fdbd62df05635ea607, State: Initialized, Role: FOLLOWER
I20250625 13:58:51.498641 18146 consensus_queue.cc:260] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } } peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } attrs { promote: false } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } }
I20250625 13:58:51.503895 18146 ts_tablet_manager.cc:1428] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607: Time spent starting tablet: real 0.025s user 0.022s sys 0.001s
I20250625 13:58:51.504787 18146 tablet_bootstrap.cc:492] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607: Bootstrap starting.
I20250625 13:58:51.519066 18112 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.194:44817
I20250625 13:58:51.519492 18258 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.194:44817 every 8 connection(s)
I20250625 13:58:51.522321 18112 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250625 13:58:51.530715 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 18112
I20250625 13:58:51.532788 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.155.195:36029
--local_ip_for_outbound_sockets=127.13.155.195
--tserver_master_addrs=127.13.155.254:45625
--webserver_port=40695
--webserver_interface=127.13.155.195
--builtin_ntp_servers=127.13.155.212:39675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 13:58:51.576467 18259 heartbeater.cc:344] Connected to a master server at 127.13.155.254:45625
I20250625 13:58:51.576969 18259 heartbeater.cc:461] Registering TS with master...
I20250625 13:58:51.578251 18259 heartbeater.cc:507] Master 127.13.155.254:45625 requested a full tablet report, sending...
I20250625 13:58:51.582674 17921 ts_manager.cc:194] Registered new tserver with Master: 3db2009bfdf246fdbd62df05635ea607 (127.13.155.194:44817)
I20250625 13:58:51.586874 17921 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.155.194:56579
I20250625 13:58:51.639562 18146 tablet_bootstrap.cc:492] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 13:58:51.640249 18146 tablet_bootstrap.cc:492] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607: Bootstrap complete.
I20250625 13:58:51.641744 18146 ts_tablet_manager.cc:1397] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607: Time spent bootstrapping tablet: real 0.137s user 0.107s sys 0.023s
I20250625 13:58:51.643321 18146 raft_consensus.cc:357] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } } peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } attrs { promote: false } }
I20250625 13:58:51.643746 18146 raft_consensus.cc:738] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3db2009bfdf246fdbd62df05635ea607, State: Initialized, Role: FOLLOWER
I20250625 13:58:51.644214 18146 consensus_queue.cc:260] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } } peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } attrs { promote: false } }
I20250625 13:58:51.645156 18259 heartbeater.cc:499] Master 127.13.155.254:45625 was elected leader, sending a full tablet report...
I20250625 13:58:51.645731 18146 ts_tablet_manager.cc:1428] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250625 13:58:51.747627 18169 raft_consensus.cc:491] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:58:51.748145 18169 raft_consensus.cc:513] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } } peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } attrs { promote: false } }
I20250625 13:58:51.750072 18169 leader_election.cc:290] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3db2009bfdf246fdbd62df05635ea607 (127.13.155.194:44817), 0442350ae42f400fa4d5eca497bc9141 (127.13.155.195:36029)
W20250625 13:58:51.754350 17996 leader_election.cc:336] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0442350ae42f400fa4d5eca497bc9141 (127.13.155.195:36029): Network error: Client connection negotiation failed: client connection to 127.13.155.195:36029: connect: Connection refused (error 111)
I20250625 13:58:51.763495 18213 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f5556481b084e90931ef5a1f435126e" candidate_uuid: "4fc2202863b14ddcb7c61b763182bad8" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "3db2009bfdf246fdbd62df05635ea607" is_pre_election: true
I20250625 13:58:51.764225 18213 raft_consensus.cc:2466] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fc2202863b14ddcb7c61b763182bad8 in term 2.
I20250625 13:58:51.765393 17996 leader_election.cc:304] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3db2009bfdf246fdbd62df05635ea607, 4fc2202863b14ddcb7c61b763182bad8; no voters: 0442350ae42f400fa4d5eca497bc9141
I20250625 13:58:51.766032 18169 raft_consensus.cc:2802] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250625 13:58:51.766383 18169 raft_consensus.cc:491] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:58:51.766747 18169 raft_consensus.cc:3058] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8 [term 2 FOLLOWER]: Advancing to term 3
I20250625 13:58:51.773576 18169 raft_consensus.cc:513] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } } peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } attrs { promote: false } }
I20250625 13:58:51.775532 18169 leader_election.cc:290] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8 [CANDIDATE]: Term 3 election: Requested vote from peers 3db2009bfdf246fdbd62df05635ea607 (127.13.155.194:44817), 0442350ae42f400fa4d5eca497bc9141 (127.13.155.195:36029)
I20250625 13:58:51.776296 18213 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f5556481b084e90931ef5a1f435126e" candidate_uuid: "4fc2202863b14ddcb7c61b763182bad8" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "3db2009bfdf246fdbd62df05635ea607"
I20250625 13:58:51.776767 18213 raft_consensus.cc:3058] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [term 2 FOLLOWER]: Advancing to term 3
W20250625 13:58:51.779559 17996 leader_election.cc:336] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 0442350ae42f400fa4d5eca497bc9141 (127.13.155.195:36029): Network error: Client connection negotiation failed: client connection to 127.13.155.195:36029: connect: Connection refused (error 111)
I20250625 13:58:51.783989 18213 raft_consensus.cc:2466] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fc2202863b14ddcb7c61b763182bad8 in term 3.
I20250625 13:58:51.785187 17996 leader_election.cc:304] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3db2009bfdf246fdbd62df05635ea607, 4fc2202863b14ddcb7c61b763182bad8; no voters: 0442350ae42f400fa4d5eca497bc9141
I20250625 13:58:51.786093 18169 raft_consensus.cc:2802] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8 [term 3 FOLLOWER]: Leader election won for term 3
I20250625 13:58:51.788170 18169 raft_consensus.cc:695] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8 [term 3 LEADER]: Becoming Leader. State: Replica: 4fc2202863b14ddcb7c61b763182bad8, State: Running, Role: LEADER
I20250625 13:58:51.789180 18169 consensus_queue.cc:237] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } } peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } attrs { promote: false } }
I20250625 13:58:51.797930 17921 catalog_manager.cc:5582] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8 reported cstate change: term changed from 2 to 3, leader changed from <none> to 4fc2202863b14ddcb7c61b763182bad8 (127.13.155.193). New cstate: current_term: 3 leader_uuid: "4fc2202863b14ddcb7c61b763182bad8" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250625 13:58:51.877550 18263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:58:51.878012 18263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:58:51.878535 18263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:58:51.910135 18263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:58:51.910954 18263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.155.195
I20250625 13:58:51.945384 18263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:39675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.155.195:36029
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.13.155.195
--webserver_port=40695
--tserver_master_addrs=127.13.155.254:45625
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.155.195
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:58:51.946730 18263 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:58:51.948344 18263 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:58:51.963886 18277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:58:52.176322 18213 raft_consensus.cc:1273] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [term 3 FOLLOWER]: Refusing update from remote peer 4fc2202863b14ddcb7c61b763182bad8: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20250625 13:58:52.177784 18169 consensus_queue.cc:1035] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
W20250625 13:58:52.204006 17996 consensus_peers.cc:487] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8 -> Peer 0442350ae42f400fa4d5eca497bc9141 (127.13.155.195:36029): Couldn't send request to peer 0442350ae42f400fa4d5eca497bc9141. Status: Network error: Client connection negotiation failed: client connection to 127.13.155.195:36029: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250625 13:58:52.231979 18063 consensus_queue.cc:237] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } }
I20250625 13:58:52.236621 18213 raft_consensus.cc:1273] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [term 3 FOLLOWER]: Refusing update from remote peer 4fc2202863b14ddcb7c61b763182bad8: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20250625 13:58:52.238411 18287 consensus_queue.cc:1035] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20250625 13:58:52.246393 18266 raft_consensus.cc:2953] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8 [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER 0442350ae42f400fa4d5eca497bc9141 (127.13.155.195) evicted. New config: { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } } }
I20250625 13:58:52.248507 18213 raft_consensus.cc:2953] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER 0442350ae42f400fa4d5eca497bc9141 (127.13.155.195) evicted. New config: { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } } }
I20250625 13:58:52.257160 17906 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 6f5556481b084e90931ef5a1f435126e with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250625 13:58:52.261620 17921 catalog_manager.cc:5582] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8 reported cstate change: config changed from index 12 to 14, VOTER 0442350ae42f400fa4d5eca497bc9141 (127.13.155.195) evicted. New cstate: current_term: 3 leader_uuid: "4fc2202863b14ddcb7c61b763182bad8" committed_config { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250625 13:58:52.271301 17921 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet 6f5556481b084e90931ef5a1f435126e on TS 0442350ae42f400fa4d5eca497bc9141: Not found: failed to reset TS proxy: Could not find TS for UUID 0442350ae42f400fa4d5eca497bc9141
I20250625 13:58:52.274701 18063 consensus_queue.cc:237] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 12, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } }
I20250625 13:58:52.276466 18287 raft_consensus.cc:2953] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8 [term 3 LEADER]: Committing config change with OpId 3.15: config changed from index 14 to 15, VOTER 3db2009bfdf246fdbd62df05635ea607 (127.13.155.194) evicted. New config: { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } } }
I20250625 13:58:52.282531 17906 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 6f5556481b084e90931ef5a1f435126e with cas_config_opid_index 14: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250625 13:58:52.287473 17921 catalog_manager.cc:5582] T 6f5556481b084e90931ef5a1f435126e P 4fc2202863b14ddcb7c61b763182bad8 reported cstate change: config changed from index 14 to 15, VOTER 3db2009bfdf246fdbd62df05635ea607 (127.13.155.194) evicted. New cstate: current_term: 3 leader_uuid: "4fc2202863b14ddcb7c61b763182bad8" committed_config { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250625 13:58:52.309661 18193 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f5556481b084e90931ef5a1f435126e with delete_type TABLET_DATA_TOMBSTONED (TS 3db2009bfdf246fdbd62df05635ea607 not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:44400
I20250625 13:58:52.314690 18291 tablet_replica.cc:331] stopping tablet replica
I20250625 13:58:52.315313 18291 raft_consensus.cc:2241] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250625 13:58:52.315802 18291 raft_consensus.cc:2270] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250625 13:58:52.318727 18291 ts_tablet_manager.cc:1905] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20250625 13:58:52.323848 17906 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 6f5556481b084e90931ef5a1f435126e on TS 0442350ae42f400fa4d5eca497bc9141 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 0442350ae42f400fa4d5eca497bc9141
I20250625 13:58:52.328603 18291 ts_tablet_manager.cc:1918] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.14
I20250625 13:58:52.328914 18291 log.cc:1199] T 6f5556481b084e90931ef5a1f435126e P 3db2009bfdf246fdbd62df05635ea607: Deleting WAL directory at /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal/wals/6f5556481b084e90931ef5a1f435126e
I20250625 13:58:52.330271 17907 catalog_manager.cc:4928] TS 3db2009bfdf246fdbd62df05635ea607 (127.13.155.194:44817): tablet 6f5556481b084e90931ef5a1f435126e (table TestTable [id=35a14a3a82ec47b0b866bf2fc544a710]) successfully deleted
W20250625 13:58:51.964605 18278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:58:51.967005 18263 server_base.cc:1048] running on GCE node
W20250625 13:58:51.965452 18280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:58:53.174906 18263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:58:53.177016 18263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:58:53.178352 18263 hybrid_clock.cc:648] HybridClock initialized: now 1750859933178324 us; error 46 us; skew 500 ppm
I20250625 13:58:53.179167 18263 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:58:53.185084 18263 webserver.cc:469] Webserver started at http://127.13.155.195:40695/ using document root <none> and password file <none>
I20250625 13:58:53.185956 18263 fs_manager.cc:362] Metadata directory not provided
I20250625 13:58:53.186146 18263 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:58:53.193845 18263 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20250625 13:58:53.198405 18297 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:58:53.199466 18263 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20250625 13:58:53.199743 18263 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "0442350ae42f400fa4d5eca497bc9141"
format_stamp: "Formatted at 2025-06-25 13:58:23 on dist-test-slave-k028"
I20250625 13:58:53.201603 18263 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:58:53.204205 18298 raft_consensus.cc:491] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:58:53.204835 18298 raft_consensus.cc:513] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } } peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } attrs { promote: false } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } }
I20250625 13:58:53.207845 18298 leader_election.cc:290] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0442350ae42f400fa4d5eca497bc9141 (127.13.155.195:36029), 4fc2202863b14ddcb7c61b763182bad8 (127.13.155.193:41377)
W20250625 13:58:53.216922 18142 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.155.195:36029: connect: Connection refused (error 111)
W20250625 13:58:53.222256 18142 leader_election.cc:336] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0442350ae42f400fa4d5eca497bc9141 (127.13.155.195:36029): Network error: Client connection negotiation failed: client connection to 127.13.155.195:36029: connect: Connection refused (error 111)
I20250625 13:58:53.226989 18063 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14807c1bcb064549b794601cf576aa4b" candidate_uuid: "3db2009bfdf246fdbd62df05635ea607" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "4fc2202863b14ddcb7c61b763182bad8" is_pre_election: true
I20250625 13:58:53.227568 18063 raft_consensus.cc:2466] T 14807c1bcb064549b794601cf576aa4b P 4fc2202863b14ddcb7c61b763182bad8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3db2009bfdf246fdbd62df05635ea607 in term 2.
I20250625 13:58:53.228539 18141 leader_election.cc:304] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3db2009bfdf246fdbd62df05635ea607, 4fc2202863b14ddcb7c61b763182bad8; no voters: 0442350ae42f400fa4d5eca497bc9141
I20250625 13:58:53.229233 18298 raft_consensus.cc:2802] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250625 13:58:53.229598 18298 raft_consensus.cc:491] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:58:53.229933 18298 raft_consensus.cc:3058] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607 [term 2 FOLLOWER]: Advancing to term 3
I20250625 13:58:53.237668 18298 raft_consensus.cc:513] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } } peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } attrs { promote: false } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } }
I20250625 13:58:53.239776 18298 leader_election.cc:290] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607 [CANDIDATE]: Term 3 election: Requested vote from peers 0442350ae42f400fa4d5eca497bc9141 (127.13.155.195:36029), 4fc2202863b14ddcb7c61b763182bad8 (127.13.155.193:41377)
I20250625 13:58:53.240829 18063 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14807c1bcb064549b794601cf576aa4b" candidate_uuid: "3db2009bfdf246fdbd62df05635ea607" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "4fc2202863b14ddcb7c61b763182bad8"
I20250625 13:58:53.241462 18063 raft_consensus.cc:3058] T 14807c1bcb064549b794601cf576aa4b P 4fc2202863b14ddcb7c61b763182bad8 [term 2 FOLLOWER]: Advancing to term 3
W20250625 13:58:53.242899 18142 leader_election.cc:336] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 0442350ae42f400fa4d5eca497bc9141 (127.13.155.195:36029): Network error: Client connection negotiation failed: client connection to 127.13.155.195:36029: connect: Connection refused (error 111)
I20250625 13:58:53.248267 18063 raft_consensus.cc:2466] T 14807c1bcb064549b794601cf576aa4b P 4fc2202863b14ddcb7c61b763182bad8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3db2009bfdf246fdbd62df05635ea607 in term 3.
I20250625 13:58:53.249228 18141 leader_election.cc:304] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3db2009bfdf246fdbd62df05635ea607, 4fc2202863b14ddcb7c61b763182bad8; no voters: 0442350ae42f400fa4d5eca497bc9141
I20250625 13:58:53.250070 18298 raft_consensus.cc:2802] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607 [term 3 FOLLOWER]: Leader election won for term 3
I20250625 13:58:53.252117 18298 raft_consensus.cc:695] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607 [term 3 LEADER]: Becoming Leader. State: Replica: 3db2009bfdf246fdbd62df05635ea607, State: Running, Role: LEADER
I20250625 13:58:53.253041 18298 consensus_queue.cc:237] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } } peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } attrs { promote: false } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } }
I20250625 13:58:53.261199 17921 catalog_manager.cc:5582] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607 reported cstate change: term changed from 2 to 3, leader changed from 0442350ae42f400fa4d5eca497bc9141 (127.13.155.195) to 3db2009bfdf246fdbd62df05635ea607 (127.13.155.194). New cstate: current_term: 3 leader_uuid: "3db2009bfdf246fdbd62df05635ea607" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250625 13:58:53.271656 18263 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:58:53.273090 18263 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:58:53.273526 18263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:58:53.276033 18263 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 13:58:53.281630 18314 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250625 13:58:53.293095 18263 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250625 13:58:53.293366 18263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.001s sys 0.001s
I20250625 13:58:53.293634 18263 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250625 13:58:53.298936 18314 tablet_bootstrap.cc:492] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141: Bootstrap starting.
I20250625 13:58:53.302791 18263 ts_tablet_manager.cc:610] Registered 2 tablets
I20250625 13:58:53.303045 18263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.009s user 0.010s sys 0.000s
I20250625 13:58:53.357643 18314 log.cc:826] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141: Log is configured to *not* fsync() on all Append() calls
I20250625 13:58:53.439422 18314 tablet_bootstrap.cc:492] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 13:58:53.440244 18314 tablet_bootstrap.cc:492] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141: Bootstrap complete.
I20250625 13:58:53.441557 18314 ts_tablet_manager.cc:1397] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141: Time spent bootstrapping tablet: real 0.143s user 0.110s sys 0.028s
I20250625 13:58:53.457176 18314 raft_consensus.cc:357] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } } peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } attrs { promote: false } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } }
I20250625 13:58:53.459767 18314 raft_consensus.cc:738] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0442350ae42f400fa4d5eca497bc9141, State: Initialized, Role: FOLLOWER
I20250625 13:58:53.460604 18314 consensus_queue.cc:260] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } } peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } attrs { promote: false } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } }
I20250625 13:58:53.464915 18314 ts_tablet_manager.cc:1428] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141: Time spent starting tablet: real 0.023s user 0.025s sys 0.000s
I20250625 13:58:53.465740 18314 tablet_bootstrap.cc:492] T 6f5556481b084e90931ef5a1f435126e P 0442350ae42f400fa4d5eca497bc9141: Bootstrap starting.
I20250625 13:58:53.487039 18263 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.195:36029
I20250625 13:58:53.487497 18422 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.195:36029 every 8 connection(s)
I20250625 13:58:53.491184 18263 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data/info.pb
W20250625 13:58:53.491472 17906 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 6f5556481b084e90931ef5a1f435126e on TS 0442350ae42f400fa4d5eca497bc9141 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 0442350ae42f400fa4d5eca497bc9141
I20250625 13:58:53.500322 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 18263
I20250625 13:58:53.514803 18423 heartbeater.cc:344] Connected to a master server at 127.13.155.254:45625
I20250625 13:58:53.515344 18423 heartbeater.cc:461] Registering TS with master...
I20250625 13:58:53.516758 18423 heartbeater.cc:507] Master 127.13.155.254:45625 requested a full tablet report, sending...
I20250625 13:58:53.521670 17921 ts_manager.cc:194] Registered new tserver with Master: 0442350ae42f400fa4d5eca497bc9141 (127.13.155.195:36029)
I20250625 13:58:53.526824 17921 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.155.195:48691
I20250625 13:58:53.573299 13935 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 13:58:53.578711 13935 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
W20250625 13:58:53.582752 13935 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 6f5556481b084e90931ef5a1f435126e: tablet_id: "6f5556481b084e90931ef5a1f435126e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
I20250625 13:58:53.620051 18314 tablet_bootstrap.cc:492] T 6f5556481b084e90931ef5a1f435126e P 0442350ae42f400fa4d5eca497bc9141: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 13:58:53.620975 18314 tablet_bootstrap.cc:492] T 6f5556481b084e90931ef5a1f435126e P 0442350ae42f400fa4d5eca497bc9141: Bootstrap complete.
I20250625 13:58:53.622503 18314 ts_tablet_manager.cc:1397] T 6f5556481b084e90931ef5a1f435126e P 0442350ae42f400fa4d5eca497bc9141: Time spent bootstrapping tablet: real 0.157s user 0.127s sys 0.016s
I20250625 13:58:53.624302 18314 raft_consensus.cc:357] T 6f5556481b084e90931ef5a1f435126e P 0442350ae42f400fa4d5eca497bc9141 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } } peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } attrs { promote: false } }
I20250625 13:58:53.624801 18314 raft_consensus.cc:738] T 6f5556481b084e90931ef5a1f435126e P 0442350ae42f400fa4d5eca497bc9141 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0442350ae42f400fa4d5eca497bc9141, State: Initialized, Role: FOLLOWER
I20250625 13:58:53.625444 18314 consensus_queue.cc:260] T 6f5556481b084e90931ef5a1f435126e P 0442350ae42f400fa4d5eca497bc9141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "3db2009bfdf246fdbd62df05635ea607" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 44817 } } peers { permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false } } peers { permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 } attrs { promote: false } }
I20250625 13:58:53.626385 18423 heartbeater.cc:499] Master 127.13.155.254:45625 was elected leader, sending a full tablet report...
I20250625 13:58:53.626843 18314 ts_tablet_manager.cc:1428] T 6f5556481b084e90931ef5a1f435126e P 0442350ae42f400fa4d5eca497bc9141: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250625 13:58:53.793083 18063 raft_consensus.cc:1273] T 14807c1bcb064549b794601cf576aa4b P 4fc2202863b14ddcb7c61b763182bad8 [term 3 FOLLOWER]: Refusing update from remote peer 3db2009bfdf246fdbd62df05635ea607: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20250625 13:58:53.794337 18430 consensus_queue.cc:1035] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fc2202863b14ddcb7c61b763182bad8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250625 13:58:53.827476 18377 raft_consensus.cc:3058] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [term 2 FOLLOWER]: Advancing to term 3
I20250625 13:58:53.833806 18377 raft_consensus.cc:1273] T 14807c1bcb064549b794601cf576aa4b P 0442350ae42f400fa4d5eca497bc9141 [term 3 FOLLOWER]: Refusing update from remote peer 3db2009bfdf246fdbd62df05635ea607: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20250625 13:58:53.835443 18435 consensus_queue.cc:1035] T 14807c1bcb064549b794601cf576aa4b P 3db2009bfdf246fdbd62df05635ea607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0442350ae42f400fa4d5eca497bc9141" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 36029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250625 13:58:54.575369 18357 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f5556481b084e90931ef5a1f435126e with delete_type TABLET_DATA_TOMBSTONED (TS 0442350ae42f400fa4d5eca497bc9141 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:42586
I20250625 13:58:54.577287 18446 tablet_replica.cc:331] stopping tablet replica
I20250625 13:58:54.577889 18446 raft_consensus.cc:2241] T 6f5556481b084e90931ef5a1f435126e P 0442350ae42f400fa4d5eca497bc9141 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250625 13:58:54.578302 18446 raft_consensus.cc:2270] T 6f5556481b084e90931ef5a1f435126e P 0442350ae42f400fa4d5eca497bc9141 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250625 13:58:54.580559 18446 ts_tablet_manager.cc:1905] T 6f5556481b084e90931ef5a1f435126e P 0442350ae42f400fa4d5eca497bc9141: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20250625 13:58:54.586774 13935 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 6f5556481b084e90931ef5a1f435126e: tablet_id: "6f5556481b084e90931ef5a1f435126e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
I20250625 13:58:54.593487 18446 ts_tablet_manager.cc:1918] T 6f5556481b084e90931ef5a1f435126e P 0442350ae42f400fa4d5eca497bc9141: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12
I20250625 13:58:54.593859 18446 log.cc:1199] T 6f5556481b084e90931ef5a1f435126e P 0442350ae42f400fa4d5eca497bc9141: Deleting WAL directory at /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal/wals/6f5556481b084e90931ef5a1f435126e
I20250625 13:58:54.595372 17907 catalog_manager.cc:4928] TS 0442350ae42f400fa4d5eca497bc9141 (127.13.155.195:36029): tablet 6f5556481b084e90931ef5a1f435126e (table TestTable [id=35a14a3a82ec47b0b866bf2fc544a710]) successfully deleted
W20250625 13:58:55.590243 13935 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 6f5556481b084e90931ef5a1f435126e: tablet_id: "6f5556481b084e90931ef5a1f435126e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250625 13:58:56.593684 13935 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 6f5556481b084e90931ef5a1f435126e: tablet_id: "6f5556481b084e90931ef5a1f435126e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250625 13:58:56.909914 18419 debug-util.cc:398] Leaking SignalData structure 0x7b08000c98c0 after lost signal to thread 18272
W20250625 13:58:56.910686 18419 debug-util.cc:398] Leaking SignalData structure 0x7b08000be6c0 after lost signal to thread 18422
W20250625 13:58:57.597112 13935 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 6f5556481b084e90931ef5a1f435126e: tablet_id: "6f5556481b084e90931ef5a1f435126e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250625 13:58:58.601089 13935 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 6f5556481b084e90931ef5a1f435126e: tablet_id: "6f5556481b084e90931ef5a1f435126e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250625 13:58:59.604897 13935 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 6f5556481b084e90931ef5a1f435126e: tablet_id: "6f5556481b084e90931ef5a1f435126e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250625 13:59:00.608830 13935 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 6f5556481b084e90931ef5a1f435126e: tablet_id: "6f5556481b084e90931ef5a1f435126e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250625 13:59:01.612298 13935 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 6f5556481b084e90931ef5a1f435126e: tablet_id: "6f5556481b084e90931ef5a1f435126e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250625 13:59:02.616554 13935 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 6f5556481b084e90931ef5a1f435126e: tablet_id: "6f5556481b084e90931ef5a1f435126e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250625 13:59:03.620528 13935 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 6f5556481b084e90931ef5a1f435126e: tablet_id: "6f5556481b084e90931ef5a1f435126e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250625 13:59:04.624269 13935 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 6f5556481b084e90931ef5a1f435126e: tablet_id: "6f5556481b084e90931ef5a1f435126e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250625 13:59:05.628078 13935 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 6f5556481b084e90931ef5a1f435126e: tablet_id: "6f5556481b084e90931ef5a1f435126e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250625 13:59:06.631819 13935 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 6f5556481b084e90931ef5a1f435126e: tablet_id: "6f5556481b084e90931ef5a1f435126e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250625 13:59:07.635219 13935 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 6f5556481b084e90931ef5a1f435126e: tablet_id: "6f5556481b084e90931ef5a1f435126e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250625 13:59:08.638384 13935 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 6f5556481b084e90931ef5a1f435126e: tablet_id: "6f5556481b084e90931ef5a1f435126e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250625 13:59:09.641837 13935 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 6f5556481b084e90931ef5a1f435126e: tablet_id: "6f5556481b084e90931ef5a1f435126e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250625 13:59:10.645145 13935 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 6f5556481b084e90931ef5a1f435126e: tablet_id: "6f5556481b084e90931ef5a1f435126e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250625 13:59:11.648561 13935 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 6f5556481b084e90931ef5a1f435126e: tablet_id: "6f5556481b084e90931ef5a1f435126e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250625 13:59:12.652621 13935 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 6f5556481b084e90931ef5a1f435126e: tablet_id: "6f5556481b084e90931ef5a1f435126e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/kudu-admin-test.cc:3914: Failure
Failed
Bad status: Not found: not all replicas of tablets comprising table TestTable are registered yet
I20250625 13:59:13.655969 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 17958
I20250625 13:59:13.683097 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 18112
I20250625 13:59:13.711251 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 18263
I20250625 13:59:13.738982 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 17888
2025-06-25T13:59:13Z chronyd exiting
I20250625 13:59:13.789453 13935 test_util.cc:183] -----------------------------------------------
I20250625 13:59:13.789732 13935 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750859832260225-13935-0
[ FAILED ] AdminCliTest.TestRebuildTables (57390 ms)
[----------] 5 tests from AdminCliTest (121462 ms total)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest
[ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4
I20250625 13:59:13.794909 13935 test_util.cc:276] Using random seed: -644288702
I20250625 13:59:13.799260 13935 ts_itest-base.cc:115] Starting cluster with:
I20250625 13:59:13.799419 13935 ts_itest-base.cc:116] --------------
I20250625 13:59:13.799583 13935 ts_itest-base.cc:117] 5 tablet servers
I20250625 13:59:13.799731 13935 ts_itest-base.cc:118] 3 replicas per TS
I20250625 13:59:13.799875 13935 ts_itest-base.cc:119] --------------
2025-06-25T13:59:13Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-25T13:59:13Z Disabled control of system clock
I20250625 13:59:13.843561 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.155.254:38231
--webserver_interface=127.13.155.254
--webserver_port=0
--builtin_ntp_servers=127.13.155.212:37067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.155.254:38231
--raft_prepare_replacement_before_eviction=true with env {}
W20250625 13:59:14.139425 18464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:59:14.139964 18464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:59:14.140374 18464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:59:14.171454 18464 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250625 13:59:14.171806 18464 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 13:59:14.172014 18464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:59:14.172210 18464 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 13:59:14.172415 18464 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 13:59:14.209158 18464 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:37067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.155.254:38231
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.155.254:38231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.13.155.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:59:14.210448 18464 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:59:14.212163 18464 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:59:14.228914 18473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:59:14.228905 18471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:59:14.230218 18464 server_base.cc:1048] running on GCE node
W20250625 13:59:14.228909 18470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:59:15.419978 18464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:59:15.422626 18464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:59:15.424005 18464 hybrid_clock.cc:648] HybridClock initialized: now 1750859955423968 us; error 57 us; skew 500 ppm
I20250625 13:59:15.424813 18464 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:59:15.431135 18464 webserver.cc:469] Webserver started at http://127.13.155.254:44259/ using document root <none> and password file <none>
I20250625 13:59:15.432056 18464 fs_manager.cc:362] Metadata directory not provided
I20250625 13:59:15.432260 18464 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:59:15.432732 18464 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:59:15.437110 18464 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "21d2d49d743e482196f0dc744a79a067"
format_stamp: "Formatted at 2025-06-25 13:59:15 on dist-test-slave-k028"
I20250625 13:59:15.438208 18464 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "21d2d49d743e482196f0dc744a79a067"
format_stamp: "Formatted at 2025-06-25 13:59:15 on dist-test-slave-k028"
I20250625 13:59:15.445151 18464 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250625 13:59:15.450439 18480 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:59:15.451427 18464 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250625 13:59:15.451735 18464 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
uuid: "21d2d49d743e482196f0dc744a79a067"
format_stamp: "Formatted at 2025-06-25 13:59:15 on dist-test-slave-k028"
I20250625 13:59:15.452069 18464 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:59:15.498260 18464 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:59:15.499709 18464 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:59:15.500277 18464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:59:15.570217 18464 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.254:38231
I20250625 13:59:15.570307 18531 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.254:38231 every 8 connection(s)
I20250625 13:59:15.572948 18464 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250625 13:59:15.577216 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 18464
I20250625 13:59:15.577725 13935 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250625 13:59:15.578768 18532 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:59:15.602963 18532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21d2d49d743e482196f0dc744a79a067: Bootstrap starting.
I20250625 13:59:15.608357 18532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21d2d49d743e482196f0dc744a79a067: Neither blocks nor log segments found. Creating new log.
I20250625 13:59:15.609998 18532 log.cc:826] T 00000000000000000000000000000000 P 21d2d49d743e482196f0dc744a79a067: Log is configured to *not* fsync() on all Append() calls
I20250625 13:59:15.614379 18532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21d2d49d743e482196f0dc744a79a067: No bootstrap required, opened a new log
I20250625 13:59:15.631529 18532 raft_consensus.cc:357] T 00000000000000000000000000000000 P 21d2d49d743e482196f0dc744a79a067 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21d2d49d743e482196f0dc744a79a067" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 38231 } }
I20250625 13:59:15.632190 18532 raft_consensus.cc:383] T 00000000000000000000000000000000 P 21d2d49d743e482196f0dc744a79a067 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:59:15.632390 18532 raft_consensus.cc:738] T 00000000000000000000000000000000 P 21d2d49d743e482196f0dc744a79a067 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21d2d49d743e482196f0dc744a79a067, State: Initialized, Role: FOLLOWER
I20250625 13:59:15.633002 18532 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21d2d49d743e482196f0dc744a79a067 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21d2d49d743e482196f0dc744a79a067" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 38231 } }
I20250625 13:59:15.633466 18532 raft_consensus.cc:397] T 00000000000000000000000000000000 P 21d2d49d743e482196f0dc744a79a067 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 13:59:15.633688 18532 raft_consensus.cc:491] T 00000000000000000000000000000000 P 21d2d49d743e482196f0dc744a79a067 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 13:59:15.633939 18532 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 21d2d49d743e482196f0dc744a79a067 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:59:15.638113 18532 raft_consensus.cc:513] T 00000000000000000000000000000000 P 21d2d49d743e482196f0dc744a79a067 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21d2d49d743e482196f0dc744a79a067" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 38231 } }
I20250625 13:59:15.638880 18532 leader_election.cc:304] T 00000000000000000000000000000000 P 21d2d49d743e482196f0dc744a79a067 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21d2d49d743e482196f0dc744a79a067; no voters:
I20250625 13:59:15.640470 18532 leader_election.cc:290] T 00000000000000000000000000000000 P 21d2d49d743e482196f0dc744a79a067 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 13:59:15.641203 18537 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 21d2d49d743e482196f0dc744a79a067 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:59:15.643347 18537 raft_consensus.cc:695] T 00000000000000000000000000000000 P 21d2d49d743e482196f0dc744a79a067 [term 1 LEADER]: Becoming Leader. State: Replica: 21d2d49d743e482196f0dc744a79a067, State: Running, Role: LEADER
I20250625 13:59:15.644081 18537 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21d2d49d743e482196f0dc744a79a067 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21d2d49d743e482196f0dc744a79a067" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 38231 } }
I20250625 13:59:15.645051 18532 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21d2d49d743e482196f0dc744a79a067 [sys.catalog]: configured and running, proceeding with master startup.
I20250625 13:59:15.655331 18539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21d2d49d743e482196f0dc744a79a067 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21d2d49d743e482196f0dc744a79a067. Latest consensus state: current_term: 1 leader_uuid: "21d2d49d743e482196f0dc744a79a067" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21d2d49d743e482196f0dc744a79a067" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 38231 } } }
I20250625 13:59:15.656044 18539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21d2d49d743e482196f0dc744a79a067 [sys.catalog]: This master's current role is: LEADER
I20250625 13:59:15.657245 18538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21d2d49d743e482196f0dc744a79a067 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21d2d49d743e482196f0dc744a79a067" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21d2d49d743e482196f0dc744a79a067" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 38231 } } }
I20250625 13:59:15.658125 18538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21d2d49d743e482196f0dc744a79a067 [sys.catalog]: This master's current role is: LEADER
I20250625 13:59:15.658893 18545 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 13:59:15.670099 18545 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 13:59:15.686858 18545 catalog_manager.cc:1349] Generated new cluster ID: 207039424ca0488eba3e8ce04740c491
I20250625 13:59:15.687129 18545 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 13:59:15.712244 18545 catalog_manager.cc:1372] Generated new certificate authority record
I20250625 13:59:15.713639 18545 catalog_manager.cc:1506] Loading token signing keys...
I20250625 13:59:15.727154 18545 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 21d2d49d743e482196f0dc744a79a067: Generated new TSK 0
I20250625 13:59:15.728044 18545 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 13:59:15.744424 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.155.193:0
--local_ip_for_outbound_sockets=127.13.155.193
--webserver_interface=127.13.155.193
--webserver_port=0
--tserver_master_addrs=127.13.155.254:38231
--builtin_ntp_servers=127.13.155.212:37067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250625 13:59:16.101307 18556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:59:16.101806 18556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:59:16.102294 18556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:59:16.133224 18556 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250625 13:59:16.133626 18556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:59:16.134406 18556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.155.193
I20250625 13:59:16.169517 18556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:37067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.155.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.13.155.193
--webserver_port=0
--tserver_master_addrs=127.13.155.254:38231
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.155.193
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:59:16.170869 18556 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:59:16.172462 18556 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:59:16.189813 18563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:59:16.195277 18565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:59:16.190075 18562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:59:17.605760 18564 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1414 milliseconds
I20250625 13:59:17.605897 18556 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 13:59:17.607149 18556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:59:17.609885 18556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:59:17.611315 18556 hybrid_clock.cc:648] HybridClock initialized: now 1750859957611270 us; error 62 us; skew 500 ppm
I20250625 13:59:17.612145 18556 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:59:17.618945 18556 webserver.cc:469] Webserver started at http://127.13.155.193:43719/ using document root <none> and password file <none>
I20250625 13:59:17.619834 18556 fs_manager.cc:362] Metadata directory not provided
I20250625 13:59:17.620038 18556 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:59:17.620487 18556 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:59:17.624877 18556 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "26c0eb89d20b466d92bac61d8b207449"
format_stamp: "Formatted at 2025-06-25 13:59:17 on dist-test-slave-k028"
I20250625 13:59:17.625962 18556 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "26c0eb89d20b466d92bac61d8b207449"
format_stamp: "Formatted at 2025-06-25 13:59:17 on dist-test-slave-k028"
I20250625 13:59:17.633517 18556 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20250625 13:59:17.639744 18572 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:59:17.640923 18556 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250625 13:59:17.641253 18556 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "26c0eb89d20b466d92bac61d8b207449"
format_stamp: "Formatted at 2025-06-25 13:59:17 on dist-test-slave-k028"
I20250625 13:59:17.641579 18556 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:59:17.692893 18556 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:59:17.694331 18556 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:59:17.694783 18556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:59:17.697824 18556 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 13:59:17.702028 18556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 13:59:17.702239 18556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 13:59:17.702507 18556 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 13:59:17.702680 18556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 13:59:17.863903 18556 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.193:41941
I20250625 13:59:17.864035 18684 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.193:41941 every 8 connection(s)
I20250625 13:59:17.866385 18556 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250625 13:59:17.875308 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 18556
I20250625 13:59:17.875820 13935 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250625 13:59:17.883788 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.155.194:0
--local_ip_for_outbound_sockets=127.13.155.194
--webserver_interface=127.13.155.194
--webserver_port=0
--tserver_master_addrs=127.13.155.254:38231
--builtin_ntp_servers=127.13.155.212:37067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250625 13:59:17.895052 18685 heartbeater.cc:344] Connected to a master server at 127.13.155.254:38231
I20250625 13:59:17.895605 18685 heartbeater.cc:461] Registering TS with master...
I20250625 13:59:17.896697 18685 heartbeater.cc:507] Master 127.13.155.254:38231 requested a full tablet report, sending...
I20250625 13:59:17.899150 18497 ts_manager.cc:194] Registered new tserver with Master: 26c0eb89d20b466d92bac61d8b207449 (127.13.155.193:41941)
I20250625 13:59:17.901029 18497 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.155.193:45549
W20250625 13:59:18.186139 18689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:59:18.186731 18689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:59:18.187266 18689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:59:18.219321 18689 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250625 13:59:18.219754 18689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:59:18.220551 18689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.155.194
I20250625 13:59:18.255797 18689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:37067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.155.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.13.155.194
--webserver_port=0
--tserver_master_addrs=127.13.155.254:38231
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.155.194
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:59:18.257118 18689 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:59:18.258769 18689 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:59:18.275846 18696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:59:18.904469 18685 heartbeater.cc:499] Master 127.13.155.254:38231 was elected leader, sending a full tablet report...
W20250625 13:59:18.276201 18698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:59:18.275903 18695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:59:19.474504 18697 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1196 milliseconds
I20250625 13:59:19.474648 18689 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 13:59:19.475909 18689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:59:19.478256 18689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:59:19.479593 18689 hybrid_clock.cc:648] HybridClock initialized: now 1750859959479569 us; error 69 us; skew 500 ppm
I20250625 13:59:19.480370 18689 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:59:19.487197 18689 webserver.cc:469] Webserver started at http://127.13.155.194:44689/ using document root <none> and password file <none>
I20250625 13:59:19.488165 18689 fs_manager.cc:362] Metadata directory not provided
I20250625 13:59:19.488456 18689 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:59:19.489017 18689 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:59:19.494058 18689 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "9b75809d4b114b0a9e3685ed75a17d33"
format_stamp: "Formatted at 2025-06-25 13:59:19 on dist-test-slave-k028"
I20250625 13:59:19.495178 18689 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "9b75809d4b114b0a9e3685ed75a17d33"
format_stamp: "Formatted at 2025-06-25 13:59:19 on dist-test-slave-k028"
I20250625 13:59:19.502225 18689 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20250625 13:59:19.507668 18705 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:59:19.508656 18689 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250625 13:59:19.508946 18689 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "9b75809d4b114b0a9e3685ed75a17d33"
format_stamp: "Formatted at 2025-06-25 13:59:19 on dist-test-slave-k028"
I20250625 13:59:19.509236 18689 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:59:19.561594 18689 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:59:19.563089 18689 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:59:19.563493 18689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:59:19.565902 18689 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 13:59:19.570402 18689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 13:59:19.570683 18689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250625 13:59:19.570976 18689 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 13:59:19.571185 18689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 13:59:19.709955 18689 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.194:37871
I20250625 13:59:19.710083 18817 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.194:37871 every 8 connection(s)
I20250625 13:59:19.712481 18689 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250625 13:59:19.714350 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 18689
I20250625 13:59:19.714905 13935 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250625 13:59:19.722054 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.155.195:0
--local_ip_for_outbound_sockets=127.13.155.195
--webserver_interface=127.13.155.195
--webserver_port=0
--tserver_master_addrs=127.13.155.254:38231
--builtin_ntp_servers=127.13.155.212:37067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250625 13:59:19.738969 18818 heartbeater.cc:344] Connected to a master server at 127.13.155.254:38231
I20250625 13:59:19.739418 18818 heartbeater.cc:461] Registering TS with master...
I20250625 13:59:19.740427 18818 heartbeater.cc:507] Master 127.13.155.254:38231 requested a full tablet report, sending...
I20250625 13:59:19.742666 18497 ts_manager.cc:194] Registered new tserver with Master: 9b75809d4b114b0a9e3685ed75a17d33 (127.13.155.194:37871)
I20250625 13:59:19.743849 18497 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.155.194:53271
W20250625 13:59:20.021667 18822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:59:20.022178 18822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:59:20.022689 18822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:59:20.054425 18822 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250625 13:59:20.054811 18822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:59:20.055532 18822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.155.195
I20250625 13:59:20.090229 18822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:37067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.155.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.13.155.195
--webserver_port=0
--tserver_master_addrs=127.13.155.254:38231
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.155.195
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:59:20.091472 18822 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:59:20.093053 18822 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:59:20.108690 18828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:59:20.746984 18818 heartbeater.cc:499] Master 127.13.155.254:38231 was elected leader, sending a full tablet report...
W20250625 13:59:20.108793 18829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:59:20.108701 18831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:59:21.300421 18830 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1189 milliseconds
I20250625 13:59:21.300529 18822 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 13:59:21.301740 18822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:59:21.304421 18822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:59:21.305895 18822 hybrid_clock.cc:648] HybridClock initialized: now 1750859961305842 us; error 57 us; skew 500 ppm
I20250625 13:59:21.306784 18822 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:59:21.313524 18822 webserver.cc:469] Webserver started at http://127.13.155.195:35751/ using document root <none> and password file <none>
I20250625 13:59:21.314432 18822 fs_manager.cc:362] Metadata directory not provided
I20250625 13:59:21.314682 18822 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:59:21.315111 18822 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:59:21.319736 18822 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "37008f6ca50a481aa609b44cd70c61f4"
format_stamp: "Formatted at 2025-06-25 13:59:21 on dist-test-slave-k028"
I20250625 13:59:21.320861 18822 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "37008f6ca50a481aa609b44cd70c61f4"
format_stamp: "Formatted at 2025-06-25 13:59:21 on dist-test-slave-k028"
I20250625 13:59:21.327811 18822 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250625 13:59:21.333333 18838 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:59:21.334328 18822 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250625 13:59:21.334671 18822 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "37008f6ca50a481aa609b44cd70c61f4"
format_stamp: "Formatted at 2025-06-25 13:59:21 on dist-test-slave-k028"
I20250625 13:59:21.334964 18822 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:59:21.396232 18822 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:59:21.397728 18822 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:59:21.398130 18822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:59:21.400586 18822 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 13:59:21.404536 18822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 13:59:21.404743 18822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 13:59:21.404937 18822 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 13:59:21.405064 18822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 13:59:21.541805 18822 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.195:34579
I20250625 13:59:21.541944 18950 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.195:34579 every 8 connection(s)
I20250625 13:59:21.544365 18822 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250625 13:59:21.549677 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 18822
I20250625 13:59:21.550302 13935 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250625 13:59:21.557725 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.155.196:0
--local_ip_for_outbound_sockets=127.13.155.196
--webserver_interface=127.13.155.196
--webserver_port=0
--tserver_master_addrs=127.13.155.254:38231
--builtin_ntp_servers=127.13.155.212:37067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250625 13:59:21.568426 18951 heartbeater.cc:344] Connected to a master server at 127.13.155.254:38231
I20250625 13:59:21.568964 18951 heartbeater.cc:461] Registering TS with master...
I20250625 13:59:21.570279 18951 heartbeater.cc:507] Master 127.13.155.254:38231 requested a full tablet report, sending...
I20250625 13:59:21.573366 18497 ts_manager.cc:194] Registered new tserver with Master: 37008f6ca50a481aa609b44cd70c61f4 (127.13.155.195:34579)
I20250625 13:59:21.575388 18497 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.155.195:40021
W20250625 13:59:21.862092 18955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:59:21.862692 18955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:59:21.863204 18955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:59:21.895174 18955 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250625 13:59:21.895591 18955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:59:21.896476 18955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.155.196
I20250625 13:59:21.932127 18955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:37067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.155.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.13.155.196
--webserver_port=0
--tserver_master_addrs=127.13.155.254:38231
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.155.196
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:59:21.933369 18955 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:59:21.934988 18955 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:59:21.952596 18964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:59:22.579751 18951 heartbeater.cc:499] Master 127.13.155.254:38231 was elected leader, sending a full tablet report...
W20250625 13:59:21.952634 18962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:59:21.953028 18961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:59:21.954985 18955 server_base.cc:1048] running on GCE node
I20250625 13:59:23.122545 18955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:59:23.125252 18955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:59:23.126698 18955 hybrid_clock.cc:648] HybridClock initialized: now 1750859963126667 us; error 45 us; skew 500 ppm
I20250625 13:59:23.127477 18955 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:59:23.134272 18955 webserver.cc:469] Webserver started at http://127.13.155.196:46121/ using document root <none> and password file <none>
I20250625 13:59:23.135241 18955 fs_manager.cc:362] Metadata directory not provided
I20250625 13:59:23.135447 18955 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:59:23.135852 18955 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:59:23.140213 18955 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "e8c385dce502433e8f1c96868679b7e6"
format_stamp: "Formatted at 2025-06-25 13:59:23 on dist-test-slave-k028"
I20250625 13:59:23.141251 18955 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "e8c385dce502433e8f1c96868679b7e6"
format_stamp: "Formatted at 2025-06-25 13:59:23 on dist-test-slave-k028"
I20250625 13:59:23.148273 18955 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20250625 13:59:23.153963 18972 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:59:23.155005 18955 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250625 13:59:23.155300 18955 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "e8c385dce502433e8f1c96868679b7e6"
format_stamp: "Formatted at 2025-06-25 13:59:23 on dist-test-slave-k028"
I20250625 13:59:23.155604 18955 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:59:23.212025 18955 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:59:23.213498 18955 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:59:23.213907 18955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:59:23.216358 18955 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 13:59:23.220388 18955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 13:59:23.220656 18955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 13:59:23.220903 18955 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 13:59:23.221065 18955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 13:59:23.355051 18955 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.196:38533
I20250625 13:59:23.355162 19084 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.196:38533 every 8 connection(s)
I20250625 13:59:23.357529 18955 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250625 13:59:23.360071 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 18955
I20250625 13:59:23.360940 13935 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250625 13:59:23.368914 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.155.197:0
--local_ip_for_outbound_sockets=127.13.155.197
--webserver_interface=127.13.155.197
--webserver_port=0
--tserver_master_addrs=127.13.155.254:38231
--builtin_ntp_servers=127.13.155.212:37067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250625 13:59:23.380000 19085 heartbeater.cc:344] Connected to a master server at 127.13.155.254:38231
I20250625 13:59:23.380436 19085 heartbeater.cc:461] Registering TS with master...
I20250625 13:59:23.381472 19085 heartbeater.cc:507] Master 127.13.155.254:38231 requested a full tablet report, sending...
I20250625 13:59:23.383666 18497 ts_manager.cc:194] Registered new tserver with Master: e8c385dce502433e8f1c96868679b7e6 (127.13.155.196:38533)
I20250625 13:59:23.384902 18497 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.155.196:51093
W20250625 13:59:23.678864 19089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:59:23.679360 19089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:59:23.679878 19089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:59:23.711647 19089 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250625 13:59:23.712064 19089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:59:23.712863 19089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.155.197
I20250625 13:59:23.748106 19089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:37067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.155.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.13.155.197
--webserver_port=0
--tserver_master_addrs=127.13.155.254:38231
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.155.197
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:59:23.749430 19089 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:59:23.751066 19089 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:59:23.768069 19096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:59:24.388190 19085 heartbeater.cc:499] Master 127.13.155.254:38231 was elected leader, sending a full tablet report...
W20250625 13:59:23.768060 19095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:59:23.768170 19098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:59:23.770362 19089 server_base.cc:1048] running on GCE node
I20250625 13:59:24.968818 19089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:59:24.971014 19089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:59:24.972348 19089 hybrid_clock.cc:648] HybridClock initialized: now 1750859964972328 us; error 41 us; skew 500 ppm
I20250625 13:59:24.973131 19089 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:59:24.996652 19089 webserver.cc:469] Webserver started at http://127.13.155.197:34205/ using document root <none> and password file <none>
I20250625 13:59:24.997547 19089 fs_manager.cc:362] Metadata directory not provided
I20250625 13:59:24.997740 19089 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:59:24.998131 19089 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:59:25.002499 19089 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "0963e2b68041473c8c57f915c525ecae"
format_stamp: "Formatted at 2025-06-25 13:59:24 on dist-test-slave-k028"
I20250625 13:59:25.003558 19089 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "0963e2b68041473c8c57f915c525ecae"
format_stamp: "Formatted at 2025-06-25 13:59:24 on dist-test-slave-k028"
I20250625 13:59:25.010563 19089 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20250625 13:59:25.015902 19106 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:59:25.016916 19089 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20250625 13:59:25.017215 19089 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "0963e2b68041473c8c57f915c525ecae"
format_stamp: "Formatted at 2025-06-25 13:59:24 on dist-test-slave-k028"
I20250625 13:59:25.017534 19089 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:59:25.064107 19089 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:59:25.065523 19089 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:59:25.065945 19089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:59:25.069051 19089 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 13:59:25.073048 19089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 13:59:25.073247 19089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 13:59:25.073498 19089 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 13:59:25.073662 19089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 13:59:25.221221 19089 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.197:40079
I20250625 13:59:25.221302 19218 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.197:40079 every 8 connection(s)
I20250625 13:59:25.224054 19089 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250625 13:59:25.231127 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 19089
I20250625 13:59:25.231523 13935 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250625 13:59:25.247519 19219 heartbeater.cc:344] Connected to a master server at 127.13.155.254:38231
I20250625 13:59:25.247920 19219 heartbeater.cc:461] Registering TS with master...
I20250625 13:59:25.248857 19219 heartbeater.cc:507] Master 127.13.155.254:38231 requested a full tablet report, sending...
I20250625 13:59:25.251134 18497 ts_manager.cc:194] Registered new tserver with Master: 0963e2b68041473c8c57f915c525ecae (127.13.155.197:40079)
I20250625 13:59:25.252943 18497 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.155.197:34213
I20250625 13:59:25.253003 13935 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250625 13:59:25.290347 18497 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50490:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250625 13:59:25.379477 18620 tablet_service.cc:1468] Processing CreateTablet for tablet 01b7cc87536c405f94e3261cd7d7125c (DEFAULT_TABLE table=TestTable [id=76b478acbb3d4ce8b11453a05e15fb73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:59:25.381316 18620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01b7cc87536c405f94e3261cd7d7125c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:59:25.386278 18886 tablet_service.cc:1468] Processing CreateTablet for tablet 01b7cc87536c405f94e3261cd7d7125c (DEFAULT_TABLE table=TestTable [id=76b478acbb3d4ce8b11453a05e15fb73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:59:25.388051 18886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01b7cc87536c405f94e3261cd7d7125c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:59:25.389914 18753 tablet_service.cc:1468] Processing CreateTablet for tablet 01b7cc87536c405f94e3261cd7d7125c (DEFAULT_TABLE table=TestTable [id=76b478acbb3d4ce8b11453a05e15fb73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:59:25.394240 18753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01b7cc87536c405f94e3261cd7d7125c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:59:25.406569 19238 tablet_bootstrap.cc:492] T 01b7cc87536c405f94e3261cd7d7125c P 26c0eb89d20b466d92bac61d8b207449: Bootstrap starting.
I20250625 13:59:25.414708 19238 tablet_bootstrap.cc:654] T 01b7cc87536c405f94e3261cd7d7125c P 26c0eb89d20b466d92bac61d8b207449: Neither blocks nor log segments found. Creating new log.
I20250625 13:59:25.417773 19238 log.cc:826] T 01b7cc87536c405f94e3261cd7d7125c P 26c0eb89d20b466d92bac61d8b207449: Log is configured to *not* fsync() on all Append() calls
I20250625 13:59:25.420086 19240 tablet_bootstrap.cc:492] T 01b7cc87536c405f94e3261cd7d7125c P 9b75809d4b114b0a9e3685ed75a17d33: Bootstrap starting.
I20250625 13:59:25.420921 19239 tablet_bootstrap.cc:492] T 01b7cc87536c405f94e3261cd7d7125c P 37008f6ca50a481aa609b44cd70c61f4: Bootstrap starting.
I20250625 13:59:25.424917 19238 tablet_bootstrap.cc:492] T 01b7cc87536c405f94e3261cd7d7125c P 26c0eb89d20b466d92bac61d8b207449: No bootstrap required, opened a new log
I20250625 13:59:25.425316 19238 ts_tablet_manager.cc:1397] T 01b7cc87536c405f94e3261cd7d7125c P 26c0eb89d20b466d92bac61d8b207449: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.013s
I20250625 13:59:25.429054 19240 tablet_bootstrap.cc:654] T 01b7cc87536c405f94e3261cd7d7125c P 9b75809d4b114b0a9e3685ed75a17d33: Neither blocks nor log segments found. Creating new log.
I20250625 13:59:25.429117 19239 tablet_bootstrap.cc:654] T 01b7cc87536c405f94e3261cd7d7125c P 37008f6ca50a481aa609b44cd70c61f4: Neither blocks nor log segments found. Creating new log.
I20250625 13:59:25.431818 19239 log.cc:826] T 01b7cc87536c405f94e3261cd7d7125c P 37008f6ca50a481aa609b44cd70c61f4: Log is configured to *not* fsync() on all Append() calls
I20250625 13:59:25.431788 19240 log.cc:826] T 01b7cc87536c405f94e3261cd7d7125c P 9b75809d4b114b0a9e3685ed75a17d33: Log is configured to *not* fsync() on all Append() calls
I20250625 13:59:25.437357 19240 tablet_bootstrap.cc:492] T 01b7cc87536c405f94e3261cd7d7125c P 9b75809d4b114b0a9e3685ed75a17d33: No bootstrap required, opened a new log
I20250625 13:59:25.437764 19240 ts_tablet_manager.cc:1397] T 01b7cc87536c405f94e3261cd7d7125c P 9b75809d4b114b0a9e3685ed75a17d33: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.010s
I20250625 13:59:25.438354 19239 tablet_bootstrap.cc:492] T 01b7cc87536c405f94e3261cd7d7125c P 37008f6ca50a481aa609b44cd70c61f4: No bootstrap required, opened a new log
I20250625 13:59:25.438968 19239 ts_tablet_manager.cc:1397] T 01b7cc87536c405f94e3261cd7d7125c P 37008f6ca50a481aa609b44cd70c61f4: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.012s
I20250625 13:59:25.451889 19238 raft_consensus.cc:357] T 01b7cc87536c405f94e3261cd7d7125c P 26c0eb89d20b466d92bac61d8b207449 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c0eb89d20b466d92bac61d8b207449" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41941 } } peers { permanent_uuid: "9b75809d4b114b0a9e3685ed75a17d33" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 37871 } } peers { permanent_uuid: "37008f6ca50a481aa609b44cd70c61f4" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 34579 } }
I20250625 13:59:25.452816 19238 raft_consensus.cc:383] T 01b7cc87536c405f94e3261cd7d7125c P 26c0eb89d20b466d92bac61d8b207449 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:59:25.453127 19238 raft_consensus.cc:738] T 01b7cc87536c405f94e3261cd7d7125c P 26c0eb89d20b466d92bac61d8b207449 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c0eb89d20b466d92bac61d8b207449, State: Initialized, Role: FOLLOWER
I20250625 13:59:25.454985 19238 consensus_queue.cc:260] T 01b7cc87536c405f94e3261cd7d7125c P 26c0eb89d20b466d92bac61d8b207449 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c0eb89d20b466d92bac61d8b207449" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41941 } } peers { permanent_uuid: "9b75809d4b114b0a9e3685ed75a17d33" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 37871 } } peers { permanent_uuid: "37008f6ca50a481aa609b44cd70c61f4" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 34579 } }
I20250625 13:59:25.459280 19240 raft_consensus.cc:357] T 01b7cc87536c405f94e3261cd7d7125c P 9b75809d4b114b0a9e3685ed75a17d33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c0eb89d20b466d92bac61d8b207449" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41941 } } peers { permanent_uuid: "9b75809d4b114b0a9e3685ed75a17d33" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 37871 } } peers { permanent_uuid: "37008f6ca50a481aa609b44cd70c61f4" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 34579 } }
I20250625 13:59:25.460141 19240 raft_consensus.cc:383] T 01b7cc87536c405f94e3261cd7d7125c P 9b75809d4b114b0a9e3685ed75a17d33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:59:25.460482 19240 raft_consensus.cc:738] T 01b7cc87536c405f94e3261cd7d7125c P 9b75809d4b114b0a9e3685ed75a17d33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b75809d4b114b0a9e3685ed75a17d33, State: Initialized, Role: FOLLOWER
I20250625 13:59:25.461464 19240 consensus_queue.cc:260] T 01b7cc87536c405f94e3261cd7d7125c P 9b75809d4b114b0a9e3685ed75a17d33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c0eb89d20b466d92bac61d8b207449" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41941 } } peers { permanent_uuid: "9b75809d4b114b0a9e3685ed75a17d33" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 37871 } } peers { permanent_uuid: "37008f6ca50a481aa609b44cd70c61f4" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 34579 } }
I20250625 13:59:25.462360 19238 ts_tablet_manager.cc:1428] T 01b7cc87536c405f94e3261cd7d7125c P 26c0eb89d20b466d92bac61d8b207449: Time spent starting tablet: real 0.037s user 0.029s sys 0.008s
I20250625 13:59:25.466389 19239 raft_consensus.cc:357] T 01b7cc87536c405f94e3261cd7d7125c P 37008f6ca50a481aa609b44cd70c61f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c0eb89d20b466d92bac61d8b207449" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41941 } } peers { permanent_uuid: "9b75809d4b114b0a9e3685ed75a17d33" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 37871 } } peers { permanent_uuid: "37008f6ca50a481aa609b44cd70c61f4" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 34579 } }
I20250625 13:59:25.467279 19240 ts_tablet_manager.cc:1428] T 01b7cc87536c405f94e3261cd7d7125c P 9b75809d4b114b0a9e3685ed75a17d33: Time spent starting tablet: real 0.029s user 0.020s sys 0.009s
I20250625 13:59:25.467442 19239 raft_consensus.cc:383] T 01b7cc87536c405f94e3261cd7d7125c P 37008f6ca50a481aa609b44cd70c61f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:59:25.467800 19239 raft_consensus.cc:738] T 01b7cc87536c405f94e3261cd7d7125c P 37008f6ca50a481aa609b44cd70c61f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37008f6ca50a481aa609b44cd70c61f4, State: Initialized, Role: FOLLOWER
I20250625 13:59:25.468662 19239 consensus_queue.cc:260] T 01b7cc87536c405f94e3261cd7d7125c P 37008f6ca50a481aa609b44cd70c61f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c0eb89d20b466d92bac61d8b207449" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41941 } } peers { permanent_uuid: "9b75809d4b114b0a9e3685ed75a17d33" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 37871 } } peers { permanent_uuid: "37008f6ca50a481aa609b44cd70c61f4" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 34579 } }
W20250625 13:59:25.473693 18819 tablet.cc:2378] T 01b7cc87536c405f94e3261cd7d7125c P 9b75809d4b114b0a9e3685ed75a17d33: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 13:59:25.474090 19239 ts_tablet_manager.cc:1428] T 01b7cc87536c405f94e3261cd7d7125c P 37008f6ca50a481aa609b44cd70c61f4: Time spent starting tablet: real 0.035s user 0.033s sys 0.000s
W20250625 13:59:25.522679 18947 debug-util.cc:398] Leaking SignalData structure 0x7b08000ac160 after lost signal to thread 18823
W20250625 13:59:25.552814 18952 tablet.cc:2378] T 01b7cc87536c405f94e3261cd7d7125c P 37008f6ca50a481aa609b44cd70c61f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 13:59:25.595743 19244 raft_consensus.cc:491] T 01b7cc87536c405f94e3261cd7d7125c P 26c0eb89d20b466d92bac61d8b207449 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:59:25.596271 19244 raft_consensus.cc:513] T 01b7cc87536c405f94e3261cd7d7125c P 26c0eb89d20b466d92bac61d8b207449 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c0eb89d20b466d92bac61d8b207449" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41941 } } peers { permanent_uuid: "9b75809d4b114b0a9e3685ed75a17d33" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 37871 } } peers { permanent_uuid: "37008f6ca50a481aa609b44cd70c61f4" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 34579 } }
I20250625 13:59:25.598933 19244 leader_election.cc:290] T 01b7cc87536c405f94e3261cd7d7125c P 26c0eb89d20b466d92bac61d8b207449 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b75809d4b114b0a9e3685ed75a17d33 (127.13.155.194:37871), 37008f6ca50a481aa609b44cd70c61f4 (127.13.155.195:34579)
I20250625 13:59:25.628358 18773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "01b7cc87536c405f94e3261cd7d7125c" candidate_uuid: "26c0eb89d20b466d92bac61d8b207449" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b75809d4b114b0a9e3685ed75a17d33" is_pre_election: true
I20250625 13:59:25.629297 18773 raft_consensus.cc:2466] T 01b7cc87536c405f94e3261cd7d7125c P 9b75809d4b114b0a9e3685ed75a17d33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c0eb89d20b466d92bac61d8b207449 in term 0.
W20250625 13:59:25.630473 18686 tablet.cc:2378] T 01b7cc87536c405f94e3261cd7d7125c P 26c0eb89d20b466d92bac61d8b207449: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 13:59:25.630645 18906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "01b7cc87536c405f94e3261cd7d7125c" candidate_uuid: "26c0eb89d20b466d92bac61d8b207449" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37008f6ca50a481aa609b44cd70c61f4" is_pre_election: true
I20250625 13:59:25.631429 18906 raft_consensus.cc:2466] T 01b7cc87536c405f94e3261cd7d7125c P 37008f6ca50a481aa609b44cd70c61f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c0eb89d20b466d92bac61d8b207449 in term 0.
I20250625 13:59:25.632352 18573 leader_election.cc:304] T 01b7cc87536c405f94e3261cd7d7125c P 26c0eb89d20b466d92bac61d8b207449 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c0eb89d20b466d92bac61d8b207449, 9b75809d4b114b0a9e3685ed75a17d33; no voters:
I20250625 13:59:25.633365 19244 raft_consensus.cc:2802] T 01b7cc87536c405f94e3261cd7d7125c P 26c0eb89d20b466d92bac61d8b207449 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:59:25.633714 19244 raft_consensus.cc:491] T 01b7cc87536c405f94e3261cd7d7125c P 26c0eb89d20b466d92bac61d8b207449 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:59:25.634027 19244 raft_consensus.cc:3058] T 01b7cc87536c405f94e3261cd7d7125c P 26c0eb89d20b466d92bac61d8b207449 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:59:25.638895 19244 raft_consensus.cc:513] T 01b7cc87536c405f94e3261cd7d7125c P 26c0eb89d20b466d92bac61d8b207449 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c0eb89d20b466d92bac61d8b207449" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41941 } } peers { permanent_uuid: "9b75809d4b114b0a9e3685ed75a17d33" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 37871 } } peers { permanent_uuid: "37008f6ca50a481aa609b44cd70c61f4" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 34579 } }
I20250625 13:59:25.640276 19244 leader_election.cc:290] T 01b7cc87536c405f94e3261cd7d7125c P 26c0eb89d20b466d92bac61d8b207449 [CANDIDATE]: Term 1 election: Requested vote from peers 9b75809d4b114b0a9e3685ed75a17d33 (127.13.155.194:37871), 37008f6ca50a481aa609b44cd70c61f4 (127.13.155.195:34579)
I20250625 13:59:25.641113 18773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "01b7cc87536c405f94e3261cd7d7125c" candidate_uuid: "26c0eb89d20b466d92bac61d8b207449" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b75809d4b114b0a9e3685ed75a17d33"
I20250625 13:59:25.641652 18773 raft_consensus.cc:3058] T 01b7cc87536c405f94e3261cd7d7125c P 9b75809d4b114b0a9e3685ed75a17d33 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:59:25.642005 18906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "01b7cc87536c405f94e3261cd7d7125c" candidate_uuid: "26c0eb89d20b466d92bac61d8b207449" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37008f6ca50a481aa609b44cd70c61f4"
I20250625 13:59:25.642624 18906 raft_consensus.cc:3058] T 01b7cc87536c405f94e3261cd7d7125c P 37008f6ca50a481aa609b44cd70c61f4 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:59:25.650252 18906 raft_consensus.cc:2466] T 01b7cc87536c405f94e3261cd7d7125c P 37008f6ca50a481aa609b44cd70c61f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c0eb89d20b466d92bac61d8b207449 in term 1.
I20250625 13:59:25.650743 18773 raft_consensus.cc:2466] T 01b7cc87536c405f94e3261cd7d7125c P 9b75809d4b114b0a9e3685ed75a17d33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c0eb89d20b466d92bac61d8b207449 in term 1.
I20250625 13:59:25.651567 18574 leader_election.cc:304] T 01b7cc87536c405f94e3261cd7d7125c P 26c0eb89d20b466d92bac61d8b207449 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c0eb89d20b466d92bac61d8b207449, 37008f6ca50a481aa609b44cd70c61f4; no voters:
I20250625 13:59:25.652377 19244 raft_consensus.cc:2802] T 01b7cc87536c405f94e3261cd7d7125c P 26c0eb89d20b466d92bac61d8b207449 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:59:25.655176 19244 raft_consensus.cc:695] T 01b7cc87536c405f94e3261cd7d7125c P 26c0eb89d20b466d92bac61d8b207449 [term 1 LEADER]: Becoming Leader. State: Replica: 26c0eb89d20b466d92bac61d8b207449, State: Running, Role: LEADER
I20250625 13:59:25.656139 19244 consensus_queue.cc:237] T 01b7cc87536c405f94e3261cd7d7125c P 26c0eb89d20b466d92bac61d8b207449 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c0eb89d20b466d92bac61d8b207449" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41941 } } peers { permanent_uuid: "9b75809d4b114b0a9e3685ed75a17d33" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 37871 } } peers { permanent_uuid: "37008f6ca50a481aa609b44cd70c61f4" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 34579 } }
I20250625 13:59:25.669626 18496 catalog_manager.cc:5582] T 01b7cc87536c405f94e3261cd7d7125c P 26c0eb89d20b466d92bac61d8b207449 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26c0eb89d20b466d92bac61d8b207449 (127.13.155.193). New cstate: current_term: 1 leader_uuid: "26c0eb89d20b466d92bac61d8b207449" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c0eb89d20b466d92bac61d8b207449" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 41941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b75809d4b114b0a9e3685ed75a17d33" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 37871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37008f6ca50a481aa609b44cd70c61f4" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 34579 } health_report { overall_health: UNKNOWN } } }
I20250625 13:59:25.752451 13935 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250625 13:59:25.756242 13935 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 26c0eb89d20b466d92bac61d8b207449 to finish bootstrapping
I20250625 13:59:25.772691 13935 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 9b75809d4b114b0a9e3685ed75a17d33 to finish bootstrapping
I20250625 13:59:25.785562 13935 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 37008f6ca50a481aa609b44cd70c61f4 to finish bootstrapping
I20250625 13:59:25.802428 13935 test_util.cc:276] Using random seed: -632281180
I20250625 13:59:25.831154 13935 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20250625 13:59:25.832312 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 18689
I20250625 13:59:25.893718 18906 raft_consensus.cc:1273] T 01b7cc87536c405f94e3261cd7d7125c P 37008f6ca50a481aa609b44cd70c61f4 [term 1 FOLLOWER]: Refusing update from remote peer 26c0eb89d20b466d92bac61d8b207449: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250625 13:59:25.894536 18573 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.155.194:37871: connect: Connection refused (error 111)
I20250625 13:59:25.895700 19244 consensus_queue.cc:1035] T 01b7cc87536c405f94e3261cd7d7125c P 26c0eb89d20b466d92bac61d8b207449 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37008f6ca50a481aa609b44cd70c61f4" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 34579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250625 13:59:25.923420 18573 consensus_peers.cc:487] T 01b7cc87536c405f94e3261cd7d7125c P 26c0eb89d20b466d92bac61d8b207449 -> Peer 9b75809d4b114b0a9e3685ed75a17d33 (127.13.155.194:37871): Couldn't send request to peer 9b75809d4b114b0a9e3685ed75a17d33. Status: Network error: Client connection negotiation failed: client connection to 127.13.155.194:37871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250625 13:59:25.941924 19260 mvcc.cc:204] Tried to move back new op lower bound from 7171522420283154432 to 7171522419334897664. Current Snapshot: MvccSnapshot[applied={T|T < 7171522420283154432}]
I20250625 13:59:26.255908 19219 heartbeater.cc:499] Master 127.13.155.254:38231 was elected leader, sending a full tablet report...
W20250625 13:59:28.122826 18573 consensus_peers.cc:487] T 01b7cc87536c405f94e3261cd7d7125c P 26c0eb89d20b466d92bac61d8b207449 -> Peer 9b75809d4b114b0a9e3685ed75a17d33 (127.13.155.194:37871): Couldn't send request to peer 9b75809d4b114b0a9e3685ed75a17d33. Status: Network error: Client connection negotiation failed: client connection to 127.13.155.194:37871: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250625 13:59:28.990569 19154 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 13:59:28.998832 18620 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250625 13:59:29.007073 18886 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 13:59:29.020071 19020 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250625 13:59:30.677114 18573 consensus_peers.cc:487] T 01b7cc87536c405f94e3261cd7d7125c P 26c0eb89d20b466d92bac61d8b207449 -> Peer 9b75809d4b114b0a9e3685ed75a17d33 (127.13.155.194:37871): Couldn't send request to peer 9b75809d4b114b0a9e3685ed75a17d33. Status: Network error: Client connection negotiation failed: client connection to 127.13.155.194:37871: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250625 13:59:30.824970 18620 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250625 13:59:30.840655 19020 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 13:59:30.855087 18886 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 13:59:30.859063 19154 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250625 13:59:31.206221 18573 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.155.194:37871: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20250625 13:59:33.174482 18573 consensus_peers.cc:487] T 01b7cc87536c405f94e3261cd7d7125c P 26c0eb89d20b466d92bac61d8b207449 -> Peer 9b75809d4b114b0a9e3685ed75a17d33 (127.13.155.194:37871): Couldn't send request to peer 9b75809d4b114b0a9e3685ed75a17d33. Status: Network error: Client connection negotiation failed: client connection to 127.13.155.194:37871: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250625 13:59:33.306514 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 18556
I20250625 13:59:33.345393 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 18822
I20250625 13:59:33.387530 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 18955
I20250625 13:59:33.412750 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 19089
I20250625 13:59:33.438026 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 18464
2025-06-25T13:59:33Z chronyd exiting
[ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4 (19705 ms)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest (19705 ms total)
[----------] 1 test from ListTableCliSimpleParamTest
[ RUN ] ListTableCliSimpleParamTest.TestListTables/2
I20250625 13:59:33.499401 13935 test_util.cc:276] Using random seed: -624584207
I20250625 13:59:33.503599 13935 ts_itest-base.cc:115] Starting cluster with:
I20250625 13:59:33.503762 13935 ts_itest-base.cc:116] --------------
I20250625 13:59:33.503875 13935 ts_itest-base.cc:117] 1 tablet servers
I20250625 13:59:33.503980 13935 ts_itest-base.cc:118] 1 replicas per TS
I20250625 13:59:33.504091 13935 ts_itest-base.cc:119] --------------
2025-06-25T13:59:33Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-25T13:59:33Z Disabled control of system clock
I20250625 13:59:33.549023 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.155.254:38739
--webserver_interface=127.13.155.254
--webserver_port=0
--builtin_ntp_servers=127.13.155.212:35285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.155.254:38739 with env {}
W20250625 13:59:33.850430 19367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:59:33.851006 19367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:59:33.851421 19367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:59:33.882753 19367 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 13:59:33.883047 19367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:59:33.883248 19367 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 13:59:33.883450 19367 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 13:59:33.918816 19367 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:35285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.155.254:38739
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.155.254:38739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.13.155.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:59:33.920058 19367 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:59:33.921653 19367 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:59:33.936650 19373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:59:33.936710 19374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:59:33.938061 19367 server_base.cc:1048] running on GCE node
W20250625 13:59:33.936655 19376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:59:35.132346 19367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:59:35.135071 19367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:59:35.136438 19367 hybrid_clock.cc:648] HybridClock initialized: now 1750859975136386 us; error 70 us; skew 500 ppm
I20250625 13:59:35.137281 19367 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:59:35.143985 19367 webserver.cc:469] Webserver started at http://127.13.155.254:32985/ using document root <none> and password file <none>
I20250625 13:59:35.144948 19367 fs_manager.cc:362] Metadata directory not provided
I20250625 13:59:35.145129 19367 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:59:35.145653 19367 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:59:35.150147 19367 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "1ed829e6dbb14b8aaee6997bec022d4e"
format_stamp: "Formatted at 2025-06-25 13:59:35 on dist-test-slave-k028"
I20250625 13:59:35.151412 19367 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "1ed829e6dbb14b8aaee6997bec022d4e"
format_stamp: "Formatted at 2025-06-25 13:59:35 on dist-test-slave-k028"
I20250625 13:59:35.158501 19367 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20250625 13:59:35.163945 19383 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:59:35.164994 19367 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250625 13:59:35.165319 19367 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
uuid: "1ed829e6dbb14b8aaee6997bec022d4e"
format_stamp: "Formatted at 2025-06-25 13:59:35 on dist-test-slave-k028"
I20250625 13:59:35.165638 19367 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:59:35.215906 19367 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:59:35.217422 19367 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:59:35.217880 19367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:59:35.287505 19367 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.254:38739
I20250625 13:59:35.287578 19434 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.254:38739 every 8 connection(s)
I20250625 13:59:35.290148 19367 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250625 13:59:35.294601 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 19367
I20250625 13:59:35.295013 13935 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750859832260225-13935-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250625 13:59:35.295197 19435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:59:35.320921 19435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ed829e6dbb14b8aaee6997bec022d4e: Bootstrap starting.
I20250625 13:59:35.326076 19435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ed829e6dbb14b8aaee6997bec022d4e: Neither blocks nor log segments found. Creating new log.
I20250625 13:59:35.327965 19435 log.cc:826] T 00000000000000000000000000000000 P 1ed829e6dbb14b8aaee6997bec022d4e: Log is configured to *not* fsync() on all Append() calls
I20250625 13:59:35.332285 19435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ed829e6dbb14b8aaee6997bec022d4e: No bootstrap required, opened a new log
I20250625 13:59:35.349582 19435 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1ed829e6dbb14b8aaee6997bec022d4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed829e6dbb14b8aaee6997bec022d4e" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 38739 } }
I20250625 13:59:35.350263 19435 raft_consensus.cc:383] T 00000000000000000000000000000000 P 1ed829e6dbb14b8aaee6997bec022d4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:59:35.350555 19435 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1ed829e6dbb14b8aaee6997bec022d4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ed829e6dbb14b8aaee6997bec022d4e, State: Initialized, Role: FOLLOWER
I20250625 13:59:35.351339 19435 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ed829e6dbb14b8aaee6997bec022d4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed829e6dbb14b8aaee6997bec022d4e" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 38739 } }
I20250625 13:59:35.351861 19435 raft_consensus.cc:397] T 00000000000000000000000000000000 P 1ed829e6dbb14b8aaee6997bec022d4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 13:59:35.352124 19435 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1ed829e6dbb14b8aaee6997bec022d4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 13:59:35.352409 19435 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ed829e6dbb14b8aaee6997bec022d4e [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:59:35.356470 19435 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1ed829e6dbb14b8aaee6997bec022d4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed829e6dbb14b8aaee6997bec022d4e" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 38739 } }
I20250625 13:59:35.357170 19435 leader_election.cc:304] T 00000000000000000000000000000000 P 1ed829e6dbb14b8aaee6997bec022d4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ed829e6dbb14b8aaee6997bec022d4e; no voters:
I20250625 13:59:35.359092 19435 leader_election.cc:290] T 00000000000000000000000000000000 P 1ed829e6dbb14b8aaee6997bec022d4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 13:59:35.359941 19440 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1ed829e6dbb14b8aaee6997bec022d4e [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:59:35.362082 19440 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1ed829e6dbb14b8aaee6997bec022d4e [term 1 LEADER]: Becoming Leader. State: Replica: 1ed829e6dbb14b8aaee6997bec022d4e, State: Running, Role: LEADER
I20250625 13:59:35.362879 19440 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ed829e6dbb14b8aaee6997bec022d4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed829e6dbb14b8aaee6997bec022d4e" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 38739 } }
I20250625 13:59:35.363819 19435 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ed829e6dbb14b8aaee6997bec022d4e [sys.catalog]: configured and running, proceeding with master startup.
I20250625 13:59:35.373278 19442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ed829e6dbb14b8aaee6997bec022d4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ed829e6dbb14b8aaee6997bec022d4e. Latest consensus state: current_term: 1 leader_uuid: "1ed829e6dbb14b8aaee6997bec022d4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed829e6dbb14b8aaee6997bec022d4e" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 38739 } } }
I20250625 13:59:35.374366 19442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ed829e6dbb14b8aaee6997bec022d4e [sys.catalog]: This master's current role is: LEADER
I20250625 13:59:35.373448 19441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ed829e6dbb14b8aaee6997bec022d4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ed829e6dbb14b8aaee6997bec022d4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed829e6dbb14b8aaee6997bec022d4e" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 38739 } } }
I20250625 13:59:35.374886 19441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ed829e6dbb14b8aaee6997bec022d4e [sys.catalog]: This master's current role is: LEADER
I20250625 13:59:35.378777 19450 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 13:59:35.390797 19450 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 13:59:35.405928 19450 catalog_manager.cc:1349] Generated new cluster ID: 19d441f3186c4ec28f2cd56217a695f3
I20250625 13:59:35.406172 19450 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 13:59:35.425908 19450 catalog_manager.cc:1372] Generated new certificate authority record
I20250625 13:59:35.427361 19450 catalog_manager.cc:1506] Loading token signing keys...
I20250625 13:59:35.445313 19450 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 1ed829e6dbb14b8aaee6997bec022d4e: Generated new TSK 0
I20250625 13:59:35.446218 19450 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 13:59:35.464355 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.155.193:0
--local_ip_for_outbound_sockets=127.13.155.193
--webserver_interface=127.13.155.193
--webserver_port=0
--tserver_master_addrs=127.13.155.254:38739
--builtin_ntp_servers=127.13.155.212:35285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250625 13:59:35.853688 19459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:59:35.854188 19459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:59:35.854743 19459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:59:35.885962 19459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:59:35.886891 19459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.155.193
I20250625 13:59:35.921604 19459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:35285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.155.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.13.155.193
--webserver_port=0
--tserver_master_addrs=127.13.155.254:38739
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.155.193
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:59:35.922986 19459 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:59:35.924590 19459 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:59:35.941852 19466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:59:35.943519 19459 server_base.cc:1048] running on GCE node
W20250625 13:59:35.944118 19468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:59:35.944547 19465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:59:37.118075 19459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:59:37.120774 19459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:59:37.122190 19459 hybrid_clock.cc:648] HybridClock initialized: now 1750859977122135 us; error 72 us; skew 500 ppm
I20250625 13:59:37.123008 19459 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:59:37.133808 19459 webserver.cc:469] Webserver started at http://127.13.155.193:38651/ using document root <none> and password file <none>
I20250625 13:59:37.134765 19459 fs_manager.cc:362] Metadata directory not provided
I20250625 13:59:37.134963 19459 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:59:37.135432 19459 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:59:37.139755 19459 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "717e9d1927f840fd8251148b210b4ffe"
format_stamp: "Formatted at 2025-06-25 13:59:37 on dist-test-slave-k028"
I20250625 13:59:37.140859 19459 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "717e9d1927f840fd8251148b210b4ffe"
format_stamp: "Formatted at 2025-06-25 13:59:37 on dist-test-slave-k028"
I20250625 13:59:37.148696 19459 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20250625 13:59:37.154870 19475 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:59:37.155975 19459 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20250625 13:59:37.156288 19459 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "717e9d1927f840fd8251148b210b4ffe"
format_stamp: "Formatted at 2025-06-25 13:59:37 on dist-test-slave-k028"
I20250625 13:59:37.156630 19459 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:59:37.207003 19459 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:59:37.208570 19459 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:59:37.209090 19459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:59:37.212214 19459 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 13:59:37.217034 19459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 13:59:37.217257 19459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 13:59:37.217700 19459 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 13:59:37.217888 19459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 13:59:37.356178 19459 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.193:37967
I20250625 13:59:37.356326 19587 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.193:37967 every 8 connection(s)
I20250625 13:59:37.358762 19459 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250625 13:59:37.367487 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 19459
I20250625 13:59:37.368028 13935 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750859832260225-13935-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250625 13:59:37.380227 19588 heartbeater.cc:344] Connected to a master server at 127.13.155.254:38739
I20250625 13:59:37.380653 19588 heartbeater.cc:461] Registering TS with master...
I20250625 13:59:37.381654 19588 heartbeater.cc:507] Master 127.13.155.254:38739 requested a full tablet report, sending...
I20250625 13:59:37.384060 19400 ts_manager.cc:194] Registered new tserver with Master: 717e9d1927f840fd8251148b210b4ffe (127.13.155.193:37967)
I20250625 13:59:37.386298 19400 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.155.193:45527
I20250625 13:59:37.387926 13935 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250625 13:59:37.417227 19399 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55430:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250625 13:59:37.473105 19523 tablet_service.cc:1468] Processing CreateTablet for tablet ec9298dfcff04a4ca4393bd34bc06bc1 (DEFAULT_TABLE table=TestTable [id=4ce5ff77fdd84da8b02f5e8ceaac76f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:59:37.474567 19523 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec9298dfcff04a4ca4393bd34bc06bc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:59:37.493098 19603 tablet_bootstrap.cc:492] T ec9298dfcff04a4ca4393bd34bc06bc1 P 717e9d1927f840fd8251148b210b4ffe: Bootstrap starting.
I20250625 13:59:37.498548 19603 tablet_bootstrap.cc:654] T ec9298dfcff04a4ca4393bd34bc06bc1 P 717e9d1927f840fd8251148b210b4ffe: Neither blocks nor log segments found. Creating new log.
I20250625 13:59:37.500308 19603 log.cc:826] T ec9298dfcff04a4ca4393bd34bc06bc1 P 717e9d1927f840fd8251148b210b4ffe: Log is configured to *not* fsync() on all Append() calls
I20250625 13:59:37.504896 19603 tablet_bootstrap.cc:492] T ec9298dfcff04a4ca4393bd34bc06bc1 P 717e9d1927f840fd8251148b210b4ffe: No bootstrap required, opened a new log
I20250625 13:59:37.505383 19603 ts_tablet_manager.cc:1397] T ec9298dfcff04a4ca4393bd34bc06bc1 P 717e9d1927f840fd8251148b210b4ffe: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20250625 13:59:37.522634 19603 raft_consensus.cc:357] T ec9298dfcff04a4ca4393bd34bc06bc1 P 717e9d1927f840fd8251148b210b4ffe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "717e9d1927f840fd8251148b210b4ffe" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 37967 } }
I20250625 13:59:37.523195 19603 raft_consensus.cc:383] T ec9298dfcff04a4ca4393bd34bc06bc1 P 717e9d1927f840fd8251148b210b4ffe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:59:37.523439 19603 raft_consensus.cc:738] T ec9298dfcff04a4ca4393bd34bc06bc1 P 717e9d1927f840fd8251148b210b4ffe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717e9d1927f840fd8251148b210b4ffe, State: Initialized, Role: FOLLOWER
I20250625 13:59:37.524101 19603 consensus_queue.cc:260] T ec9298dfcff04a4ca4393bd34bc06bc1 P 717e9d1927f840fd8251148b210b4ffe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "717e9d1927f840fd8251148b210b4ffe" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 37967 } }
I20250625 13:59:37.524593 19603 raft_consensus.cc:397] T ec9298dfcff04a4ca4393bd34bc06bc1 P 717e9d1927f840fd8251148b210b4ffe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 13:59:37.524873 19603 raft_consensus.cc:491] T ec9298dfcff04a4ca4393bd34bc06bc1 P 717e9d1927f840fd8251148b210b4ffe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 13:59:37.525170 19603 raft_consensus.cc:3058] T ec9298dfcff04a4ca4393bd34bc06bc1 P 717e9d1927f840fd8251148b210b4ffe [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:59:37.529924 19603 raft_consensus.cc:513] T ec9298dfcff04a4ca4393bd34bc06bc1 P 717e9d1927f840fd8251148b210b4ffe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "717e9d1927f840fd8251148b210b4ffe" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 37967 } }
I20250625 13:59:37.530612 19603 leader_election.cc:304] T ec9298dfcff04a4ca4393bd34bc06bc1 P 717e9d1927f840fd8251148b210b4ffe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 717e9d1927f840fd8251148b210b4ffe; no voters:
I20250625 13:59:37.532253 19603 leader_election.cc:290] T ec9298dfcff04a4ca4393bd34bc06bc1 P 717e9d1927f840fd8251148b210b4ffe [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 13:59:37.532639 19605 raft_consensus.cc:2802] T ec9298dfcff04a4ca4393bd34bc06bc1 P 717e9d1927f840fd8251148b210b4ffe [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:59:37.536031 19588 heartbeater.cc:499] Master 127.13.155.254:38739 was elected leader, sending a full tablet report...
I20250625 13:59:37.535548 19605 raft_consensus.cc:695] T ec9298dfcff04a4ca4393bd34bc06bc1 P 717e9d1927f840fd8251148b210b4ffe [term 1 LEADER]: Becoming Leader. State: Replica: 717e9d1927f840fd8251148b210b4ffe, State: Running, Role: LEADER
I20250625 13:59:37.536825 19603 ts_tablet_manager.cc:1428] T ec9298dfcff04a4ca4393bd34bc06bc1 P 717e9d1927f840fd8251148b210b4ffe: Time spent starting tablet: real 0.031s user 0.025s sys 0.007s
I20250625 13:59:37.536940 19605 consensus_queue.cc:237] T ec9298dfcff04a4ca4393bd34bc06bc1 P 717e9d1927f840fd8251148b210b4ffe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "717e9d1927f840fd8251148b210b4ffe" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 37967 } }
I20250625 13:59:37.549793 19399 catalog_manager.cc:5582] T ec9298dfcff04a4ca4393bd34bc06bc1 P 717e9d1927f840fd8251148b210b4ffe reported cstate change: term changed from 0 to 1, leader changed from <none> to 717e9d1927f840fd8251148b210b4ffe (127.13.155.193). New cstate: current_term: 1 leader_uuid: "717e9d1927f840fd8251148b210b4ffe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "717e9d1927f840fd8251148b210b4ffe" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 37967 } health_report { overall_health: HEALTHY } } }
I20250625 13:59:37.574441 13935 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250625 13:59:37.577402 13935 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 717e9d1927f840fd8251148b210b4ffe to finish bootstrapping
I20250625 13:59:40.238801 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 19459
I20250625 13:59:40.264634 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 19367
2025-06-25T13:59:40Z chronyd exiting
[ OK ] ListTableCliSimpleParamTest.TestListTables/2 (6818 ms)
[----------] 1 test from ListTableCliSimpleParamTest (6818 ms total)
[----------] 1 test from ListTableCliParamTest
[ RUN ] ListTableCliParamTest.ListTabletWithPartitionInfo/4
I20250625 13:59:40.318070 13935 test_util.cc:276] Using random seed: -617765541
[ OK ] ListTableCliParamTest.ListTabletWithPartitionInfo/4 (12 ms)
[----------] 1 test from ListTableCliParamTest (12 ms total)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest
[ RUN ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0
2025-06-25T13:59:40Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-25T13:59:40Z Disabled control of system clock
I20250625 13:59:40.371061 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.155.254:37319
--webserver_interface=127.13.155.254
--webserver_port=0
--builtin_ntp_servers=127.13.155.212:40803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.155.254:37319 with env {}
W20250625 13:59:40.673689 19631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:59:40.674250 19631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:59:40.674800 19631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:59:40.706053 19631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 13:59:40.706350 19631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:59:40.706574 19631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 13:59:40.706769 19631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 13:59:40.741962 19631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:40803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.155.254:37319
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.155.254:37319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.155.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:59:40.743297 19631 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:59:40.744863 19631 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:59:40.760525 19637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:59:40.760687 19640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:59:40.761168 19638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:59:40.762944 19631 server_base.cc:1048] running on GCE node
I20250625 13:59:41.939805 19631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:59:41.942945 19631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:59:41.944389 19631 hybrid_clock.cc:648] HybridClock initialized: now 1750859981944327 us; error 71 us; skew 500 ppm
I20250625 13:59:41.945158 19631 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:59:41.952216 19631 webserver.cc:469] Webserver started at http://127.13.155.254:44423/ using document root <none> and password file <none>
I20250625 13:59:41.953145 19631 fs_manager.cc:362] Metadata directory not provided
I20250625 13:59:41.953364 19631 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:59:41.953815 19631 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:59:41.958215 19631 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/master-0/data/instance:
uuid: "30167523b98d49a4aa989ad450ea8123"
format_stamp: "Formatted at 2025-06-25 13:59:41 on dist-test-slave-k028"
I20250625 13:59:41.959369 19631 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/master-0/wal/instance:
uuid: "30167523b98d49a4aa989ad450ea8123"
format_stamp: "Formatted at 2025-06-25 13:59:41 on dist-test-slave-k028"
I20250625 13:59:41.966590 19631 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250625 13:59:41.971983 19647 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:59:41.972990 19631 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250625 13:59:41.973351 19631 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/master-0/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/master-0/wal
uuid: "30167523b98d49a4aa989ad450ea8123"
format_stamp: "Formatted at 2025-06-25 13:59:41 on dist-test-slave-k028"
I20250625 13:59:41.973676 19631 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:59:42.023499 19631 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:59:42.024945 19631 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:59:42.025388 19631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:59:42.095907 19631 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.254:37319
I20250625 13:59:42.095996 19698 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.254:37319 every 8 connection(s)
I20250625 13:59:42.098628 19631 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/master-0/data/info.pb
I20250625 13:59:42.103626 19699 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:59:42.105993 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 19631
I20250625 13:59:42.106422 13935 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/master-0/wal/instance
I20250625 13:59:42.124385 19699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30167523b98d49a4aa989ad450ea8123: Bootstrap starting.
I20250625 13:59:42.130568 19699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30167523b98d49a4aa989ad450ea8123: Neither blocks nor log segments found. Creating new log.
I20250625 13:59:42.132268 19699 log.cc:826] T 00000000000000000000000000000000 P 30167523b98d49a4aa989ad450ea8123: Log is configured to *not* fsync() on all Append() calls
I20250625 13:59:42.136667 19699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30167523b98d49a4aa989ad450ea8123: No bootstrap required, opened a new log
I20250625 13:59:42.153818 19699 raft_consensus.cc:357] T 00000000000000000000000000000000 P 30167523b98d49a4aa989ad450ea8123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30167523b98d49a4aa989ad450ea8123" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 37319 } }
I20250625 13:59:42.154531 19699 raft_consensus.cc:383] T 00000000000000000000000000000000 P 30167523b98d49a4aa989ad450ea8123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:59:42.154767 19699 raft_consensus.cc:738] T 00000000000000000000000000000000 P 30167523b98d49a4aa989ad450ea8123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30167523b98d49a4aa989ad450ea8123, State: Initialized, Role: FOLLOWER
I20250625 13:59:42.155484 19699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30167523b98d49a4aa989ad450ea8123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30167523b98d49a4aa989ad450ea8123" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 37319 } }
I20250625 13:59:42.155994 19699 raft_consensus.cc:397] T 00000000000000000000000000000000 P 30167523b98d49a4aa989ad450ea8123 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 13:59:42.156256 19699 raft_consensus.cc:491] T 00000000000000000000000000000000 P 30167523b98d49a4aa989ad450ea8123 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 13:59:42.156539 19699 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 30167523b98d49a4aa989ad450ea8123 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:59:42.160921 19699 raft_consensus.cc:513] T 00000000000000000000000000000000 P 30167523b98d49a4aa989ad450ea8123 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30167523b98d49a4aa989ad450ea8123" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 37319 } }
I20250625 13:59:42.161597 19699 leader_election.cc:304] T 00000000000000000000000000000000 P 30167523b98d49a4aa989ad450ea8123 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30167523b98d49a4aa989ad450ea8123; no voters:
I20250625 13:59:42.163339 19699 leader_election.cc:290] T 00000000000000000000000000000000 P 30167523b98d49a4aa989ad450ea8123 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 13:59:42.163993 19704 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 30167523b98d49a4aa989ad450ea8123 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:59:42.166147 19704 raft_consensus.cc:695] T 00000000000000000000000000000000 P 30167523b98d49a4aa989ad450ea8123 [term 1 LEADER]: Becoming Leader. State: Replica: 30167523b98d49a4aa989ad450ea8123, State: Running, Role: LEADER
I20250625 13:59:42.166980 19704 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30167523b98d49a4aa989ad450ea8123 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30167523b98d49a4aa989ad450ea8123" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 37319 } }
I20250625 13:59:42.167518 19699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30167523b98d49a4aa989ad450ea8123 [sys.catalog]: configured and running, proceeding with master startup.
I20250625 13:59:42.177994 19706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30167523b98d49a4aa989ad450ea8123 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30167523b98d49a4aa989ad450ea8123. Latest consensus state: current_term: 1 leader_uuid: "30167523b98d49a4aa989ad450ea8123" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30167523b98d49a4aa989ad450ea8123" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 37319 } } }
I20250625 13:59:42.179454 19706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30167523b98d49a4aa989ad450ea8123 [sys.catalog]: This master's current role is: LEADER
I20250625 13:59:42.179494 19705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30167523b98d49a4aa989ad450ea8123 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30167523b98d49a4aa989ad450ea8123" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30167523b98d49a4aa989ad450ea8123" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 37319 } } }
I20250625 13:59:42.180884 19705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30167523b98d49a4aa989ad450ea8123 [sys.catalog]: This master's current role is: LEADER
I20250625 13:59:42.184109 19712 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 13:59:42.194365 19712 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 13:59:42.214118 19712 catalog_manager.cc:1349] Generated new cluster ID: bc60b07480b44740ac6121c0a94323dd
I20250625 13:59:42.214690 19712 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 13:59:42.230602 19712 catalog_manager.cc:1372] Generated new certificate authority record
I20250625 13:59:42.232669 19712 catalog_manager.cc:1506] Loading token signing keys...
I20250625 13:59:42.248937 19712 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 30167523b98d49a4aa989ad450ea8123: Generated new TSK 0
I20250625 13:59:42.250053 19712 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 13:59:42.274612 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.155.193:0
--local_ip_for_outbound_sockets=127.13.155.193
--webserver_interface=127.13.155.193
--webserver_port=0
--tserver_master_addrs=127.13.155.254:37319
--builtin_ntp_servers=127.13.155.212:40803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20250625 13:59:42.583667 19723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:59:42.584162 19723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:59:42.584651 19723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:59:42.615968 19723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:59:42.616856 19723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.155.193
I20250625 13:59:42.652076 19723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:40803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.155.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.155.193
--webserver_port=0
--tserver_master_addrs=127.13.155.254:37319
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.155.193
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:59:42.653426 19723 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:59:42.655041 19723 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:59:42.673417 19730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:59:42.677970 19731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:59:44.070290 19729 debug-util.cc:398] Leaking SignalData structure 0x7b0800000a80 after lost signal to thread 19723
W20250625 13:59:44.178954 19729 kernel_stack_watchdog.cc:198] Thread 19723 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:641 for 391ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250625 13:59:44.181658 19732 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1503 milliseconds
W20250625 13:59:44.181855 19723 thread.cc:640] OpenStack (cloud detector) Time spent creating pthread: real 1.505s user 0.001s sys 0.000s
W20250625 13:59:44.182267 19723 thread.cc:606] OpenStack (cloud detector) Time spent starting thread: real 1.506s user 0.002s sys 0.001s
W20250625 13:59:44.183717 19733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:59:44.183805 19723 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 13:59:44.184950 19723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:59:44.187502 19723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:59:44.188939 19723 hybrid_clock.cc:648] HybridClock initialized: now 1750859984188886 us; error 52 us; skew 500 ppm
I20250625 13:59:44.189759 19723 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:59:44.223373 19723 webserver.cc:469] Webserver started at http://127.13.155.193:38793/ using document root <none> and password file <none>
I20250625 13:59:44.224355 19723 fs_manager.cc:362] Metadata directory not provided
I20250625 13:59:44.224555 19723 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:59:44.225036 19723 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:59:44.229389 19723 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-0/data/instance:
uuid: "43a77c62d30c4a3ab625b25b481d6aa8"
format_stamp: "Formatted at 2025-06-25 13:59:44 on dist-test-slave-k028"
I20250625 13:59:44.230489 19723 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-0/wal/instance:
uuid: "43a77c62d30c4a3ab625b25b481d6aa8"
format_stamp: "Formatted at 2025-06-25 13:59:44 on dist-test-slave-k028"
I20250625 13:59:44.238044 19723 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250625 13:59:44.244104 19740 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:59:44.245182 19723 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.005s
I20250625 13:59:44.245471 19723 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-0/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-0/wal
uuid: "43a77c62d30c4a3ab625b25b481d6aa8"
format_stamp: "Formatted at 2025-06-25 13:59:44 on dist-test-slave-k028"
I20250625 13:59:44.245781 19723 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:59:44.305011 19723 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:59:44.306449 19723 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:59:44.306891 19723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:59:44.309564 19723 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 13:59:44.314077 19723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 13:59:44.314294 19723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 13:59:44.314569 19723 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 13:59:44.314734 19723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 13:59:44.479218 19723 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.193:44777
I20250625 13:59:44.479347 19852 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.193:44777 every 8 connection(s)
I20250625 13:59:44.481858 19723 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-0/data/info.pb
I20250625 13:59:44.483626 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 19723
I20250625 13:59:44.484467 13935 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-0/wal/instance
I20250625 13:59:44.494922 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.155.194:0
--local_ip_for_outbound_sockets=127.13.155.194
--webserver_interface=127.13.155.194
--webserver_port=0
--tserver_master_addrs=127.13.155.254:37319
--builtin_ntp_servers=127.13.155.212:40803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250625 13:59:44.514693 19853 heartbeater.cc:344] Connected to a master server at 127.13.155.254:37319
I20250625 13:59:44.515115 19853 heartbeater.cc:461] Registering TS with master...
I20250625 13:59:44.516098 19853 heartbeater.cc:507] Master 127.13.155.254:37319 requested a full tablet report, sending...
I20250625 13:59:44.518548 19664 ts_manager.cc:194] Registered new tserver with Master: 43a77c62d30c4a3ab625b25b481d6aa8 (127.13.155.193:44777)
I20250625 13:59:44.520478 19664 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.155.193:49915
W20250625 13:59:44.838588 19857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:59:44.839108 19857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:59:44.839627 19857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:59:44.870870 19857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:59:44.871716 19857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.155.194
I20250625 13:59:44.905928 19857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:40803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.155.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.155.194
--webserver_port=0
--tserver_master_addrs=127.13.155.254:37319
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.155.194
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:59:44.907330 19857 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:59:44.908978 19857 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:59:44.924507 19866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:59:45.523669 19853 heartbeater.cc:499] Master 127.13.155.254:37319 was elected leader, sending a full tablet report...
W20250625 13:59:44.924484 19864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:59:44.926061 19857 server_base.cc:1048] running on GCE node
W20250625 13:59:44.924484 19863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:59:46.115468 19857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:59:46.117756 19857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:59:46.119109 19857 hybrid_clock.cc:648] HybridClock initialized: now 1750859986119072 us; error 61 us; skew 500 ppm
I20250625 13:59:46.119904 19857 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:59:46.125972 19857 webserver.cc:469] Webserver started at http://127.13.155.194:35289/ using document root <none> and password file <none>
I20250625 13:59:46.126945 19857 fs_manager.cc:362] Metadata directory not provided
I20250625 13:59:46.127158 19857 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:59:46.127657 19857 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:59:46.132130 19857 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-1/data/instance:
uuid: "b8dffb1a245d4ba9b1f383d122e00441"
format_stamp: "Formatted at 2025-06-25 13:59:46 on dist-test-slave-k028"
I20250625 13:59:46.133273 19857 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-1/wal/instance:
uuid: "b8dffb1a245d4ba9b1f383d122e00441"
format_stamp: "Formatted at 2025-06-25 13:59:46 on dist-test-slave-k028"
I20250625 13:59:46.140664 19857 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.001s sys 0.008s
I20250625 13:59:46.146446 19873 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:59:46.147622 19857 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20250625 13:59:46.147924 19857 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-1/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-1/wal
uuid: "b8dffb1a245d4ba9b1f383d122e00441"
format_stamp: "Formatted at 2025-06-25 13:59:46 on dist-test-slave-k028"
I20250625 13:59:46.148259 19857 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20250625 13:59:46.177731 19849 debug-util.cc:398] Leaking SignalData structure 0x7b08000ac060 after lost signal to thread 19724
W20250625 13:59:46.179006 19849 debug-util.cc:398] Leaking SignalData structure 0x7b08000acf40 after lost signal to thread 19852
I20250625 13:59:46.210160 19857 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:59:46.212311 19857 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:59:46.212955 19857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:59:46.216627 19857 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 13:59:46.222710 19857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 13:59:46.223043 19857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 13:59:46.223393 19857 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 13:59:46.223659 19857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 13:59:46.384419 19857 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.194:39411
I20250625 13:59:46.384522 19985 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.194:39411 every 8 connection(s)
I20250625 13:59:46.388454 19857 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-1/data/info.pb
I20250625 13:59:46.399091 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 19857
I20250625 13:59:46.399652 13935 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-1/wal/instance
I20250625 13:59:46.407322 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.155.195:0
--local_ip_for_outbound_sockets=127.13.155.195
--webserver_interface=127.13.155.195
--webserver_port=0
--tserver_master_addrs=127.13.155.254:37319
--builtin_ntp_servers=127.13.155.212:40803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250625 13:59:46.415127 19986 heartbeater.cc:344] Connected to a master server at 127.13.155.254:37319
I20250625 13:59:46.415693 19986 heartbeater.cc:461] Registering TS with master...
I20250625 13:59:46.417259 19986 heartbeater.cc:507] Master 127.13.155.254:37319 requested a full tablet report, sending...
I20250625 13:59:46.420079 19664 ts_manager.cc:194] Registered new tserver with Master: b8dffb1a245d4ba9b1f383d122e00441 (127.13.155.194:39411)
I20250625 13:59:46.421870 19664 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.155.194:36731
W20250625 13:59:46.719822 19990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:59:46.720367 19990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:59:46.720870 19990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:59:46.752353 19990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:59:46.753232 19990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.155.195
I20250625 13:59:46.790046 19990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:40803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.155.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.155.195
--webserver_port=0
--tserver_master_addrs=127.13.155.254:37319
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.155.195
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:59:46.791388 19990 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:59:46.793092 19990 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:59:46.811630 19996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:59:47.426017 19986 heartbeater.cc:499] Master 127.13.155.254:37319 was elected leader, sending a full tablet report...
W20250625 13:59:46.816134 19999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:59:46.812060 19997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:59:48.153934 19998 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1340 milliseconds
I20250625 13:59:48.154018 19990 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 13:59:48.155319 19990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:59:48.157943 19990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:59:48.159356 19990 hybrid_clock.cc:648] HybridClock initialized: now 1750859988159299 us; error 75 us; skew 500 ppm
I20250625 13:59:48.160157 19990 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:59:48.167160 19990 webserver.cc:469] Webserver started at http://127.13.155.195:38691/ using document root <none> and password file <none>
I20250625 13:59:48.168049 19990 fs_manager.cc:362] Metadata directory not provided
I20250625 13:59:48.168252 19990 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:59:48.168741 19990 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:59:48.173115 19990 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-2/data/instance:
uuid: "90ff53d19113439ba739ff76bde14294"
format_stamp: "Formatted at 2025-06-25 13:59:48 on dist-test-slave-k028"
I20250625 13:59:48.174198 19990 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-2/wal/instance:
uuid: "90ff53d19113439ba739ff76bde14294"
format_stamp: "Formatted at 2025-06-25 13:59:48 on dist-test-slave-k028"
I20250625 13:59:48.181885 19990 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250625 13:59:48.188061 20006 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:59:48.189137 19990 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250625 13:59:48.189448 19990 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-2/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-2/wal
uuid: "90ff53d19113439ba739ff76bde14294"
format_stamp: "Formatted at 2025-06-25 13:59:48 on dist-test-slave-k028"
I20250625 13:59:48.189765 19990 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:59:48.253018 19990 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:59:48.254683 19990 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:59:48.255175 19990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:59:48.257937 19990 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 13:59:48.263067 19990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 13:59:48.263279 19990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 13:59:48.263579 19990 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 13:59:48.263754 19990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 13:59:48.408252 19990 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.195:38557
I20250625 13:59:48.408350 20118 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.195:38557 every 8 connection(s)
I20250625 13:59:48.411055 19990 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-2/data/info.pb
I20250625 13:59:48.419616 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 19990
I20250625 13:59:48.420059 13935 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-2/wal/instance
I20250625 13:59:48.433615 20119 heartbeater.cc:344] Connected to a master server at 127.13.155.254:37319
I20250625 13:59:48.434043 20119 heartbeater.cc:461] Registering TS with master...
I20250625 13:59:48.435055 20119 heartbeater.cc:507] Master 127.13.155.254:37319 requested a full tablet report, sending...
I20250625 13:59:48.437068 19664 ts_manager.cc:194] Registered new tserver with Master: 90ff53d19113439ba739ff76bde14294 (127.13.155.195:38557)
I20250625 13:59:48.438542 19664 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.155.195:55239
I20250625 13:59:48.440416 13935 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 13:59:48.470506 13935 test_util.cc:276] Using random seed: -609613105
I20250625 13:59:48.509670 19663 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50048:
name: "pre_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:59:48.512025 19663 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table pre_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:59:48.569248 19788 tablet_service.cc:1468] Processing CreateTablet for tablet db73a6fe1f18431982913f428e57588c (DEFAULT_TABLE table=pre_rebuild [id=39eb9570fe9847e7b19fdebb406a83ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:59:48.570133 20054 tablet_service.cc:1468] Processing CreateTablet for tablet db73a6fe1f18431982913f428e57588c (DEFAULT_TABLE table=pre_rebuild [id=39eb9570fe9847e7b19fdebb406a83ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:59:48.570660 19921 tablet_service.cc:1468] Processing CreateTablet for tablet db73a6fe1f18431982913f428e57588c (DEFAULT_TABLE table=pre_rebuild [id=39eb9570fe9847e7b19fdebb406a83ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:59:48.571198 19788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db73a6fe1f18431982913f428e57588c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:59:48.572019 20054 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db73a6fe1f18431982913f428e57588c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:59:48.572110 19921 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db73a6fe1f18431982913f428e57588c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:59:48.596503 20143 tablet_bootstrap.cc:492] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441: Bootstrap starting.
I20250625 13:59:48.602609 20143 tablet_bootstrap.cc:654] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441: Neither blocks nor log segments found. Creating new log.
I20250625 13:59:48.604836 20145 tablet_bootstrap.cc:492] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8: Bootstrap starting.
I20250625 13:59:48.605604 20144 tablet_bootstrap.cc:492] T db73a6fe1f18431982913f428e57588c P 90ff53d19113439ba739ff76bde14294: Bootstrap starting.
I20250625 13:59:48.607177 20143 log.cc:826] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441: Log is configured to *not* fsync() on all Append() calls
I20250625 13:59:48.612386 20144 tablet_bootstrap.cc:654] T db73a6fe1f18431982913f428e57588c P 90ff53d19113439ba739ff76bde14294: Neither blocks nor log segments found. Creating new log.
I20250625 13:59:48.612560 20145 tablet_bootstrap.cc:654] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8: Neither blocks nor log segments found. Creating new log.
I20250625 13:59:48.613363 20143 tablet_bootstrap.cc:492] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441: No bootstrap required, opened a new log
I20250625 13:59:48.613911 20143 ts_tablet_manager.cc:1397] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.000s
I20250625 13:59:48.614189 20144 log.cc:826] T db73a6fe1f18431982913f428e57588c P 90ff53d19113439ba739ff76bde14294: Log is configured to *not* fsync() on all Append() calls
I20250625 13:59:48.614895 20145 log.cc:826] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8: Log is configured to *not* fsync() on all Append() calls
I20250625 13:59:48.619421 20144 tablet_bootstrap.cc:492] T db73a6fe1f18431982913f428e57588c P 90ff53d19113439ba739ff76bde14294: No bootstrap required, opened a new log
I20250625 13:59:48.619832 20144 ts_tablet_manager.cc:1397] T db73a6fe1f18431982913f428e57588c P 90ff53d19113439ba739ff76bde14294: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20250625 13:59:48.621492 20145 tablet_bootstrap.cc:492] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8: No bootstrap required, opened a new log
I20250625 13:59:48.621964 20145 ts_tablet_manager.cc:1397] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.006s
I20250625 13:59:48.637961 20144 raft_consensus.cc:357] T db73a6fe1f18431982913f428e57588c P 90ff53d19113439ba739ff76bde14294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ff53d19113439ba739ff76bde14294" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 38557 } } peers { permanent_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44777 } } peers { permanent_uuid: "b8dffb1a245d4ba9b1f383d122e00441" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 39411 } }
I20250625 13:59:48.639102 20144 raft_consensus.cc:383] T db73a6fe1f18431982913f428e57588c P 90ff53d19113439ba739ff76bde14294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:59:48.639468 20144 raft_consensus.cc:738] T db73a6fe1f18431982913f428e57588c P 90ff53d19113439ba739ff76bde14294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90ff53d19113439ba739ff76bde14294, State: Initialized, Role: FOLLOWER
I20250625 13:59:48.640188 20143 raft_consensus.cc:357] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ff53d19113439ba739ff76bde14294" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 38557 } } peers { permanent_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44777 } } peers { permanent_uuid: "b8dffb1a245d4ba9b1f383d122e00441" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 39411 } }
I20250625 13:59:48.640519 20144 consensus_queue.cc:260] T db73a6fe1f18431982913f428e57588c P 90ff53d19113439ba739ff76bde14294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ff53d19113439ba739ff76bde14294" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 38557 } } peers { permanent_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44777 } } peers { permanent_uuid: "b8dffb1a245d4ba9b1f383d122e00441" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 39411 } }
I20250625 13:59:48.641103 20143 raft_consensus.cc:383] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:59:48.641417 20143 raft_consensus.cc:738] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8dffb1a245d4ba9b1f383d122e00441, State: Initialized, Role: FOLLOWER
I20250625 13:59:48.642539 20143 consensus_queue.cc:260] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ff53d19113439ba739ff76bde14294" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 38557 } } peers { permanent_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44777 } } peers { permanent_uuid: "b8dffb1a245d4ba9b1f383d122e00441" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 39411 } }
I20250625 13:59:48.643682 20119 heartbeater.cc:499] Master 127.13.155.254:37319 was elected leader, sending a full tablet report...
I20250625 13:59:48.645017 20144 ts_tablet_manager.cc:1428] T db73a6fe1f18431982913f428e57588c P 90ff53d19113439ba739ff76bde14294: Time spent starting tablet: real 0.025s user 0.023s sys 0.000s
I20250625 13:59:48.647848 20143 ts_tablet_manager.cc:1428] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441: Time spent starting tablet: real 0.034s user 0.027s sys 0.009s
W20250625 13:59:48.648671 19987 tablet.cc:2378] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 13:59:48.649039 20145 raft_consensus.cc:357] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ff53d19113439ba739ff76bde14294" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 38557 } } peers { permanent_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44777 } } peers { permanent_uuid: "b8dffb1a245d4ba9b1f383d122e00441" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 39411 } }
I20250625 13:59:48.649953 20145 raft_consensus.cc:383] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:59:48.650329 20145 raft_consensus.cc:738] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43a77c62d30c4a3ab625b25b481d6aa8, State: Initialized, Role: FOLLOWER
I20250625 13:59:48.651203 20145 consensus_queue.cc:260] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ff53d19113439ba739ff76bde14294" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 38557 } } peers { permanent_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44777 } } peers { permanent_uuid: "b8dffb1a245d4ba9b1f383d122e00441" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 39411 } }
I20250625 13:59:48.655508 20145 ts_tablet_manager.cc:1428] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8: Time spent starting tablet: real 0.033s user 0.029s sys 0.007s
W20250625 13:59:48.666746 20120 tablet.cc:2378] T db73a6fe1f18431982913f428e57588c P 90ff53d19113439ba739ff76bde14294: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250625 13:59:48.762698 19854 tablet.cc:2378] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 13:59:48.809564 20150 raft_consensus.cc:491] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:59:48.809975 20150 raft_consensus.cc:513] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ff53d19113439ba739ff76bde14294" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 38557 } } peers { permanent_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44777 } } peers { permanent_uuid: "b8dffb1a245d4ba9b1f383d122e00441" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 39411 } }
I20250625 13:59:48.812319 20150 leader_election.cc:290] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90ff53d19113439ba739ff76bde14294 (127.13.155.195:38557), 43a77c62d30c4a3ab625b25b481d6aa8 (127.13.155.193:44777)
I20250625 13:59:48.824270 19808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db73a6fe1f18431982913f428e57588c" candidate_uuid: "b8dffb1a245d4ba9b1f383d122e00441" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" is_pre_election: true
I20250625 13:59:48.824270 20074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db73a6fe1f18431982913f428e57588c" candidate_uuid: "b8dffb1a245d4ba9b1f383d122e00441" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90ff53d19113439ba739ff76bde14294" is_pre_election: true
I20250625 13:59:48.825162 20074 raft_consensus.cc:2466] T db73a6fe1f18431982913f428e57588c P 90ff53d19113439ba739ff76bde14294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8dffb1a245d4ba9b1f383d122e00441 in term 0.
I20250625 13:59:48.825165 19808 raft_consensus.cc:2466] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8dffb1a245d4ba9b1f383d122e00441 in term 0.
I20250625 13:59:48.826619 19876 leader_election.cc:304] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90ff53d19113439ba739ff76bde14294, b8dffb1a245d4ba9b1f383d122e00441; no voters:
I20250625 13:59:48.827415 20150 raft_consensus.cc:2802] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:59:48.827702 20150 raft_consensus.cc:491] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:59:48.828009 20150 raft_consensus.cc:3058] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:59:48.832880 20150 raft_consensus.cc:513] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ff53d19113439ba739ff76bde14294" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 38557 } } peers { permanent_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44777 } } peers { permanent_uuid: "b8dffb1a245d4ba9b1f383d122e00441" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 39411 } }
I20250625 13:59:48.834198 20150 leader_election.cc:290] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441 [CANDIDATE]: Term 1 election: Requested vote from peers 90ff53d19113439ba739ff76bde14294 (127.13.155.195:38557), 43a77c62d30c4a3ab625b25b481d6aa8 (127.13.155.193:44777)
I20250625 13:59:48.835059 20074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db73a6fe1f18431982913f428e57588c" candidate_uuid: "b8dffb1a245d4ba9b1f383d122e00441" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90ff53d19113439ba739ff76bde14294"
I20250625 13:59:48.835114 19808 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db73a6fe1f18431982913f428e57588c" candidate_uuid: "b8dffb1a245d4ba9b1f383d122e00441" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43a77c62d30c4a3ab625b25b481d6aa8"
I20250625 13:59:48.835480 20074 raft_consensus.cc:3058] T db73a6fe1f18431982913f428e57588c P 90ff53d19113439ba739ff76bde14294 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:59:48.835556 19808 raft_consensus.cc:3058] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:59:48.839795 20074 raft_consensus.cc:2466] T db73a6fe1f18431982913f428e57588c P 90ff53d19113439ba739ff76bde14294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8dffb1a245d4ba9b1f383d122e00441 in term 1.
I20250625 13:59:48.839803 19808 raft_consensus.cc:2466] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8dffb1a245d4ba9b1f383d122e00441 in term 1.
I20250625 13:59:48.840657 19876 leader_election.cc:304] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90ff53d19113439ba739ff76bde14294, b8dffb1a245d4ba9b1f383d122e00441; no voters:
I20250625 13:59:48.841233 20150 raft_consensus.cc:2802] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:59:48.842767 20150 raft_consensus.cc:695] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441 [term 1 LEADER]: Becoming Leader. State: Replica: b8dffb1a245d4ba9b1f383d122e00441, State: Running, Role: LEADER
I20250625 13:59:48.843477 20150 consensus_queue.cc:237] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ff53d19113439ba739ff76bde14294" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 38557 } } peers { permanent_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44777 } } peers { permanent_uuid: "b8dffb1a245d4ba9b1f383d122e00441" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 39411 } }
I20250625 13:59:48.852558 19662 catalog_manager.cc:5582] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441 reported cstate change: term changed from 0 to 1, leader changed from <none> to b8dffb1a245d4ba9b1f383d122e00441 (127.13.155.194). New cstate: current_term: 1 leader_uuid: "b8dffb1a245d4ba9b1f383d122e00441" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ff53d19113439ba739ff76bde14294" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 38557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8dffb1a245d4ba9b1f383d122e00441" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 39411 } health_report { overall_health: HEALTHY } } }
I20250625 13:59:48.998359 19808 raft_consensus.cc:1273] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [term 1 FOLLOWER]: Refusing update from remote peer b8dffb1a245d4ba9b1f383d122e00441: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 13:59:48.998790 20074 raft_consensus.cc:1273] T db73a6fe1f18431982913f428e57588c P 90ff53d19113439ba739ff76bde14294 [term 1 FOLLOWER]: Refusing update from remote peer b8dffb1a245d4ba9b1f383d122e00441: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 13:59:48.999964 20154 consensus_queue.cc:1035] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250625 13:59:49.000802 20150 consensus_queue.cc:1035] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90ff53d19113439ba739ff76bde14294" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 38557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:59:54.088447 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 19631
W20250625 13:59:54.123632 20010 connection.cc:537] client connection to 127.13.155.254:37319 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 13:59:54.124176 19744 connection.cc:537] client connection to 127.13.155.254:37319 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 13:59:54.124663 20119 heartbeater.cc:646] Failed to heartbeat to 127.13.155.254:37319 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 13:59:54.125164 19853 heartbeater.cc:646] Failed to heartbeat to 127.13.155.254:37319 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 13:59:54.145023 19986 heartbeater.cc:646] Failed to heartbeat to 127.13.155.254:37319 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.155.254:37319: connect: Connection refused (error 111)
W20250625 13:59:54.510643 20191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:59:54.511447 20191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:59:54.544782 20191 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250625 13:59:55.993716 20197 debug-util.cc:398] Leaking SignalData structure 0x7b0800036040 after lost signal to thread 20191
W20250625 13:59:55.994429 20197 kernel_stack_watchdog.cc:198] Thread 20191 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:641 for 402ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250625 13:59:56.029186 20191 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.440s user 0.512s sys 0.923s
W20250625 13:59:56.145862 20191 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.557s user 0.513s sys 0.937s
I20250625 13:59:56.232965 20191 minidump.cc:252] Setting minidump size limit to 20M
I20250625 13:59:56.235482 20191 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:59:56.236573 20191 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:59:56.246865 20225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:59:56.247684 20226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:59:56.251955 20228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:59:56.255869 20191 server_base.cc:1048] running on GCE node
I20250625 13:59:56.257185 20191 hybrid_clock.cc:584] initializing the hybrid clock with 'system' time source
I20250625 13:59:56.257757 20191 hybrid_clock.cc:648] HybridClock initialized: now 1750859996257730 us; error 265516 us; skew 500 ppm
I20250625 13:59:56.258742 20191 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:59:56.264530 20191 webserver.cc:469] Webserver started at http://0.0.0.0:36575/ using document root <none> and password file <none>
I20250625 13:59:56.265645 20191 fs_manager.cc:362] Metadata directory not provided
I20250625 13:59:56.265918 20191 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:59:56.266486 20191 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:59:56.274011 20191 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/master-0/data/instance:
uuid: "441f83e12ad344b6b3d280cc6428f78c"
format_stamp: "Formatted at 2025-06-25 13:59:56 on dist-test-slave-k028"
I20250625 13:59:56.275497 20191 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/master-0/wal/instance:
uuid: "441f83e12ad344b6b3d280cc6428f78c"
format_stamp: "Formatted at 2025-06-25 13:59:56 on dist-test-slave-k028"
I20250625 13:59:56.283440 20191 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20250625 13:59:56.289482 20233 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:59:56.290498 20191 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250625 13:59:56.290867 20191 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/master-0/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/master-0/wal
uuid: "441f83e12ad344b6b3d280cc6428f78c"
format_stamp: "Formatted at 2025-06-25 13:59:56 on dist-test-slave-k028"
I20250625 13:59:56.291285 20191 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:59:56.549799 20191 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:59:56.551177 20191 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:59:56.551589 20191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:59:56.556036 20191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:59:56.569936 20191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c: Bootstrap starting.
I20250625 13:59:56.574743 20191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c: Neither blocks nor log segments found. Creating new log.
I20250625 13:59:56.576334 20191 log.cc:826] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c: Log is configured to *not* fsync() on all Append() calls
I20250625 13:59:56.580332 20191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c: No bootstrap required, opened a new log
I20250625 13:59:56.596495 20191 raft_consensus.cc:357] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441f83e12ad344b6b3d280cc6428f78c" member_type: VOTER }
I20250625 13:59:56.596997 20191 raft_consensus.cc:383] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:59:56.597240 20191 raft_consensus.cc:738] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 441f83e12ad344b6b3d280cc6428f78c, State: Initialized, Role: FOLLOWER
I20250625 13:59:56.597961 20191 consensus_queue.cc:260] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441f83e12ad344b6b3d280cc6428f78c" member_type: VOTER }
I20250625 13:59:56.598430 20191 raft_consensus.cc:397] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 13:59:56.598727 20191 raft_consensus.cc:491] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 13:59:56.598999 20191 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:59:56.602926 20191 raft_consensus.cc:513] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441f83e12ad344b6b3d280cc6428f78c" member_type: VOTER }
I20250625 13:59:56.603552 20191 leader_election.cc:304] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 441f83e12ad344b6b3d280cc6428f78c; no voters:
I20250625 13:59:56.605194 20191 leader_election.cc:290] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 13:59:56.605437 20240 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:59:56.607698 20240 raft_consensus.cc:695] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c [term 1 LEADER]: Becoming Leader. State: Replica: 441f83e12ad344b6b3d280cc6428f78c, State: Running, Role: LEADER
I20250625 13:59:56.608521 20240 consensus_queue.cc:237] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441f83e12ad344b6b3d280cc6428f78c" member_type: VOTER }
I20250625 13:59:56.616394 20241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "441f83e12ad344b6b3d280cc6428f78c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441f83e12ad344b6b3d280cc6428f78c" member_type: VOTER } }
I20250625 13:59:56.616344 20242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 441f83e12ad344b6b3d280cc6428f78c. Latest consensus state: current_term: 1 leader_uuid: "441f83e12ad344b6b3d280cc6428f78c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441f83e12ad344b6b3d280cc6428f78c" member_type: VOTER } }
I20250625 13:59:56.616902 20241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c [sys.catalog]: This master's current role is: LEADER
I20250625 13:59:56.616959 20242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c [sys.catalog]: This master's current role is: LEADER
I20250625 13:59:56.628564 20191 tablet_replica.cc:331] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c: stopping tablet replica
I20250625 13:59:56.629163 20191 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:59:56.629567 20191 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:59:56.631469 20191 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250625 13:59:56.631878 20191 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250625 13:59:56.686395 20191 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
I20250625 13:59:57.716246 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 19723
I20250625 13:59:57.755191 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 19857
I20250625 13:59:57.798887 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 19990
I20250625 13:59:57.835114 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.155.254:37319
--webserver_interface=127.13.155.254
--webserver_port=44423
--builtin_ntp_servers=127.13.155.212:40803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.155.254:37319 with env {}
W20250625 13:59:58.139097 20252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:59:58.139684 20252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:59:58.140183 20252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:59:58.170957 20252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 13:59:58.171270 20252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:59:58.171531 20252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 13:59:58.171778 20252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 13:59:58.206521 20252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:40803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.155.254:37319
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.155.254:37319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.155.254
--webserver_port=44423
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:59:58.207823 20252 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:59:58.209483 20252 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:59:58.223619 20261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:59:58.224016 20259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:59:58.224378 20258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:59:59.402350 20260 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250625 13:59:59.402513 20252 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 13:59:59.405931 20252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:59:59.408522 20252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:59:59.409871 20252 hybrid_clock.cc:648] HybridClock initialized: now 1750859999409821 us; error 58 us; skew 500 ppm
I20250625 13:59:59.410715 20252 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:59:59.417347 20252 webserver.cc:469] Webserver started at http://127.13.155.254:44423/ using document root <none> and password file <none>
I20250625 13:59:59.418267 20252 fs_manager.cc:362] Metadata directory not provided
I20250625 13:59:59.418526 20252 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:59:59.426159 20252 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20250625 13:59:59.430529 20268 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:59:59.431519 20252 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250625 13:59:59.431830 20252 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/master-0/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/master-0/wal
uuid: "441f83e12ad344b6b3d280cc6428f78c"
format_stamp: "Formatted at 2025-06-25 13:59:56 on dist-test-slave-k028"
I20250625 13:59:59.433722 20252 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:59:59.492513 20252 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:59:59.494246 20252 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:59:59.494689 20252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:59:59.563313 20252 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.254:37319
I20250625 13:59:59.563371 20319 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.254:37319 every 8 connection(s)
I20250625 13:59:59.566077 20252 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/master-0/data/info.pb
I20250625 13:59:59.570497 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 20252
I20250625 13:59:59.572563 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.155.193:44777
--local_ip_for_outbound_sockets=127.13.155.193
--tserver_master_addrs=127.13.155.254:37319
--webserver_port=38793
--webserver_interface=127.13.155.193
--builtin_ntp_servers=127.13.155.212:40803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250625 13:59:59.579980 20320 sys_catalog.cc:263] Verifying existing consensus state
I20250625 13:59:59.593472 20320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c: Bootstrap starting.
I20250625 13:59:59.603507 20320 log.cc:826] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c: Log is configured to *not* fsync() on all Append() calls
I20250625 13:59:59.615387 20320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 13:59:59.616161 20320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c: Bootstrap complete.
I20250625 13:59:59.636638 20320 raft_consensus.cc:357] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441f83e12ad344b6b3d280cc6428f78c" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 37319 } }
I20250625 13:59:59.637307 20320 raft_consensus.cc:738] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 441f83e12ad344b6b3d280cc6428f78c, State: Initialized, Role: FOLLOWER
I20250625 13:59:59.638056 20320 consensus_queue.cc:260] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441f83e12ad344b6b3d280cc6428f78c" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 37319 } }
I20250625 13:59:59.638592 20320 raft_consensus.cc:397] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 13:59:59.638855 20320 raft_consensus.cc:491] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 13:59:59.639147 20320 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c [term 1 FOLLOWER]: Advancing to term 2
I20250625 13:59:59.644735 20320 raft_consensus.cc:513] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441f83e12ad344b6b3d280cc6428f78c" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 37319 } }
I20250625 13:59:59.645418 20320 leader_election.cc:304] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 441f83e12ad344b6b3d280cc6428f78c; no voters:
I20250625 13:59:59.647665 20320 leader_election.cc:290] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c [CANDIDATE]: Term 2 election: Requested vote from peers
I20250625 13:59:59.647996 20324 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c [term 2 FOLLOWER]: Leader election won for term 2
I20250625 13:59:59.653158 20324 raft_consensus.cc:695] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c [term 2 LEADER]: Becoming Leader. State: Replica: 441f83e12ad344b6b3d280cc6428f78c, State: Running, Role: LEADER
I20250625 13:59:59.653388 20320 sys_catalog.cc:564] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c [sys.catalog]: configured and running, proceeding with master startup.
I20250625 13:59:59.654109 20324 consensus_queue.cc:237] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441f83e12ad344b6b3d280cc6428f78c" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 37319 } }
I20250625 13:59:59.665652 20325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "441f83e12ad344b6b3d280cc6428f78c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441f83e12ad344b6b3d280cc6428f78c" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 37319 } } }
I20250625 13:59:59.666700 20325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c [sys.catalog]: This master's current role is: LEADER
I20250625 13:59:59.668304 20326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 441f83e12ad344b6b3d280cc6428f78c. Latest consensus state: current_term: 2 leader_uuid: "441f83e12ad344b6b3d280cc6428f78c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441f83e12ad344b6b3d280cc6428f78c" member_type: VOTER last_known_addr { host: "127.13.155.254" port: 37319 } } }
I20250625 13:59:59.668990 20326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c [sys.catalog]: This master's current role is: LEADER
I20250625 13:59:59.678581 20332 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 13:59:59.691392 20332 catalog_manager.cc:671] Loaded metadata for table pre_rebuild [id=fe8f0595f51a4e29a8616056a06540f1]
I20250625 13:59:59.699820 20332 tablet_loader.cc:96] loaded metadata for tablet db73a6fe1f18431982913f428e57588c (table pre_rebuild [id=fe8f0595f51a4e29a8616056a06540f1])
I20250625 13:59:59.701447 20332 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 13:59:59.733150 20332 catalog_manager.cc:1349] Generated new cluster ID: 5d7762e617d04237ac66a77b5450a2ab
I20250625 13:59:59.733456 20332 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 13:59:59.779656 20332 catalog_manager.cc:1372] Generated new certificate authority record
I20250625 13:59:59.781729 20332 catalog_manager.cc:1506] Loading token signing keys...
I20250625 13:59:59.804386 20332 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c: Generated new TSK 0
I20250625 13:59:59.805564 20332 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250625 13:59:59.938438 20322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:59:59.938988 20322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:59:59.939488 20322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:59:59.970613 20322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:59:59.971467 20322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.155.193
I20250625 14:00:00.007102 20322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:40803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.155.193:44777
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.155.193
--webserver_port=38793
--tserver_master_addrs=127.13.155.254:37319
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.155.193
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 14:00:00.008451 20322 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 14:00:00.010090 20322 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 14:00:00.028431 20349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 14:00:00.029677 20351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 14:00:00.030221 20322 server_base.cc:1048] running on GCE node
W20250625 14:00:00.029161 20348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 14:00:01.209630 20322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 14:00:01.212397 20322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 14:00:01.213838 20322 hybrid_clock.cc:648] HybridClock initialized: now 1750860001213782 us; error 74 us; skew 500 ppm
I20250625 14:00:01.214722 20322 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 14:00:01.225611 20322 webserver.cc:469] Webserver started at http://127.13.155.193:38793/ using document root <none> and password file <none>
I20250625 14:00:01.226624 20322 fs_manager.cc:362] Metadata directory not provided
I20250625 14:00:01.226861 20322 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 14:00:01.235869 20322 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.008s sys 0.000s
I20250625 14:00:01.240582 20358 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 14:00:01.241883 20322 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20250625 14:00:01.242233 20322 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-0/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-0/wal
uuid: "43a77c62d30c4a3ab625b25b481d6aa8"
format_stamp: "Formatted at 2025-06-25 13:59:44 on dist-test-slave-k028"
I20250625 14:00:01.244210 20322 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 14:00:01.323402 20322 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 14:00:01.324877 20322 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 14:00:01.325342 20322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 14:00:01.328461 20322 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 14:00:01.334810 20365 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250625 14:00:01.342686 20322 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250625 14:00:01.342924 20322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.002s
I20250625 14:00:01.343232 20322 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250625 14:00:01.347939 20322 ts_tablet_manager.cc:610] Registered 1 tablets
I20250625 14:00:01.348198 20322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.003s sys 0.001s
I20250625 14:00:01.348610 20365 tablet_bootstrap.cc:492] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8: Bootstrap starting.
I20250625 14:00:01.537662 20322 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.193:44777
I20250625 14:00:01.537827 20471 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.193:44777 every 8 connection(s)
I20250625 14:00:01.540534 20322 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-0/data/info.pb
I20250625 14:00:01.548787 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 20322
I20250625 14:00:01.553244 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.155.194:39411
--local_ip_for_outbound_sockets=127.13.155.194
--tserver_master_addrs=127.13.155.254:37319
--webserver_port=35289
--webserver_interface=127.13.155.194
--builtin_ntp_servers=127.13.155.212:40803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250625 14:00:01.589563 20472 heartbeater.cc:344] Connected to a master server at 127.13.155.254:37319
I20250625 14:00:01.590148 20472 heartbeater.cc:461] Registering TS with master...
I20250625 14:00:01.591604 20472 heartbeater.cc:507] Master 127.13.155.254:37319 requested a full tablet report, sending...
I20250625 14:00:01.596728 20285 ts_manager.cc:194] Registered new tserver with Master: 43a77c62d30c4a3ab625b25b481d6aa8 (127.13.155.193:44777)
I20250625 14:00:01.604151 20285 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.155.193:36373
I20250625 14:00:01.646708 20365 log.cc:826] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8: Log is configured to *not* fsync() on all Append() calls
W20250625 14:00:01.881863 20476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 14:00:01.882375 20476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 14:00:01.882925 20476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 14:00:01.914860 20476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 14:00:01.915699 20476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.155.194
I20250625 14:00:01.950385 20476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:40803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.155.194:39411
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.155.194
--webserver_port=35289
--tserver_master_addrs=127.13.155.254:37319
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.155.194
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 14:00:01.951699 20476 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 14:00:01.953401 20476 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 14:00:01.970835 20484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 14:00:02.608522 20472 heartbeater.cc:499] Master 127.13.155.254:37319 was elected leader, sending a full tablet report...
W20250625 14:00:01.971052 20483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 14:00:01.972599 20486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 14:00:03.166169 20485 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1196 milliseconds
I20250625 14:00:03.166304 20476 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 14:00:03.170715 20476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 14:00:03.173419 20476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 14:00:03.174885 20476 hybrid_clock.cc:648] HybridClock initialized: now 1750860003174837 us; error 50 us; skew 500 ppm
I20250625 14:00:03.175966 20476 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 14:00:03.183741 20476 webserver.cc:469] Webserver started at http://127.13.155.194:35289/ using document root <none> and password file <none>
I20250625 14:00:03.184688 20476 fs_manager.cc:362] Metadata directory not provided
I20250625 14:00:03.184921 20476 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 14:00:03.193104 20476 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20250625 14:00:03.198011 20494 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 14:00:03.199232 20476 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250625 14:00:03.199611 20476 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-1/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-1/wal
uuid: "b8dffb1a245d4ba9b1f383d122e00441"
format_stamp: "Formatted at 2025-06-25 13:59:46 on dist-test-slave-k028"
I20250625 14:00:03.202371 20476 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 14:00:03.276573 20476 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 14:00:03.278595 20476 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 14:00:03.279176 20476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 14:00:03.282408 20476 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 14:00:03.289785 20501 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250625 14:00:03.297399 20476 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250625 14:00:03.297719 20476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.002s sys 0.001s
I20250625 14:00:03.298045 20476 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250625 14:00:03.305162 20476 ts_tablet_manager.cc:610] Registered 1 tablets
I20250625 14:00:03.305423 20476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.007s user 0.005s sys 0.000s
I20250625 14:00:03.305759 20501 tablet_bootstrap.cc:492] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441: Bootstrap starting.
I20250625 14:00:03.506880 20476 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.194:39411
I20250625 14:00:03.507009 20607 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.194:39411 every 8 connection(s)
I20250625 14:00:03.509641 20476 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-1/data/info.pb
I20250625 14:00:03.515843 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 20476
I20250625 14:00:03.520028 13935 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2vErUD/build/tsan/bin/kudu
/tmp/dist-test-task2vErUD/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.155.195:38557
--local_ip_for_outbound_sockets=127.13.155.195
--tserver_master_addrs=127.13.155.254:37319
--webserver_port=38691
--webserver_interface=127.13.155.195
--builtin_ntp_servers=127.13.155.212:40803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250625 14:00:03.601562 20608 heartbeater.cc:344] Connected to a master server at 127.13.155.254:37319
I20250625 14:00:03.602118 20608 heartbeater.cc:461] Registering TS with master...
I20250625 14:00:03.603441 20608 heartbeater.cc:507] Master 127.13.155.254:37319 requested a full tablet report, sending...
I20250625 14:00:03.608448 20285 ts_manager.cc:194] Registered new tserver with Master: b8dffb1a245d4ba9b1f383d122e00441 (127.13.155.194:39411)
I20250625 14:00:03.611925 20285 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.155.194:51087
I20250625 14:00:03.645787 20501 log.cc:826] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441: Log is configured to *not* fsync() on all Append() calls
I20250625 14:00:03.800855 20365 tablet_bootstrap.cc:492] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 14:00:03.801904 20365 tablet_bootstrap.cc:492] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8: Bootstrap complete.
I20250625 14:00:03.803694 20365 ts_tablet_manager.cc:1397] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8: Time spent bootstrapping tablet: real 2.456s user 2.386s sys 0.052s
I20250625 14:00:03.820592 20365 raft_consensus.cc:357] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ff53d19113439ba739ff76bde14294" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 38557 } } peers { permanent_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44777 } } peers { permanent_uuid: "b8dffb1a245d4ba9b1f383d122e00441" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 39411 } }
I20250625 14:00:03.823700 20365 raft_consensus.cc:738] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43a77c62d30c4a3ab625b25b481d6aa8, State: Initialized, Role: FOLLOWER
I20250625 14:00:03.824595 20365 consensus_queue.cc:260] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ff53d19113439ba739ff76bde14294" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 38557 } } peers { permanent_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44777 } } peers { permanent_uuid: "b8dffb1a245d4ba9b1f383d122e00441" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 39411 } }
I20250625 14:00:03.828080 20365 ts_tablet_manager.cc:1428] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8: Time spent starting tablet: real 0.024s user 0.024s sys 0.000s
W20250625 14:00:03.919046 20612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 14:00:03.919569 20612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 14:00:03.920087 20612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 14:00:03.951781 20612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 14:00:03.952721 20612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.155.195
I20250625 14:00:03.988161 20612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.155.212:40803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.155.195:38557
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.155.195
--webserver_port=38691
--tserver_master_addrs=127.13.155.254:37319
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.155.195
--log_dir=/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 14:00:03.989439 20612 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 14:00:03.991084 20612 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 14:00:04.008512 20621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 14:00:04.622231 20608 heartbeater.cc:499] Master 127.13.155.254:37319 was elected leader, sending a full tablet report...
I20250625 14:00:05.039981 20626 raft_consensus.cc:491] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 14:00:05.040475 20626 raft_consensus.cc:513] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ff53d19113439ba739ff76bde14294" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 38557 } } peers { permanent_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44777 } } peers { permanent_uuid: "b8dffb1a245d4ba9b1f383d122e00441" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 39411 } }
I20250625 14:00:05.043596 20626 leader_election.cc:290] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 90ff53d19113439ba739ff76bde14294 (127.13.155.195:38557), b8dffb1a245d4ba9b1f383d122e00441 (127.13.155.194:39411)
W20250625 14:00:05.068594 20361 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.155.195:38557: connect: Connection refused (error 111)
W20250625 14:00:05.077997 20361 leader_election.cc:336] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 90ff53d19113439ba739ff76bde14294 (127.13.155.195:38557): Network error: Client connection negotiation failed: client connection to 127.13.155.195:38557: connect: Connection refused (error 111)
I20250625 14:00:05.077571 20563 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db73a6fe1f18431982913f428e57588c" candidate_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "b8dffb1a245d4ba9b1f383d122e00441" is_pre_election: true
W20250625 14:00:05.090622 20362 leader_election.cc:343] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b8dffb1a245d4ba9b1f383d122e00441 (127.13.155.194:39411): Illegal state: must be running to vote when last-logged opid is not known
I20250625 14:00:05.091123 20362 leader_election.cc:304] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43a77c62d30c4a3ab625b25b481d6aa8; no voters: 90ff53d19113439ba739ff76bde14294, b8dffb1a245d4ba9b1f383d122e00441
I20250625 14:00:05.091996 20626 raft_consensus.cc:2747] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 14:00:04.009644 20620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 14:00:04.008818 20623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 14:00:05.217975 20622 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250625 14:00:05.218508 20612 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 14:00:05.222931 20612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 14:00:05.231050 20612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 14:00:05.232471 20612 hybrid_clock.cc:648] HybridClock initialized: now 1750860005232400 us; error 74 us; skew 500 ppm
I20250625 14:00:05.233357 20612 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 14:00:05.240567 20612 webserver.cc:469] Webserver started at http://127.13.155.195:38691/ using document root <none> and password file <none>
I20250625 14:00:05.241477 20612 fs_manager.cc:362] Metadata directory not provided
I20250625 14:00:05.241715 20612 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 14:00:05.249927 20612 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20250625 14:00:05.254943 20635 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 14:00:05.256000 20612 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20250625 14:00:05.256314 20612 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-2/data,/tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-2/wal
uuid: "90ff53d19113439ba739ff76bde14294"
format_stamp: "Formatted at 2025-06-25 13:59:48 on dist-test-slave-k028"
I20250625 14:00:05.258184 20612 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 14:00:05.325584 20612 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 14:00:05.327101 20612 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 14:00:05.327534 20612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 14:00:05.330545 20612 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 14:00:05.336426 20642 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250625 14:00:05.347208 20612 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250625 14:00:05.347514 20612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.002s sys 0.000s
I20250625 14:00:05.347831 20612 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250625 14:00:05.355018 20612 ts_tablet_manager.cc:610] Registered 1 tablets
I20250625 14:00:05.355350 20612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.008s user 0.008s sys 0.000s
I20250625 14:00:05.355697 20642 tablet_bootstrap.cc:492] T db73a6fe1f18431982913f428e57588c P 90ff53d19113439ba739ff76bde14294: Bootstrap starting.
I20250625 14:00:05.617071 20612 rpc_server.cc:307] RPC server started. Bound to: 127.13.155.195:38557
I20250625 14:00:05.617213 20748 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.155.195:38557 every 8 connection(s)
I20250625 14:00:05.620867 20612 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-2/data/info.pb
I20250625 14:00:05.631982 13935 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2vErUD/build/tsan/bin/kudu as pid 20612
I20250625 14:00:05.676509 20642 log.cc:826] T db73a6fe1f18431982913f428e57588c P 90ff53d19113439ba739ff76bde14294: Log is configured to *not* fsync() on all Append() calls
I20250625 14:00:05.677398 20749 heartbeater.cc:344] Connected to a master server at 127.13.155.254:37319
I20250625 14:00:05.677896 20749 heartbeater.cc:461] Registering TS with master...
I20250625 14:00:05.679126 20749 heartbeater.cc:507] Master 127.13.155.254:37319 requested a full tablet report, sending...
I20250625 14:00:05.682994 20285 ts_manager.cc:194] Registered new tserver with Master: 90ff53d19113439ba739ff76bde14294 (127.13.155.195:38557)
I20250625 14:00:05.685951 20285 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.155.195:42105
I20250625 14:00:05.691205 13935 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 14:00:05.935765 20501 tablet_bootstrap.cc:492] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 14:00:05.936534 20501 tablet_bootstrap.cc:492] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441: Bootstrap complete.
I20250625 14:00:05.937834 20501 ts_tablet_manager.cc:1397] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441: Time spent bootstrapping tablet: real 2.632s user 2.498s sys 0.079s
I20250625 14:00:05.942906 20501 raft_consensus.cc:357] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ff53d19113439ba739ff76bde14294" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 38557 } } peers { permanent_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44777 } } peers { permanent_uuid: "b8dffb1a245d4ba9b1f383d122e00441" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 39411 } }
I20250625 14:00:05.944818 20501 raft_consensus.cc:738] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8dffb1a245d4ba9b1f383d122e00441, State: Initialized, Role: FOLLOWER
I20250625 14:00:05.945502 20501 consensus_queue.cc:260] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ff53d19113439ba739ff76bde14294" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 38557 } } peers { permanent_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44777 } } peers { permanent_uuid: "b8dffb1a245d4ba9b1f383d122e00441" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 39411 } }
I20250625 14:00:05.948510 20501 ts_tablet_manager.cc:1428] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441: Time spent starting tablet: real 0.010s user 0.011s sys 0.002s
I20250625 14:00:06.691017 20749 heartbeater.cc:499] Master 127.13.155.254:37319 was elected leader, sending a full tablet report...
I20250625 14:00:06.918908 20762 raft_consensus.cc:491] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 14:00:06.919286 20762 raft_consensus.cc:513] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ff53d19113439ba739ff76bde14294" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 38557 } } peers { permanent_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44777 } } peers { permanent_uuid: "b8dffb1a245d4ba9b1f383d122e00441" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 39411 } }
I20250625 14:00:06.921041 20762 leader_election.cc:290] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 90ff53d19113439ba739ff76bde14294 (127.13.155.195:38557), b8dffb1a245d4ba9b1f383d122e00441 (127.13.155.194:39411)
I20250625 14:00:06.921944 20563 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db73a6fe1f18431982913f428e57588c" candidate_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "b8dffb1a245d4ba9b1f383d122e00441" is_pre_election: true
I20250625 14:00:06.922673 20563 raft_consensus.cc:2466] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43a77c62d30c4a3ab625b25b481d6aa8 in term 1.
I20250625 14:00:06.923977 20362 leader_election.cc:304] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43a77c62d30c4a3ab625b25b481d6aa8, b8dffb1a245d4ba9b1f383d122e00441; no voters:
I20250625 14:00:06.924731 20762 raft_consensus.cc:2802] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250625 14:00:06.925046 20762 raft_consensus.cc:491] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 14:00:06.925369 20762 raft_consensus.cc:3058] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [term 1 FOLLOWER]: Advancing to term 2
I20250625 14:00:06.934223 20762 raft_consensus.cc:513] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ff53d19113439ba739ff76bde14294" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 38557 } } peers { permanent_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44777 } } peers { permanent_uuid: "b8dffb1a245d4ba9b1f383d122e00441" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 39411 } }
I20250625 14:00:06.937001 20563 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db73a6fe1f18431982913f428e57588c" candidate_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "b8dffb1a245d4ba9b1f383d122e00441"
I20250625 14:00:06.937530 20563 raft_consensus.cc:3058] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441 [term 1 FOLLOWER]: Advancing to term 2
I20250625 14:00:06.937968 20762 leader_election.cc:290] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [CANDIDATE]: Term 2 election: Requested vote from peers 90ff53d19113439ba739ff76bde14294 (127.13.155.195:38557), b8dffb1a245d4ba9b1f383d122e00441 (127.13.155.194:39411)
I20250625 14:00:06.933511 20704 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db73a6fe1f18431982913f428e57588c" candidate_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "90ff53d19113439ba739ff76bde14294" is_pre_election: true
I20250625 14:00:06.937554 20703 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db73a6fe1f18431982913f428e57588c" candidate_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "90ff53d19113439ba739ff76bde14294"
W20250625 14:00:06.943435 20361 leader_election.cc:343] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 90ff53d19113439ba739ff76bde14294 (127.13.155.195:38557): Illegal state: must be running to vote when last-logged opid is not known
W20250625 14:00:06.944129 20361 leader_election.cc:343] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 90ff53d19113439ba739ff76bde14294 (127.13.155.195:38557): Illegal state: must be running to vote when last-logged opid is not known
I20250625 14:00:06.946393 20563 raft_consensus.cc:2466] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43a77c62d30c4a3ab625b25b481d6aa8 in term 2.
I20250625 14:00:06.947428 20362 leader_election.cc:304] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 43a77c62d30c4a3ab625b25b481d6aa8, b8dffb1a245d4ba9b1f383d122e00441; no voters: 90ff53d19113439ba739ff76bde14294
I20250625 14:00:06.948139 20762 raft_consensus.cc:2802] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [term 2 FOLLOWER]: Leader election won for term 2
I20250625 14:00:06.949476 20762 raft_consensus.cc:695] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [term 2 LEADER]: Becoming Leader. State: Replica: 43a77c62d30c4a3ab625b25b481d6aa8, State: Running, Role: LEADER
I20250625 14:00:06.950608 20762 consensus_queue.cc:237] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ff53d19113439ba739ff76bde14294" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 38557 } } peers { permanent_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44777 } } peers { permanent_uuid: "b8dffb1a245d4ba9b1f383d122e00441" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 39411 } }
I20250625 14:00:06.962505 20284 catalog_manager.cc:5582] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 reported cstate change: term changed from 0 to 2, leader changed from <none> to 43a77c62d30c4a3ab625b25b481d6aa8 (127.13.155.193), VOTER 43a77c62d30c4a3ab625b25b481d6aa8 (127.13.155.193) added, VOTER 90ff53d19113439ba739ff76bde14294 (127.13.155.195) added, VOTER b8dffb1a245d4ba9b1f383d122e00441 (127.13.155.194) added. New cstate: current_term: 2 leader_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ff53d19113439ba739ff76bde14294" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 38557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44777 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8dffb1a245d4ba9b1f383d122e00441" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 39411 } health_report { overall_health: UNKNOWN } } }
I20250625 14:00:07.379356 20563 raft_consensus.cc:1273] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441 [term 2 FOLLOWER]: Refusing update from remote peer 43a77c62d30c4a3ab625b25b481d6aa8: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250625 14:00:07.381228 20762 consensus_queue.cc:1035] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8dffb1a245d4ba9b1f383d122e00441" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 39411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
W20250625 14:00:07.439716 20361 consensus_peers.cc:487] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 -> Peer 90ff53d19113439ba739ff76bde14294 (127.13.155.195:38557): Couldn't send request to peer 90ff53d19113439ba739ff76bde14294. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250625 14:00:07.479280 20427 consensus_queue.cc:237] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 2.206, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44777 } } peers { permanent_uuid: "b8dffb1a245d4ba9b1f383d122e00441" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 39411 } }
I20250625 14:00:07.485028 20562 raft_consensus.cc:1273] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441 [term 2 FOLLOWER]: Refusing update from remote peer 43a77c62d30c4a3ab625b25b481d6aa8: Log matching property violated. Preceding OpId in replica: term: 2 index: 206. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
I20250625 14:00:07.486682 20765 consensus_queue.cc:1035] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8dffb1a245d4ba9b1f383d122e00441" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 39411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.001s
I20250625 14:00:07.493731 20765 raft_consensus.cc:2953] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [term 2 LEADER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 90ff53d19113439ba739ff76bde14294 (127.13.155.195) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44777 } } peers { permanent_uuid: "b8dffb1a245d4ba9b1f383d122e00441" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 39411 } } }
I20250625 14:00:07.496821 20563 raft_consensus.cc:2953] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441 [term 2 FOLLOWER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 90ff53d19113439ba739ff76bde14294 (127.13.155.195) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44777 } } peers { permanent_uuid: "b8dffb1a245d4ba9b1f383d122e00441" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 39411 } } }
I20250625 14:00:07.527310 20284 catalog_manager.cc:5582] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441 reported cstate change: config changed from index -1 to 207, VOTER 90ff53d19113439ba739ff76bde14294 (127.13.155.195) evicted. New cstate: current_term: 2 leader_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44777 } } peers { permanent_uuid: "b8dffb1a245d4ba9b1f383d122e00441" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 39411 } } }
W20250625 14:00:07.537396 13935 scanner-internal.cc:458] Time spent opening tablet: real 1.810s user 0.005s sys 0.001s
I20250625 14:00:07.553556 20269 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet db73a6fe1f18431982913f428e57588c with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250625 14:00:07.597146 20427 consensus_queue.cc:237] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 2.207, Last appended by leader: 205, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44777 } }
I20250625 14:00:07.603042 20765 raft_consensus.cc:2953] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [term 2 LEADER]: Committing config change with OpId 2.208: config changed from index 207 to 208, VOTER b8dffb1a245d4ba9b1f383d122e00441 (127.13.155.194) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44777 } } }
I20250625 14:00:07.618996 20269 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet db73a6fe1f18431982913f428e57588c with cas_config_opid_index 207: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250625 14:00:07.627209 20284 catalog_manager.cc:5582] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 reported cstate change: config changed from index 207 to 208, VOTER b8dffb1a245d4ba9b1f383d122e00441 (127.13.155.194) evicted. New cstate: current_term: 2 leader_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44777 } health_report { overall_health: HEALTHY } } }
I20250625 14:00:07.688620 20684 tablet_service.cc:1515] Processing DeleteTablet for tablet db73a6fe1f18431982913f428e57588c with delete_type TABLET_DATA_TOMBSTONED (TS 90ff53d19113439ba739ff76bde14294 not found in new config with opid_index 207) from {username='slave'} at 127.0.0.1:57404
I20250625 14:00:07.698045 20543 tablet_service.cc:1515] Processing DeleteTablet for tablet db73a6fe1f18431982913f428e57588c with delete_type TABLET_DATA_TOMBSTONED (TS b8dffb1a245d4ba9b1f383d122e00441 not found in new config with opid_index 208) from {username='slave'} at 127.0.0.1:36110
W20250625 14:00:07.704499 20271 catalog_manager.cc:4908] TS 90ff53d19113439ba739ff76bde14294 (127.13.155.195:38557): delete failed for tablet db73a6fe1f18431982913f428e57588c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db73a6fe1f18431982913f428e57588c already in progress: opening tablet
I20250625 14:00:07.727769 20787 tablet_replica.cc:331] stopping tablet replica
I20250625 14:00:07.728796 20787 raft_consensus.cc:2241] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250625 14:00:07.729530 20787 raft_consensus.cc:2270] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250625 14:00:07.822337 20642 tablet_bootstrap.cc:492] T db73a6fe1f18431982913f428e57588c P 90ff53d19113439ba739ff76bde14294: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 14:00:07.823508 20642 tablet_bootstrap.cc:492] T db73a6fe1f18431982913f428e57588c P 90ff53d19113439ba739ff76bde14294: Bootstrap complete.
I20250625 14:00:07.825332 20642 ts_tablet_manager.cc:1397] T db73a6fe1f18431982913f428e57588c P 90ff53d19113439ba739ff76bde14294: Time spent bootstrapping tablet: real 2.470s user 2.385s sys 0.060s
I20250625 14:00:07.833168 20642 raft_consensus.cc:357] T db73a6fe1f18431982913f428e57588c P 90ff53d19113439ba739ff76bde14294 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ff53d19113439ba739ff76bde14294" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 38557 } } peers { permanent_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44777 } } peers { permanent_uuid: "b8dffb1a245d4ba9b1f383d122e00441" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 39411 } }
I20250625 14:00:07.836251 20642 raft_consensus.cc:738] T db73a6fe1f18431982913f428e57588c P 90ff53d19113439ba739ff76bde14294 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90ff53d19113439ba739ff76bde14294, State: Initialized, Role: FOLLOWER
I20250625 14:00:07.837103 20642 consensus_queue.cc:260] T db73a6fe1f18431982913f428e57588c P 90ff53d19113439ba739ff76bde14294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ff53d19113439ba739ff76bde14294" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 38557 } } peers { permanent_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44777 } } peers { permanent_uuid: "b8dffb1a245d4ba9b1f383d122e00441" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 39411 } }
I20250625 14:00:07.846540 20642 ts_tablet_manager.cc:1428] T db73a6fe1f18431982913f428e57588c P 90ff53d19113439ba739ff76bde14294: Time spent starting tablet: real 0.021s user 0.012s sys 0.008s
I20250625 14:00:07.851576 20684 tablet_service.cc:1515] Processing DeleteTablet for tablet db73a6fe1f18431982913f428e57588c with delete_type TABLET_DATA_TOMBSTONED (Replica has no consensus available (current committed config index is 208)) from {username='slave'} at 127.0.0.1:57404
I20250625 14:00:07.852583 20786 tablet_replica.cc:331] stopping tablet replica
I20250625 14:00:07.853302 20786 raft_consensus.cc:2241] T db73a6fe1f18431982913f428e57588c P 90ff53d19113439ba739ff76bde14294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 14:00:07.853782 20786 raft_consensus.cc:2270] T db73a6fe1f18431982913f428e57588c P 90ff53d19113439ba739ff76bde14294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 14:00:07.896394 20787 ts_tablet_manager.cc:1905] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250625 14:00:07.911811 20787 ts_tablet_manager.cc:1918] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.207
I20250625 14:00:07.912357 20787 log.cc:1199] T db73a6fe1f18431982913f428e57588c P b8dffb1a245d4ba9b1f383d122e00441: Deleting WAL directory at /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-1/wal/wals/db73a6fe1f18431982913f428e57588c
I20250625 14:00:07.914294 20272 catalog_manager.cc:4928] TS b8dffb1a245d4ba9b1f383d122e00441 (127.13.155.194:39411): tablet db73a6fe1f18431982913f428e57588c (table pre_rebuild [id=fe8f0595f51a4e29a8616056a06540f1]) successfully deleted
I20250625 14:00:07.918030 20786 ts_tablet_manager.cc:1905] T db73a6fe1f18431982913f428e57588c P 90ff53d19113439ba739ff76bde14294: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250625 14:00:07.940549 20786 ts_tablet_manager.cc:1918] T db73a6fe1f18431982913f428e57588c P 90ff53d19113439ba739ff76bde14294: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.205
I20250625 14:00:07.941642 20786 log.cc:1199] T db73a6fe1f18431982913f428e57588c P 90ff53d19113439ba739ff76bde14294: Deleting WAL directory at /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-2/wal/wals/db73a6fe1f18431982913f428e57588c
I20250625 14:00:07.943521 20271 catalog_manager.cc:4928] TS 90ff53d19113439ba739ff76bde14294 (127.13.155.195:38557): tablet db73a6fe1f18431982913f428e57588c (table pre_rebuild [id=fe8f0595f51a4e29a8616056a06540f1]) successfully deleted
I20250625 14:00:08.326985 20684 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 14:00:08.329223 20407 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250625 14:00:08.340122 20543 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
441f83e12ad344b6b3d280cc6428f78c | 127.13.155.254:37319 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.13.155.212:40803 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
43a77c62d30c4a3ab625b25b481d6aa8 | 127.13.155.193:44777 | HEALTHY | <none> | 1 | 0
90ff53d19113439ba739ff76bde14294 | 127.13.155.195:38557 | HEALTHY | <none> | 0 | 0
b8dffb1a245d4ba9b1f383d122e00441 | 127.13.155.194:39411 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.13.155.193 | experimental | 127.13.155.193:44777
local_ip_for_outbound_sockets | 127.13.155.194 | experimental | 127.13.155.194:39411
local_ip_for_outbound_sockets | 127.13.155.195 | experimental | 127.13.155.195:38557
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-0/data/info.pb | hidden | 127.13.155.193:44777
server_dump_info_path | /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-1/data/info.pb | hidden | 127.13.155.194:39411
server_dump_info_path | /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-2/data/info.pb | hidden | 127.13.155.195:38557
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.13.155.212:40803 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------+----+---------+---------------+---------+------------+------------------+-------------
pre_rebuild | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 0
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 1
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250625 14:00:08.653232 13935 log_verifier.cc:126] Checking tablet db73a6fe1f18431982913f428e57588c
I20250625 14:00:08.907318 13935 log_verifier.cc:177] Verified matching terms for 208 ops in tablet db73a6fe1f18431982913f428e57588c
I20250625 14:00:08.909700 20285 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44886:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250625 14:00:08.910226 20285 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44886:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250625 14:00:08.922559 20285 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 441f83e12ad344b6b3d280cc6428f78c: Sending DeleteTablet for 1 replicas of tablet db73a6fe1f18431982913f428e57588c
I20250625 14:00:08.924515 13935 test_util.cc:276] Using random seed: -589159096
I20250625 14:00:08.924376 20407 tablet_service.cc:1515] Processing DeleteTablet for tablet db73a6fe1f18431982913f428e57588c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 14:00:08 UTC) from {username='slave'} at 127.0.0.1:36438
I20250625 14:00:08.926185 20821 tablet_replica.cc:331] stopping tablet replica
I20250625 14:00:08.926965 20821 raft_consensus.cc:2241] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [term 2 LEADER]: Raft consensus shutting down.
I20250625 14:00:08.927621 20821 raft_consensus.cc:2270] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250625 14:00:08.953994 20285 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44920:
name: "post_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 14:00:08.956630 20285 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table post_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 14:00:08.964572 20821 ts_tablet_manager.cc:1905] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 14:00:08.979388 20543 tablet_service.cc:1468] Processing CreateTablet for tablet 71ff3718b9b3401fa392a99b1ea3de77 (DEFAULT_TABLE table=post_rebuild [id=7c63460683714c1c8e74dc019e81cd61]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 14:00:08.979977 20684 tablet_service.cc:1468] Processing CreateTablet for tablet 71ff3718b9b3401fa392a99b1ea3de77 (DEFAULT_TABLE table=post_rebuild [id=7c63460683714c1c8e74dc019e81cd61]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 14:00:08.980743 20543 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71ff3718b9b3401fa392a99b1ea3de77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 14:00:08.981267 20684 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71ff3718b9b3401fa392a99b1ea3de77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 14:00:08.983436 20407 tablet_service.cc:1468] Processing CreateTablet for tablet 71ff3718b9b3401fa392a99b1ea3de77 (DEFAULT_TABLE table=post_rebuild [id=7c63460683714c1c8e74dc019e81cd61]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 14:00:08.984737 20407 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71ff3718b9b3401fa392a99b1ea3de77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 14:00:08.993640 20821 ts_tablet_manager.cc:1918] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.208
I20250625 14:00:08.994132 20821 log.cc:1199] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8: Deleting WAL directory at /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-0/wal/wals/db73a6fe1f18431982913f428e57588c
I20250625 14:00:08.995247 20821 ts_tablet_manager.cc:1939] T db73a6fe1f18431982913f428e57588c P 43a77c62d30c4a3ab625b25b481d6aa8: Deleting consensus metadata
I20250625 14:00:09.003960 20269 catalog_manager.cc:4928] TS 43a77c62d30c4a3ab625b25b481d6aa8 (127.13.155.193:44777): tablet db73a6fe1f18431982913f428e57588c (table pre_rebuild [id=fe8f0595f51a4e29a8616056a06540f1]) successfully deleted
I20250625 14:00:09.004320 20829 tablet_bootstrap.cc:492] T 71ff3718b9b3401fa392a99b1ea3de77 P b8dffb1a245d4ba9b1f383d122e00441: Bootstrap starting.
I20250625 14:00:09.013006 20829 tablet_bootstrap.cc:654] T 71ff3718b9b3401fa392a99b1ea3de77 P b8dffb1a245d4ba9b1f383d122e00441: Neither blocks nor log segments found. Creating new log.
I20250625 14:00:09.014223 20828 tablet_bootstrap.cc:492] T 71ff3718b9b3401fa392a99b1ea3de77 P 90ff53d19113439ba739ff76bde14294: Bootstrap starting.
I20250625 14:00:09.020447 20828 tablet_bootstrap.cc:654] T 71ff3718b9b3401fa392a99b1ea3de77 P 90ff53d19113439ba739ff76bde14294: Neither blocks nor log segments found. Creating new log.
I20250625 14:00:09.023590 20830 tablet_bootstrap.cc:492] T 71ff3718b9b3401fa392a99b1ea3de77 P 43a77c62d30c4a3ab625b25b481d6aa8: Bootstrap starting.
I20250625 14:00:09.026938 20829 tablet_bootstrap.cc:492] T 71ff3718b9b3401fa392a99b1ea3de77 P b8dffb1a245d4ba9b1f383d122e00441: No bootstrap required, opened a new log
I20250625 14:00:09.027392 20828 tablet_bootstrap.cc:492] T 71ff3718b9b3401fa392a99b1ea3de77 P 90ff53d19113439ba739ff76bde14294: No bootstrap required, opened a new log
I20250625 14:00:09.027817 20828 ts_tablet_manager.cc:1397] T 71ff3718b9b3401fa392a99b1ea3de77 P 90ff53d19113439ba739ff76bde14294: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20250625 14:00:09.029873 20829 ts_tablet_manager.cc:1397] T 71ff3718b9b3401fa392a99b1ea3de77 P b8dffb1a245d4ba9b1f383d122e00441: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.004s
I20250625 14:00:09.030408 20830 tablet_bootstrap.cc:654] T 71ff3718b9b3401fa392a99b1ea3de77 P 43a77c62d30c4a3ab625b25b481d6aa8: Neither blocks nor log segments found. Creating new log.
I20250625 14:00:09.030592 20828 raft_consensus.cc:357] T 71ff3718b9b3401fa392a99b1ea3de77 P 90ff53d19113439ba739ff76bde14294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8dffb1a245d4ba9b1f383d122e00441" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 39411 } } peers { permanent_uuid: "90ff53d19113439ba739ff76bde14294" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 38557 } } peers { permanent_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44777 } }
I20250625 14:00:09.031381 20828 raft_consensus.cc:383] T 71ff3718b9b3401fa392a99b1ea3de77 P 90ff53d19113439ba739ff76bde14294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 14:00:09.031708 20828 raft_consensus.cc:738] T 71ff3718b9b3401fa392a99b1ea3de77 P 90ff53d19113439ba739ff76bde14294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90ff53d19113439ba739ff76bde14294, State: Initialized, Role: FOLLOWER
I20250625 14:00:09.032613 20829 raft_consensus.cc:357] T 71ff3718b9b3401fa392a99b1ea3de77 P b8dffb1a245d4ba9b1f383d122e00441 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8dffb1a245d4ba9b1f383d122e00441" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 39411 } } peers { permanent_uuid: "90ff53d19113439ba739ff76bde14294" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 38557 } } peers { permanent_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44777 } }
I20250625 14:00:09.033329 20829 raft_consensus.cc:383] T 71ff3718b9b3401fa392a99b1ea3de77 P b8dffb1a245d4ba9b1f383d122e00441 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 14:00:09.033699 20829 raft_consensus.cc:738] T 71ff3718b9b3401fa392a99b1ea3de77 P b8dffb1a245d4ba9b1f383d122e00441 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8dffb1a245d4ba9b1f383d122e00441, State: Initialized, Role: FOLLOWER
I20250625 14:00:09.032517 20828 consensus_queue.cc:260] T 71ff3718b9b3401fa392a99b1ea3de77 P 90ff53d19113439ba739ff76bde14294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8dffb1a245d4ba9b1f383d122e00441" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 39411 } } peers { permanent_uuid: "90ff53d19113439ba739ff76bde14294" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 38557 } } peers { permanent_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44777 } }
I20250625 14:00:09.034541 20829 consensus_queue.cc:260] T 71ff3718b9b3401fa392a99b1ea3de77 P b8dffb1a245d4ba9b1f383d122e00441 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8dffb1a245d4ba9b1f383d122e00441" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 39411 } } peers { permanent_uuid: "90ff53d19113439ba739ff76bde14294" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 38557 } } peers { permanent_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44777 } }
I20250625 14:00:09.041736 20830 tablet_bootstrap.cc:492] T 71ff3718b9b3401fa392a99b1ea3de77 P 43a77c62d30c4a3ab625b25b481d6aa8: No bootstrap required, opened a new log
I20250625 14:00:09.042201 20830 ts_tablet_manager.cc:1397] T 71ff3718b9b3401fa392a99b1ea3de77 P 43a77c62d30c4a3ab625b25b481d6aa8: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.009s
I20250625 14:00:09.043496 20828 ts_tablet_manager.cc:1428] T 71ff3718b9b3401fa392a99b1ea3de77 P 90ff53d19113439ba739ff76bde14294: Time spent starting tablet: real 0.015s user 0.005s sys 0.007s
I20250625 14:00:09.044804 20830 raft_consensus.cc:357] T 71ff3718b9b3401fa392a99b1ea3de77 P 43a77c62d30c4a3ab625b25b481d6aa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8dffb1a245d4ba9b1f383d122e00441" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 39411 } } peers { permanent_uuid: "90ff53d19113439ba739ff76bde14294" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 38557 } } peers { permanent_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44777 } }
I20250625 14:00:09.045617 20830 raft_consensus.cc:383] T 71ff3718b9b3401fa392a99b1ea3de77 P 43a77c62d30c4a3ab625b25b481d6aa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 14:00:09.045949 20830 raft_consensus.cc:738] T 71ff3718b9b3401fa392a99b1ea3de77 P 43a77c62d30c4a3ab625b25b481d6aa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43a77c62d30c4a3ab625b25b481d6aa8, State: Initialized, Role: FOLLOWER
I20250625 14:00:09.046725 20830 consensus_queue.cc:260] T 71ff3718b9b3401fa392a99b1ea3de77 P 43a77c62d30c4a3ab625b25b481d6aa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8dffb1a245d4ba9b1f383d122e00441" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 39411 } } peers { permanent_uuid: "90ff53d19113439ba739ff76bde14294" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 38557 } } peers { permanent_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44777 } }
I20250625 14:00:09.060191 20830 ts_tablet_manager.cc:1428] T 71ff3718b9b3401fa392a99b1ea3de77 P 43a77c62d30c4a3ab625b25b481d6aa8: Time spent starting tablet: real 0.018s user 0.006s sys 0.006s
I20250625 14:00:09.060320 20829 ts_tablet_manager.cc:1428] T 71ff3718b9b3401fa392a99b1ea3de77 P b8dffb1a245d4ba9b1f383d122e00441: Time spent starting tablet: real 0.030s user 0.020s sys 0.004s
W20250625 14:00:09.071223 20473 tablet.cc:2378] T 71ff3718b9b3401fa392a99b1ea3de77 P 43a77c62d30c4a3ab625b25b481d6aa8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 14:00:09.077858 20834 raft_consensus.cc:491] T 71ff3718b9b3401fa392a99b1ea3de77 P 90ff53d19113439ba739ff76bde14294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 14:00:09.078536 20834 raft_consensus.cc:513] T 71ff3718b9b3401fa392a99b1ea3de77 P 90ff53d19113439ba739ff76bde14294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8dffb1a245d4ba9b1f383d122e00441" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 39411 } } peers { permanent_uuid: "90ff53d19113439ba739ff76bde14294" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 38557 } } peers { permanent_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44777 } }
I20250625 14:00:09.081452 20834 leader_election.cc:290] T 71ff3718b9b3401fa392a99b1ea3de77 P 90ff53d19113439ba739ff76bde14294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8dffb1a245d4ba9b1f383d122e00441 (127.13.155.194:39411), 43a77c62d30c4a3ab625b25b481d6aa8 (127.13.155.193:44777)
W20250625 14:00:09.092250 20609 tablet.cc:2378] T 71ff3718b9b3401fa392a99b1ea3de77 P b8dffb1a245d4ba9b1f383d122e00441: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 14:00:09.098701 20563 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71ff3718b9b3401fa392a99b1ea3de77" candidate_uuid: "90ff53d19113439ba739ff76bde14294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8dffb1a245d4ba9b1f383d122e00441" is_pre_election: true
I20250625 14:00:09.099329 20563 raft_consensus.cc:2466] T 71ff3718b9b3401fa392a99b1ea3de77 P b8dffb1a245d4ba9b1f383d122e00441 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90ff53d19113439ba739ff76bde14294 in term 0.
I20250625 14:00:09.100610 20639 leader_election.cc:304] T 71ff3718b9b3401fa392a99b1ea3de77 P 90ff53d19113439ba739ff76bde14294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90ff53d19113439ba739ff76bde14294, b8dffb1a245d4ba9b1f383d122e00441; no voters:
I20250625 14:00:09.101505 20834 raft_consensus.cc:2802] T 71ff3718b9b3401fa392a99b1ea3de77 P 90ff53d19113439ba739ff76bde14294 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 14:00:09.101869 20834 raft_consensus.cc:491] T 71ff3718b9b3401fa392a99b1ea3de77 P 90ff53d19113439ba739ff76bde14294 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 14:00:09.102187 20834 raft_consensus.cc:3058] T 71ff3718b9b3401fa392a99b1ea3de77 P 90ff53d19113439ba739ff76bde14294 [term 0 FOLLOWER]: Advancing to term 1
I20250625 14:00:09.105386 20427 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71ff3718b9b3401fa392a99b1ea3de77" candidate_uuid: "90ff53d19113439ba739ff76bde14294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" is_pre_election: true
I20250625 14:00:09.106112 20427 raft_consensus.cc:2466] T 71ff3718b9b3401fa392a99b1ea3de77 P 43a77c62d30c4a3ab625b25b481d6aa8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90ff53d19113439ba739ff76bde14294 in term 0.
I20250625 14:00:09.107431 20834 raft_consensus.cc:513] T 71ff3718b9b3401fa392a99b1ea3de77 P 90ff53d19113439ba739ff76bde14294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8dffb1a245d4ba9b1f383d122e00441" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 39411 } } peers { permanent_uuid: "90ff53d19113439ba739ff76bde14294" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 38557 } } peers { permanent_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44777 } }
I20250625 14:00:09.108731 20834 leader_election.cc:290] T 71ff3718b9b3401fa392a99b1ea3de77 P 90ff53d19113439ba739ff76bde14294 [CANDIDATE]: Term 1 election: Requested vote from peers b8dffb1a245d4ba9b1f383d122e00441 (127.13.155.194:39411), 43a77c62d30c4a3ab625b25b481d6aa8 (127.13.155.193:44777)
I20250625 14:00:09.109545 20563 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71ff3718b9b3401fa392a99b1ea3de77" candidate_uuid: "90ff53d19113439ba739ff76bde14294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8dffb1a245d4ba9b1f383d122e00441"
I20250625 14:00:09.109752 20427 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71ff3718b9b3401fa392a99b1ea3de77" candidate_uuid: "90ff53d19113439ba739ff76bde14294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43a77c62d30c4a3ab625b25b481d6aa8"
I20250625 14:00:09.109966 20563 raft_consensus.cc:3058] T 71ff3718b9b3401fa392a99b1ea3de77 P b8dffb1a245d4ba9b1f383d122e00441 [term 0 FOLLOWER]: Advancing to term 1
I20250625 14:00:09.110263 20427 raft_consensus.cc:3058] T 71ff3718b9b3401fa392a99b1ea3de77 P 43a77c62d30c4a3ab625b25b481d6aa8 [term 0 FOLLOWER]: Advancing to term 1
I20250625 14:00:09.115226 20563 raft_consensus.cc:2466] T 71ff3718b9b3401fa392a99b1ea3de77 P b8dffb1a245d4ba9b1f383d122e00441 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90ff53d19113439ba739ff76bde14294 in term 1.
I20250625 14:00:09.116254 20639 leader_election.cc:304] T 71ff3718b9b3401fa392a99b1ea3de77 P 90ff53d19113439ba739ff76bde14294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90ff53d19113439ba739ff76bde14294, b8dffb1a245d4ba9b1f383d122e00441; no voters:
I20250625 14:00:09.116819 20427 raft_consensus.cc:2466] T 71ff3718b9b3401fa392a99b1ea3de77 P 43a77c62d30c4a3ab625b25b481d6aa8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90ff53d19113439ba739ff76bde14294 in term 1.
I20250625 14:00:09.117264 20834 raft_consensus.cc:2802] T 71ff3718b9b3401fa392a99b1ea3de77 P 90ff53d19113439ba739ff76bde14294 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 14:00:09.120332 20834 raft_consensus.cc:695] T 71ff3718b9b3401fa392a99b1ea3de77 P 90ff53d19113439ba739ff76bde14294 [term 1 LEADER]: Becoming Leader. State: Replica: 90ff53d19113439ba739ff76bde14294, State: Running, Role: LEADER
I20250625 14:00:09.121356 20834 consensus_queue.cc:237] T 71ff3718b9b3401fa392a99b1ea3de77 P 90ff53d19113439ba739ff76bde14294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8dffb1a245d4ba9b1f383d122e00441" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 39411 } } peers { permanent_uuid: "90ff53d19113439ba739ff76bde14294" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 38557 } } peers { permanent_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44777 } }
I20250625 14:00:09.129556 20285 catalog_manager.cc:5582] T 71ff3718b9b3401fa392a99b1ea3de77 P 90ff53d19113439ba739ff76bde14294 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90ff53d19113439ba739ff76bde14294 (127.13.155.195). New cstate: current_term: 1 leader_uuid: "90ff53d19113439ba739ff76bde14294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8dffb1a245d4ba9b1f383d122e00441" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 39411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90ff53d19113439ba739ff76bde14294" member_type: VOTER last_known_addr { host: "127.13.155.195" port: 38557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44777 } health_report { overall_health: UNKNOWN } } }
W20250625 14:00:09.134748 20750 tablet.cc:2378] T 71ff3718b9b3401fa392a99b1ea3de77 P 90ff53d19113439ba739ff76bde14294: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 14:00:09.340958 20427 raft_consensus.cc:1273] T 71ff3718b9b3401fa392a99b1ea3de77 P 43a77c62d30c4a3ab625b25b481d6aa8 [term 1 FOLLOWER]: Refusing update from remote peer 90ff53d19113439ba739ff76bde14294: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 14:00:09.340953 20563 raft_consensus.cc:1273] T 71ff3718b9b3401fa392a99b1ea3de77 P b8dffb1a245d4ba9b1f383d122e00441 [term 1 FOLLOWER]: Refusing update from remote peer 90ff53d19113439ba739ff76bde14294: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 14:00:09.342803 20841 consensus_queue.cc:1035] T 71ff3718b9b3401fa392a99b1ea3de77 P 90ff53d19113439ba739ff76bde14294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43a77c62d30c4a3ab625b25b481d6aa8" member_type: VOTER last_known_addr { host: "127.13.155.193" port: 44777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250625 14:00:09.343665 20834 consensus_queue.cc:1035] T 71ff3718b9b3401fa392a99b1ea3de77 P 90ff53d19113439ba739ff76bde14294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8dffb1a245d4ba9b1f383d122e00441" member_type: VOTER last_known_addr { host: "127.13.155.194" port: 39411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 14:00:14.461370 20407 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 14:00:14.465837 20684 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250625 14:00:14.470567 20543 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
441f83e12ad344b6b3d280cc6428f78c | 127.13.155.254:37319 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.13.155.212:40803 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
43a77c62d30c4a3ab625b25b481d6aa8 | 127.13.155.193:44777 | HEALTHY | <none> | 0 | 0
90ff53d19113439ba739ff76bde14294 | 127.13.155.195:38557 | HEALTHY | <none> | 1 | 0
b8dffb1a245d4ba9b1f383d122e00441 | 127.13.155.194:39411 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.13.155.193 | experimental | 127.13.155.193:44777
local_ip_for_outbound_sockets | 127.13.155.194 | experimental | 127.13.155.194:39411
local_ip_for_outbound_sockets | 127.13.155.195 | experimental | 127.13.155.195:38557
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-0/data/info.pb | hidden | 127.13.155.193:44777
server_dump_info_path | /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-1/data/info.pb | hidden | 127.13.155.194:39411
server_dump_info_path | /tmp/dist-test-task2vErUD/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750859832260225-13935-0/minicluster-data/ts-2/data/info.pb | hidden | 127.13.155.195:38557
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.13.155.212:40803 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
--------------+----+---------+---------------+---------+------------+------------------+-------------
post_rebuild | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250625 14:00:14.778143 13935 log_verifier.cc:126] Checking tablet 71ff3718b9b3401fa392a99b1ea3de77
I20250625 14:00:15.558846 13935 log_verifier.cc:177] Verified matching terms for 205 ops in tablet 71ff3718b9b3401fa392a99b1ea3de77
I20250625 14:00:15.559653 13935 log_verifier.cc:126] Checking tablet db73a6fe1f18431982913f428e57588c
I20250625 14:00:15.559969 13935 log_verifier.cc:177] Verified matching terms for 0 ops in tablet db73a6fe1f18431982913f428e57588c
I20250625 14:00:15.592695 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 20322
I20250625 14:00:15.642027 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 20476
I20250625 14:00:15.679209 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 20612
I20250625 14:00:15.721758 13935 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2vErUD/build/tsan/bin/kudu with pid 20252
2025-06-25T14:00:15Z chronyd exiting
[ OK ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0 (35450 ms)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest (35451 ms total)
[----------] Global test environment tear-down
[==========] 9 tests from 5 test suites ran. (183451 ms total)
[ PASSED ] 8 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AdminCliTest.TestRebuildTables
1 FAILED TEST