Note: This is test shard 6 of 8.
[==========] Running 9 tests from 5 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from AdminCliTest
[ RUN ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250626 13:57:43.576402 2477 test_util.cc:276] Using random seed: -233853223
W20250626 13:57:44.691179 2477 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.073s user 0.485s sys 0.587s
W20250626 13:57:44.691491 2477 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.074s user 0.485s sys 0.587s
I20250626 13:57:44.694268 2477 ts_itest-base.cc:115] Starting cluster with:
I20250626 13:57:44.694450 2477 ts_itest-base.cc:116] --------------
I20250626 13:57:44.694619 2477 ts_itest-base.cc:117] 4 tablet servers
I20250626 13:57:44.694794 2477 ts_itest-base.cc:118] 3 replicas per TS
I20250626 13:57:44.694967 2477 ts_itest-base.cc:119] --------------
2025-06-26T13:57:44Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-26T13:57:44Z Disabled control of system clock
I20250626 13:57:44.728348 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.107.126:41849
--webserver_interface=127.2.107.126
--webserver_port=0
--builtin_ntp_servers=127.2.107.84:38437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.107.126:41849 with env {}
W20250626 13:57:45.025105 2491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:57:45.025660 2491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:57:45.026058 2491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:57:45.056176 2491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 13:57:45.056494 2491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:57:45.056751 2491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 13:57:45.057055 2491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 13:57:45.091212 2491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:38437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.107.126:41849
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.107.126:41849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.107.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:57:45.092746 2491 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:57:45.094377 2491 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:57:45.109879 2498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:57:45.109874 2500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:57:45.110030 2497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:57:46.205062 2491 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250626 13:57:46.205576 2499 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 13:57:46.209009 2491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:57:46.211987 2491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:57:46.213325 2491 hybrid_clock.cc:648] HybridClock initialized: now 1750946266213269 us; error 63 us; skew 500 ppm
I20250626 13:57:46.214099 2491 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:57:46.220599 2491 webserver.cc:469] Webserver started at http://127.2.107.126:44237/ using document root <none> and password file <none>
I20250626 13:57:46.221558 2491 fs_manager.cc:362] Metadata directory not provided
I20250626 13:57:46.221772 2491 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:57:46.222205 2491 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 13:57:46.226521 2491 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d9b0508ef33e44f0850ad667b89cde2b"
format_stamp: "Formatted at 2025-06-26 13:57:46 on dist-test-slave-7p9x"
I20250626 13:57:46.227568 2491 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d9b0508ef33e44f0850ad667b89cde2b"
format_stamp: "Formatted at 2025-06-26 13:57:46 on dist-test-slave-7p9x"
I20250626 13:57:46.234380 2491 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20250626 13:57:46.240018 2507 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:57:46.240994 2491 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250626 13:57:46.241304 2491 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d9b0508ef33e44f0850ad667b89cde2b"
format_stamp: "Formatted at 2025-06-26 13:57:46 on dist-test-slave-7p9x"
I20250626 13:57:46.241636 2491 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:57:46.288621 2491 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:57:46.290112 2491 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:57:46.290550 2491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:57:46.363448 2491 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.126:41849
I20250626 13:57:46.363513 2558 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.126:41849 every 8 connection(s)
I20250626 13:57:46.366048 2491 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250626 13:57:46.368089 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 2491
I20250626 13:57:46.368526 2477 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250626 13:57:46.372294 2559 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:57:46.392100 2559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b: Bootstrap starting.
I20250626 13:57:46.398085 2559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b: Neither blocks nor log segments found. Creating new log.
I20250626 13:57:46.399724 2559 log.cc:826] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b: Log is configured to *not* fsync() on all Append() calls
I20250626 13:57:46.404464 2559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b: No bootstrap required, opened a new log
I20250626 13:57:46.423586 2559 raft_consensus.cc:357] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b0508ef33e44f0850ad667b89cde2b" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 41849 } }
I20250626 13:57:46.424240 2559 raft_consensus.cc:383] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 13:57:46.424467 2559 raft_consensus.cc:738] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9b0508ef33e44f0850ad667b89cde2b, State: Initialized, Role: FOLLOWER
I20250626 13:57:46.425213 2559 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b0508ef33e44f0850ad667b89cde2b" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 41849 } }
I20250626 13:57:46.425825 2559 raft_consensus.cc:397] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 13:57:46.426147 2559 raft_consensus.cc:491] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 13:57:46.426824 2559 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b [term 0 FOLLOWER]: Advancing to term 1
I20250626 13:57:46.430932 2559 raft_consensus.cc:513] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b0508ef33e44f0850ad667b89cde2b" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 41849 } }
I20250626 13:57:46.431707 2559 leader_election.cc:304] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9b0508ef33e44f0850ad667b89cde2b; no voters:
I20250626 13:57:46.433997 2559 leader_election.cc:290] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 13:57:46.434567 2564 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b [term 1 FOLLOWER]: Leader election won for term 1
I20250626 13:57:46.436800 2564 raft_consensus.cc:695] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b [term 1 LEADER]: Becoming Leader. State: Replica: d9b0508ef33e44f0850ad667b89cde2b, State: Running, Role: LEADER
I20250626 13:57:46.437516 2564 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b0508ef33e44f0850ad667b89cde2b" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 41849 } }
I20250626 13:57:46.438910 2559 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b [sys.catalog]: configured and running, proceeding with master startup.
I20250626 13:57:46.448330 2566 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9b0508ef33e44f0850ad667b89cde2b. Latest consensus state: current_term: 1 leader_uuid: "d9b0508ef33e44f0850ad667b89cde2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b0508ef33e44f0850ad667b89cde2b" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 41849 } } }
I20250626 13:57:46.449452 2566 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b [sys.catalog]: This master's current role is: LEADER
I20250626 13:57:46.454684 2573 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 13:57:46.452492 2565 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9b0508ef33e44f0850ad667b89cde2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b0508ef33e44f0850ad667b89cde2b" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 41849 } } }
I20250626 13:57:46.455644 2565 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b [sys.catalog]: This master's current role is: LEADER
I20250626 13:57:46.466167 2573 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 13:57:46.483146 2573 catalog_manager.cc:1349] Generated new cluster ID: d239439210ec4bbdb52eec4d4b0ee585
I20250626 13:57:46.483459 2573 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 13:57:46.502210 2573 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 13:57:46.504170 2573 catalog_manager.cc:1506] Loading token signing keys...
I20250626 13:57:46.525323 2573 catalog_manager.cc:5955] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b: Generated new TSK 0
I20250626 13:57:46.526389 2573 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 13:57:46.547370 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.107.65:0
--local_ip_for_outbound_sockets=127.2.107.65
--webserver_interface=127.2.107.65
--webserver_port=0
--tserver_master_addrs=127.2.107.126:41849
--builtin_ntp_servers=127.2.107.84:38437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250626 13:57:46.844794 2583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:57:46.845402 2583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:57:46.845896 2583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:57:46.877488 2583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:57:46.878330 2583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.107.65
I20250626 13:57:46.922189 2583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:38437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.107.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.107.65
--webserver_port=0
--tserver_master_addrs=127.2.107.126:41849
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.107.65
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:57:46.923777 2583 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:57:46.926016 2583 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:57:46.946964 2589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:57:46.954173 2590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:57:46.954592 2592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:57:48.043898 2591 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 13:57:48.043967 2583 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 13:57:48.048358 2583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:57:48.051270 2583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:57:48.052767 2583 hybrid_clock.cc:648] HybridClock initialized: now 1750946268052717 us; error 50 us; skew 500 ppm
I20250626 13:57:48.053853 2583 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:57:48.061666 2583 webserver.cc:469] Webserver started at http://127.2.107.65:43871/ using document root <none> and password file <none>
I20250626 13:57:48.062841 2583 fs_manager.cc:362] Metadata directory not provided
I20250626 13:57:48.063118 2583 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:57:48.063699 2583 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 13:57:48.069998 2583 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "a6abf8cb57c24c17871f9fc9db82be83"
format_stamp: "Formatted at 2025-06-26 13:57:48 on dist-test-slave-7p9x"
I20250626 13:57:48.071448 2583 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "a6abf8cb57c24c17871f9fc9db82be83"
format_stamp: "Formatted at 2025-06-26 13:57:48 on dist-test-slave-7p9x"
I20250626 13:57:48.079262 2583 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20250626 13:57:48.085903 2599 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:57:48.087275 2583 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20250626 13:57:48.087667 2583 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "a6abf8cb57c24c17871f9fc9db82be83"
format_stamp: "Formatted at 2025-06-26 13:57:48 on dist-test-slave-7p9x"
I20250626 13:57:48.088107 2583 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:57:48.139456 2583 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:57:48.140956 2583 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:57:48.141400 2583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:57:48.144188 2583 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 13:57:48.148727 2583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 13:57:48.148934 2583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 13:57:48.149211 2583 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 13:57:48.149369 2583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 13:57:48.300469 2583 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.65:42753
I20250626 13:57:48.300560 2711 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.65:42753 every 8 connection(s)
I20250626 13:57:48.303169 2583 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250626 13:57:48.306281 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 2583
I20250626 13:57:48.306799 2477 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250626 13:57:48.313488 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.107.66:0
--local_ip_for_outbound_sockets=127.2.107.66
--webserver_interface=127.2.107.66
--webserver_port=0
--tserver_master_addrs=127.2.107.126:41849
--builtin_ntp_servers=127.2.107.84:38437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 13:57:48.326112 2712 heartbeater.cc:344] Connected to a master server at 127.2.107.126:41849
I20250626 13:57:48.326545 2712 heartbeater.cc:461] Registering TS with master...
I20250626 13:57:48.327548 2712 heartbeater.cc:507] Master 127.2.107.126:41849 requested a full tablet report, sending...
I20250626 13:57:48.330263 2524 ts_manager.cc:194] Registered new tserver with Master: a6abf8cb57c24c17871f9fc9db82be83 (127.2.107.65:42753)
I20250626 13:57:48.332973 2524 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.107.65:40103
W20250626 13:57:48.616708 2716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:57:48.617210 2716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:57:48.617667 2716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:57:48.648005 2716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:57:48.648799 2716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.107.66
I20250626 13:57:48.683483 2716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:38437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.107.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.107.66
--webserver_port=0
--tserver_master_addrs=127.2.107.126:41849
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.107.66
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:57:48.684739 2716 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:57:48.686360 2716 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:57:48.703410 2722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:57:49.336585 2712 heartbeater.cc:499] Master 127.2.107.126:41849 was elected leader, sending a full tablet report...
W20250626 13:57:48.703399 2725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:57:48.704296 2723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:57:48.705785 2716 server_base.cc:1048] running on GCE node
I20250626 13:57:49.780615 2716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:57:49.783116 2716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:57:49.784530 2716 hybrid_clock.cc:648] HybridClock initialized: now 1750946269784492 us; error 57 us; skew 500 ppm
I20250626 13:57:49.785318 2716 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:57:49.791782 2716 webserver.cc:469] Webserver started at http://127.2.107.66:45051/ using document root <none> and password file <none>
I20250626 13:57:49.792622 2716 fs_manager.cc:362] Metadata directory not provided
I20250626 13:57:49.792811 2716 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:57:49.793248 2716 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 13:57:49.798049 2716 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "1a2182284f334697becb5cb14da3c22c"
format_stamp: "Formatted at 2025-06-26 13:57:49 on dist-test-slave-7p9x"
I20250626 13:57:49.799124 2716 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "1a2182284f334697becb5cb14da3c22c"
format_stamp: "Formatted at 2025-06-26 13:57:49 on dist-test-slave-7p9x"
I20250626 13:57:49.805810 2716 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250626 13:57:49.811015 2732 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:57:49.812482 2716 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20250626 13:57:49.812770 2716 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "1a2182284f334697becb5cb14da3c22c"
format_stamp: "Formatted at 2025-06-26 13:57:49 on dist-test-slave-7p9x"
I20250626 13:57:49.813089 2716 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:57:49.877156 2716 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:57:49.878572 2716 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:57:49.878983 2716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:57:49.881705 2716 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 13:57:49.885923 2716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 13:57:49.886123 2716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 13:57:49.886370 2716 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 13:57:49.886519 2716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 13:57:50.028461 2716 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.66:42025
I20250626 13:57:50.028554 2844 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.66:42025 every 8 connection(s)
I20250626 13:57:50.030923 2716 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250626 13:57:50.033090 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 2716
I20250626 13:57:50.033586 2477 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250626 13:57:50.040298 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.107.67:0
--local_ip_for_outbound_sockets=127.2.107.67
--webserver_interface=127.2.107.67
--webserver_port=0
--tserver_master_addrs=127.2.107.126:41849
--builtin_ntp_servers=127.2.107.84:38437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 13:57:50.055931 2845 heartbeater.cc:344] Connected to a master server at 127.2.107.126:41849
I20250626 13:57:50.056425 2845 heartbeater.cc:461] Registering TS with master...
I20250626 13:57:50.057503 2845 heartbeater.cc:507] Master 127.2.107.126:41849 requested a full tablet report, sending...
I20250626 13:57:50.059624 2524 ts_manager.cc:194] Registered new tserver with Master: 1a2182284f334697becb5cb14da3c22c (127.2.107.66:42025)
I20250626 13:57:50.061269 2524 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.107.66:35435
W20250626 13:57:50.340382 2849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:57:50.340894 2849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:57:50.341552 2849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:57:50.372891 2849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:57:50.373785 2849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.107.67
I20250626 13:57:50.408214 2849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:38437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.107.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.107.67
--webserver_port=0
--tserver_master_addrs=127.2.107.126:41849
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.107.67
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:57:50.409556 2849 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:57:50.411144 2849 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:57:50.427343 2858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:57:51.064872 2845 heartbeater.cc:499] Master 127.2.107.126:41849 was elected leader, sending a full tablet report...
W20250626 13:57:50.427371 2855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:57:50.427381 2856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:57:51.524797 2857 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 13:57:51.524883 2849 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 13:57:51.528760 2849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:57:51.531328 2849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:57:51.532739 2849 hybrid_clock.cc:648] HybridClock initialized: now 1750946271532702 us; error 55 us; skew 500 ppm
I20250626 13:57:51.533560 2849 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:57:51.540109 2849 webserver.cc:469] Webserver started at http://127.2.107.67:44617/ using document root <none> and password file <none>
I20250626 13:57:51.541043 2849 fs_manager.cc:362] Metadata directory not provided
I20250626 13:57:51.541296 2849 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:57:51.541742 2849 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 13:57:51.546232 2849 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "aa223e2d12164378942ccb1f6d040b86"
format_stamp: "Formatted at 2025-06-26 13:57:51 on dist-test-slave-7p9x"
I20250626 13:57:51.547374 2849 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "aa223e2d12164378942ccb1f6d040b86"
format_stamp: "Formatted at 2025-06-26 13:57:51 on dist-test-slave-7p9x"
I20250626 13:57:51.554332 2849 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250626 13:57:51.560220 2865 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:57:51.561324 2849 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20250626 13:57:51.561617 2849 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "aa223e2d12164378942ccb1f6d040b86"
format_stamp: "Formatted at 2025-06-26 13:57:51 on dist-test-slave-7p9x"
I20250626 13:57:51.561935 2849 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:57:51.614209 2849 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:57:51.615625 2849 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:57:51.616034 2849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:57:51.618538 2849 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 13:57:51.622885 2849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 13:57:51.623077 2849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 13:57:51.623328 2849 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 13:57:51.623476 2849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 13:57:51.769639 2849 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.67:36545
I20250626 13:57:51.769731 2977 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.67:36545 every 8 connection(s)
I20250626 13:57:51.772198 2849 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250626 13:57:51.775417 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 2849
I20250626 13:57:51.775882 2477 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250626 13:57:51.783020 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.107.68:0
--local_ip_for_outbound_sockets=127.2.107.68
--webserver_interface=127.2.107.68
--webserver_port=0
--tserver_master_addrs=127.2.107.126:41849
--builtin_ntp_servers=127.2.107.84:38437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 13:57:51.796128 2978 heartbeater.cc:344] Connected to a master server at 127.2.107.126:41849
I20250626 13:57:51.796543 2978 heartbeater.cc:461] Registering TS with master...
I20250626 13:57:51.797578 2978 heartbeater.cc:507] Master 127.2.107.126:41849 requested a full tablet report, sending...
I20250626 13:57:51.799921 2524 ts_manager.cc:194] Registered new tserver with Master: aa223e2d12164378942ccb1f6d040b86 (127.2.107.67:36545)
I20250626 13:57:51.801707 2524 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.107.67:42645
W20250626 13:57:52.084792 2982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:57:52.085305 2982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:57:52.085757 2982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:57:52.116514 2982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:57:52.117462 2982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.107.68
I20250626 13:57:52.151574 2982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:38437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.107.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.2.107.68
--webserver_port=0
--tserver_master_addrs=127.2.107.126:41849
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.107.68
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:57:52.152774 2982 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:57:52.154373 2982 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:57:52.170607 2989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:57:52.804854 2978 heartbeater.cc:499] Master 127.2.107.126:41849 was elected leader, sending a full tablet report...
W20250626 13:57:52.171121 2988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:57:52.172235 2991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:57:52.172350 2982 server_base.cc:1048] running on GCE node
I20250626 13:57:53.261379 2982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:57:53.264487 2982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:57:53.265898 2982 hybrid_clock.cc:648] HybridClock initialized: now 1750946273265863 us; error 56 us; skew 500 ppm
I20250626 13:57:53.266661 2982 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:57:53.273284 2982 webserver.cc:469] Webserver started at http://127.2.107.68:45385/ using document root <none> and password file <none>
I20250626 13:57:53.274148 2982 fs_manager.cc:362] Metadata directory not provided
I20250626 13:57:53.274329 2982 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:57:53.274732 2982 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 13:57:53.279026 2982 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "7100218b59a049d48f8431324f5bbfd4"
format_stamp: "Formatted at 2025-06-26 13:57:53 on dist-test-slave-7p9x"
I20250626 13:57:53.280071 2982 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "7100218b59a049d48f8431324f5bbfd4"
format_stamp: "Formatted at 2025-06-26 13:57:53 on dist-test-slave-7p9x"
I20250626 13:57:53.287536 2982 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20250626 13:57:53.293150 2998 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:57:53.294135 2982 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250626 13:57:53.294430 2982 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "7100218b59a049d48f8431324f5bbfd4"
format_stamp: "Formatted at 2025-06-26 13:57:53 on dist-test-slave-7p9x"
I20250626 13:57:53.294734 2982 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:57:53.350117 2982 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:57:53.351662 2982 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:57:53.352159 2982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:57:53.355396 2982 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 13:57:53.359382 2982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 13:57:53.359575 2982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 13:57:53.359793 2982 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 13:57:53.359974 2982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 13:57:53.507239 2982 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.68:40935
I20250626 13:57:53.507339 3110 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.68:40935 every 8 connection(s)
I20250626 13:57:53.509704 2982 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250626 13:57:53.513573 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 2982
I20250626 13:57:53.514065 2477 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250626 13:57:53.531623 3111 heartbeater.cc:344] Connected to a master server at 127.2.107.126:41849
I20250626 13:57:53.532090 3111 heartbeater.cc:461] Registering TS with master...
I20250626 13:57:53.533345 3111 heartbeater.cc:507] Master 127.2.107.126:41849 requested a full tablet report, sending...
I20250626 13:57:53.535419 2524 ts_manager.cc:194] Registered new tserver with Master: 7100218b59a049d48f8431324f5bbfd4 (127.2.107.68:40935)
I20250626 13:57:53.536573 2524 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.107.68:42439
I20250626 13:57:53.548714 2477 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250626 13:57:53.587673 2524 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41126:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250626 13:57:53.654320 2647 tablet_service.cc:1468] Processing CreateTablet for tablet 99aedc8b4f7d4290aea59589aa93c419 (DEFAULT_TABLE table=TestTable [id=0ea0fa958f5649e09bebe81150542504]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 13:57:53.654805 2913 tablet_service.cc:1468] Processing CreateTablet for tablet 99aedc8b4f7d4290aea59589aa93c419 (DEFAULT_TABLE table=TestTable [id=0ea0fa958f5649e09bebe81150542504]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 13:57:53.656244 2647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99aedc8b4f7d4290aea59589aa93c419. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:57:53.656571 2913 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99aedc8b4f7d4290aea59589aa93c419. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:57:53.662672 2780 tablet_service.cc:1468] Processing CreateTablet for tablet 99aedc8b4f7d4290aea59589aa93c419 (DEFAULT_TABLE table=TestTable [id=0ea0fa958f5649e09bebe81150542504]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 13:57:53.664011 2780 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99aedc8b4f7d4290aea59589aa93c419. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:57:53.683502 3131 tablet_bootstrap.cc:492] T 99aedc8b4f7d4290aea59589aa93c419 P aa223e2d12164378942ccb1f6d040b86: Bootstrap starting.
I20250626 13:57:53.686285 3133 tablet_bootstrap.cc:492] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83: Bootstrap starting.
I20250626 13:57:53.687865 3132 tablet_bootstrap.cc:492] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c: Bootstrap starting.
I20250626 13:57:53.692453 3131 tablet_bootstrap.cc:654] T 99aedc8b4f7d4290aea59589aa93c419 P aa223e2d12164378942ccb1f6d040b86: Neither blocks nor log segments found. Creating new log.
I20250626 13:57:53.693599 3133 tablet_bootstrap.cc:654] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83: Neither blocks nor log segments found. Creating new log.
I20250626 13:57:53.694550 3131 log.cc:826] T 99aedc8b4f7d4290aea59589aa93c419 P aa223e2d12164378942ccb1f6d040b86: Log is configured to *not* fsync() on all Append() calls
I20250626 13:57:53.695533 3132 tablet_bootstrap.cc:654] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c: Neither blocks nor log segments found. Creating new log.
I20250626 13:57:53.696002 3133 log.cc:826] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83: Log is configured to *not* fsync() on all Append() calls
I20250626 13:57:53.697862 3132 log.cc:826] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c: Log is configured to *not* fsync() on all Append() calls
I20250626 13:57:53.699980 3131 tablet_bootstrap.cc:492] T 99aedc8b4f7d4290aea59589aa93c419 P aa223e2d12164378942ccb1f6d040b86: No bootstrap required, opened a new log
I20250626 13:57:53.700459 3131 ts_tablet_manager.cc:1397] T 99aedc8b4f7d4290aea59589aa93c419 P aa223e2d12164378942ccb1f6d040b86: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.012s
I20250626 13:57:53.702510 3133 tablet_bootstrap.cc:492] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83: No bootstrap required, opened a new log
I20250626 13:57:53.702960 3133 ts_tablet_manager.cc:1397] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.004s
I20250626 13:57:53.704371 3132 tablet_bootstrap.cc:492] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c: No bootstrap required, opened a new log
I20250626 13:57:53.704818 3132 ts_tablet_manager.cc:1397] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.008s
I20250626 13:57:53.721721 3133 raft_consensus.cc:357] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 42753 } } peers { permanent_uuid: "aa223e2d12164378942ccb1f6d040b86" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 36545 } } peers { permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 } }
I20250626 13:57:53.722450 3133 raft_consensus.cc:383] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 13:57:53.722709 3133 raft_consensus.cc:738] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6abf8cb57c24c17871f9fc9db82be83, State: Initialized, Role: FOLLOWER
I20250626 13:57:53.723704 3133 consensus_queue.cc:260] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 42753 } } peers { permanent_uuid: "aa223e2d12164378942ccb1f6d040b86" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 36545 } } peers { permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 } }
I20250626 13:57:53.728025 3133 ts_tablet_manager.cc:1428] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83: Time spent starting tablet: real 0.025s user 0.016s sys 0.007s
I20250626 13:57:53.729348 3131 raft_consensus.cc:357] T 99aedc8b4f7d4290aea59589aa93c419 P aa223e2d12164378942ccb1f6d040b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 42753 } } peers { permanent_uuid: "aa223e2d12164378942ccb1f6d040b86" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 36545 } } peers { permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 } }
I20250626 13:57:53.730207 3131 raft_consensus.cc:383] T 99aedc8b4f7d4290aea59589aa93c419 P aa223e2d12164378942ccb1f6d040b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 13:57:53.730490 3131 raft_consensus.cc:738] T 99aedc8b4f7d4290aea59589aa93c419 P aa223e2d12164378942ccb1f6d040b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa223e2d12164378942ccb1f6d040b86, State: Initialized, Role: FOLLOWER
I20250626 13:57:53.731314 3131 consensus_queue.cc:260] T 99aedc8b4f7d4290aea59589aa93c419 P aa223e2d12164378942ccb1f6d040b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 42753 } } peers { permanent_uuid: "aa223e2d12164378942ccb1f6d040b86" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 36545 } } peers { permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 } }
I20250626 13:57:53.731412 3132 raft_consensus.cc:357] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 42753 } } peers { permanent_uuid: "aa223e2d12164378942ccb1f6d040b86" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 36545 } } peers { permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 } }
I20250626 13:57:53.732213 3132 raft_consensus.cc:383] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 13:57:53.732491 3132 raft_consensus.cc:738] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a2182284f334697becb5cb14da3c22c, State: Initialized, Role: FOLLOWER
I20250626 13:57:53.733659 3132 consensus_queue.cc:260] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 42753 } } peers { permanent_uuid: "aa223e2d12164378942ccb1f6d040b86" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 36545 } } peers { permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 } }
I20250626 13:57:53.735705 3131 ts_tablet_manager.cc:1428] T 99aedc8b4f7d4290aea59589aa93c419 P aa223e2d12164378942ccb1f6d040b86: Time spent starting tablet: real 0.035s user 0.026s sys 0.007s
I20250626 13:57:53.738729 3132 ts_tablet_manager.cc:1428] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c: Time spent starting tablet: real 0.034s user 0.030s sys 0.004s
I20250626 13:57:53.763669 3138 raft_consensus.cc:491] T 99aedc8b4f7d4290aea59589aa93c419 P aa223e2d12164378942ccb1f6d040b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 13:57:53.764091 3138 raft_consensus.cc:513] T 99aedc8b4f7d4290aea59589aa93c419 P aa223e2d12164378942ccb1f6d040b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 42753 } } peers { permanent_uuid: "aa223e2d12164378942ccb1f6d040b86" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 36545 } } peers { permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 } }
I20250626 13:57:53.766543 3138 leader_election.cc:290] T 99aedc8b4f7d4290aea59589aa93c419 P aa223e2d12164378942ccb1f6d040b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6abf8cb57c24c17871f9fc9db82be83 (127.2.107.65:42753), 1a2182284f334697becb5cb14da3c22c (127.2.107.66:42025)
W20250626 13:57:53.778430 2979 tablet.cc:2378] T 99aedc8b4f7d4290aea59589aa93c419 P aa223e2d12164378942ccb1f6d040b86: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 13:57:53.782048 2667 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "99aedc8b4f7d4290aea59589aa93c419" candidate_uuid: "aa223e2d12164378942ccb1f6d040b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6abf8cb57c24c17871f9fc9db82be83" is_pre_election: true
I20250626 13:57:53.782897 2667 raft_consensus.cc:2466] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa223e2d12164378942ccb1f6d040b86 in term 0.
I20250626 13:57:53.783879 2800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "99aedc8b4f7d4290aea59589aa93c419" candidate_uuid: "aa223e2d12164378942ccb1f6d040b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a2182284f334697becb5cb14da3c22c" is_pre_election: true
I20250626 13:57:53.784197 2866 leader_election.cc:304] T 99aedc8b4f7d4290aea59589aa93c419 P aa223e2d12164378942ccb1f6d040b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6abf8cb57c24c17871f9fc9db82be83, aa223e2d12164378942ccb1f6d040b86; no voters:
I20250626 13:57:53.784490 2800 raft_consensus.cc:2466] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa223e2d12164378942ccb1f6d040b86 in term 0.
I20250626 13:57:53.784996 3138 raft_consensus.cc:2802] T 99aedc8b4f7d4290aea59589aa93c419 P aa223e2d12164378942ccb1f6d040b86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 13:57:53.785343 3138 raft_consensus.cc:491] T 99aedc8b4f7d4290aea59589aa93c419 P aa223e2d12164378942ccb1f6d040b86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 13:57:53.785650 3138 raft_consensus.cc:3058] T 99aedc8b4f7d4290aea59589aa93c419 P aa223e2d12164378942ccb1f6d040b86 [term 0 FOLLOWER]: Advancing to term 1
W20250626 13:57:53.788641 2846 tablet.cc:2378] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 13:57:53.791139 3138 raft_consensus.cc:513] T 99aedc8b4f7d4290aea59589aa93c419 P aa223e2d12164378942ccb1f6d040b86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 42753 } } peers { permanent_uuid: "aa223e2d12164378942ccb1f6d040b86" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 36545 } } peers { permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 } }
I20250626 13:57:53.792464 3138 leader_election.cc:290] T 99aedc8b4f7d4290aea59589aa93c419 P aa223e2d12164378942ccb1f6d040b86 [CANDIDATE]: Term 1 election: Requested vote from peers a6abf8cb57c24c17871f9fc9db82be83 (127.2.107.65:42753), 1a2182284f334697becb5cb14da3c22c (127.2.107.66:42025)
I20250626 13:57:53.793279 2667 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "99aedc8b4f7d4290aea59589aa93c419" candidate_uuid: "aa223e2d12164378942ccb1f6d040b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6abf8cb57c24c17871f9fc9db82be83"
I20250626 13:57:53.793421 2800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "99aedc8b4f7d4290aea59589aa93c419" candidate_uuid: "aa223e2d12164378942ccb1f6d040b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a2182284f334697becb5cb14da3c22c"
I20250626 13:57:53.793797 2800 raft_consensus.cc:3058] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c [term 0 FOLLOWER]: Advancing to term 1
I20250626 13:57:53.793781 2667 raft_consensus.cc:3058] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [term 0 FOLLOWER]: Advancing to term 1
I20250626 13:57:53.800141 2800 raft_consensus.cc:2466] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa223e2d12164378942ccb1f6d040b86 in term 1.
I20250626 13:57:53.800336 2667 raft_consensus.cc:2466] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa223e2d12164378942ccb1f6d040b86 in term 1.
I20250626 13:57:53.801602 2869 leader_election.cc:304] T 99aedc8b4f7d4290aea59589aa93c419 P aa223e2d12164378942ccb1f6d040b86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a2182284f334697becb5cb14da3c22c, aa223e2d12164378942ccb1f6d040b86; no voters:
I20250626 13:57:53.802182 3138 raft_consensus.cc:2802] T 99aedc8b4f7d4290aea59589aa93c419 P aa223e2d12164378942ccb1f6d040b86 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 13:57:53.803854 3138 raft_consensus.cc:695] T 99aedc8b4f7d4290aea59589aa93c419 P aa223e2d12164378942ccb1f6d040b86 [term 1 LEADER]: Becoming Leader. State: Replica: aa223e2d12164378942ccb1f6d040b86, State: Running, Role: LEADER
I20250626 13:57:53.805271 3138 consensus_queue.cc:237] T 99aedc8b4f7d4290aea59589aa93c419 P aa223e2d12164378942ccb1f6d040b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 42753 } } peers { permanent_uuid: "aa223e2d12164378942ccb1f6d040b86" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 36545 } } peers { permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 } }
W20250626 13:57:53.812789 2713 tablet.cc:2378] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 13:57:53.820201 2522 catalog_manager.cc:5582] T 99aedc8b4f7d4290aea59589aa93c419 P aa223e2d12164378942ccb1f6d040b86 reported cstate change: term changed from 0 to 1, leader changed from <none> to aa223e2d12164378942ccb1f6d040b86 (127.2.107.67). New cstate: current_term: 1 leader_uuid: "aa223e2d12164378942ccb1f6d040b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 42753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa223e2d12164378942ccb1f6d040b86" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 36545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 } health_report { overall_health: UNKNOWN } } }
I20250626 13:57:53.847589 2477 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250626 13:57:53.851011 2477 ts_itest-base.cc:246] Waiting for 1 tablets on tserver a6abf8cb57c24c17871f9fc9db82be83 to finish bootstrapping
I20250626 13:57:53.864617 2477 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 1a2182284f334697becb5cb14da3c22c to finish bootstrapping
I20250626 13:57:53.874511 2477 ts_itest-base.cc:246] Waiting for 1 tablets on tserver aa223e2d12164378942ccb1f6d040b86 to finish bootstrapping
I20250626 13:57:53.885326 2477 kudu-admin-test.cc:709] Waiting for Master to see the current replicas...
I20250626 13:57:53.888234 2477 kudu-admin-test.cc:716] Tablet locations:
tablet_locations {
tablet_id: "99aedc8b4f7d4290aea59589aa93c419"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: LEADER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83"
rpc_addresses {
host: "127.2.107.65"
port: 42753
}
}
ts_infos {
permanent_uuid: "aa223e2d12164378942ccb1f6d040b86"
rpc_addresses {
host: "127.2.107.67"
port: 36545
}
}
ts_infos {
permanent_uuid: "1a2182284f334697becb5cb14da3c22c"
rpc_addresses {
host: "127.2.107.66"
port: 42025
}
}
I20250626 13:57:54.241339 3138 consensus_queue.cc:1035] T 99aedc8b4f7d4290aea59589aa93c419 P aa223e2d12164378942ccb1f6d040b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 13:57:54.256057 3138 consensus_queue.cc:1035] T 99aedc8b4f7d4290aea59589aa93c419 P aa223e2d12164378942ccb1f6d040b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 42753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 13:57:54.260026 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 2849
W20250626 13:57:54.281996 2603 connection.cc:537] server connection from 127.2.107.67:49257 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250626 13:57:54.283013 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 2491
I20250626 13:57:54.311619 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.107.126:41849
--webserver_interface=127.2.107.126
--webserver_port=44237
--builtin_ntp_servers=127.2.107.84:38437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.107.126:41849 with env {}
I20250626 13:57:54.539348 3111 heartbeater.cc:499] Master 127.2.107.126:41849 was elected leader, sending a full tablet report...
W20250626 13:57:54.543007 3111 heartbeater.cc:646] Failed to heartbeat to 127.2.107.126:41849 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.107.126:41849: connect: Connection refused (error 111)
W20250626 13:57:54.609031 3153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:57:54.609601 3153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:57:54.609989 3153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:57:54.640658 3153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 13:57:54.640930 3153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:57:54.641145 3153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 13:57:54.641340 3153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 13:57:54.674805 3153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:38437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.107.126:41849
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.107.126:41849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.107.126
--webserver_port=44237
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:57:54.676002 3153 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:57:54.677572 3153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:57:54.690899 3161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:57:55.271067 2845 heartbeater.cc:646] Failed to heartbeat to 127.2.107.126:41849 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.107.126:41849: connect: Connection refused (error 111)
W20250626 13:57:55.285866 2712 heartbeater.cc:646] Failed to heartbeat to 127.2.107.126:41849 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.107.126:41849: connect: Connection refused (error 111)
I20250626 13:57:55.766072 3170 raft_consensus.cc:491] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aa223e2d12164378942ccb1f6d040b86)
I20250626 13:57:55.774761 3170 raft_consensus.cc:513] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 42753 } } peers { permanent_uuid: "aa223e2d12164378942ccb1f6d040b86" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 36545 } } peers { permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 } }
I20250626 13:57:55.777788 3170 leader_election.cc:290] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aa223e2d12164378942ccb1f6d040b86 (127.2.107.67:36545), 1a2182284f334697becb5cb14da3c22c (127.2.107.66:42025)
W20250626 13:57:55.784791 2603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.107.67:36545: connect: Connection refused (error 111)
I20250626 13:57:55.886849 3169 raft_consensus.cc:491] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aa223e2d12164378942ccb1f6d040b86)
I20250626 13:57:55.887516 3169 raft_consensus.cc:513] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 42753 } } peers { permanent_uuid: "aa223e2d12164378942ccb1f6d040b86" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 36545 } } peers { permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 } }
W20250626 13:57:55.916146 2603 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.107.67:36545, user_credentials={real_user=slave}} blocked reactor thread for 131469us
I20250626 13:57:55.916872 3169 leader_election.cc:290] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a6abf8cb57c24c17871f9fc9db82be83 (127.2.107.65:42753), aa223e2d12164378942ccb1f6d040b86 (127.2.107.67:36545)
W20250626 13:57:55.918385 2736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.107.67:36545: connect: Connection refused (error 111)
W20250626 13:57:55.926599 2736 leader_election.cc:336] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aa223e2d12164378942ccb1f6d040b86 (127.2.107.67:36545): Network error: Client connection negotiation failed: client connection to 127.2.107.67:36545: connect: Connection refused (error 111)
W20250626 13:57:55.927480 2603 leader_election.cc:336] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aa223e2d12164378942ccb1f6d040b86 (127.2.107.67:36545): Network error: Client connection negotiation failed: client connection to 127.2.107.67:36545: connect: Connection refused (error 111)
I20250626 13:57:55.928776 2800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "99aedc8b4f7d4290aea59589aa93c419" candidate_uuid: "a6abf8cb57c24c17871f9fc9db82be83" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1a2182284f334697becb5cb14da3c22c" is_pre_election: true
I20250626 13:57:55.929378 2800 raft_consensus.cc:2466] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6abf8cb57c24c17871f9fc9db82be83 in term 1.
I20250626 13:57:55.930402 2603 leader_election.cc:304] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1a2182284f334697becb5cb14da3c22c, a6abf8cb57c24c17871f9fc9db82be83; no voters: aa223e2d12164378942ccb1f6d040b86
I20250626 13:57:55.931241 3170 raft_consensus.cc:2802] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 13:57:55.931571 3170 raft_consensus.cc:491] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [term 1 FOLLOWER]: Starting leader election (detected failure of leader aa223e2d12164378942ccb1f6d040b86)
I20250626 13:57:55.931926 3170 raft_consensus.cc:3058] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [term 1 FOLLOWER]: Advancing to term 2
I20250626 13:57:55.932974 2667 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "99aedc8b4f7d4290aea59589aa93c419" candidate_uuid: "1a2182284f334697becb5cb14da3c22c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a6abf8cb57c24c17871f9fc9db82be83" is_pre_election: true
I20250626 13:57:55.953315 3170 raft_consensus.cc:513] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 42753 } } peers { permanent_uuid: "aa223e2d12164378942ccb1f6d040b86" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 36545 } } peers { permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 } }
I20250626 13:57:55.954537 2667 raft_consensus.cc:2391] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1a2182284f334697becb5cb14da3c22c in current term 2: Already voted for candidate a6abf8cb57c24c17871f9fc9db82be83 in this term.
I20250626 13:57:55.955191 3170 leader_election.cc:290] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [CANDIDATE]: Term 2 election: Requested vote from peers aa223e2d12164378942ccb1f6d040b86 (127.2.107.67:36545), 1a2182284f334697becb5cb14da3c22c (127.2.107.66:42025)
I20250626 13:57:55.957747 2800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "99aedc8b4f7d4290aea59589aa93c419" candidate_uuid: "a6abf8cb57c24c17871f9fc9db82be83" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1a2182284f334697becb5cb14da3c22c"
I20250626 13:57:55.958310 2800 raft_consensus.cc:3058] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c [term 1 FOLLOWER]: Advancing to term 2
I20250626 13:57:55.958607 2733 leader_election.cc:304] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1a2182284f334697becb5cb14da3c22c; no voters: a6abf8cb57c24c17871f9fc9db82be83, aa223e2d12164378942ccb1f6d040b86
W20250626 13:57:55.959964 2603 leader_election.cc:336] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer aa223e2d12164378942ccb1f6d040b86 (127.2.107.67:36545): Network error: Client connection negotiation failed: client connection to 127.2.107.67:36545: connect: Connection refused (error 111)
W20250626 13:57:54.690927 3160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:57:54.691672 3163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:57:54.693063 3153 server_base.cc:1048] running on GCE node
I20250626 13:57:56.003021 3153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:57:56.005533 3153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:57:56.006868 3153 hybrid_clock.cc:648] HybridClock initialized: now 1750946276006830 us; error 49 us; skew 500 ppm
I20250626 13:57:56.007643 3153 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:57:56.017257 3153 webserver.cc:469] Webserver started at http://127.2.107.126:44237/ using document root <none> and password file <none>
I20250626 13:57:56.018206 3153 fs_manager.cc:362] Metadata directory not provided
I20250626 13:57:56.018431 3153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:57:56.026113 3153 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20250626 13:57:56.031096 3182 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:57:56.032153 3153 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20250626 13:57:56.032459 3153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d9b0508ef33e44f0850ad667b89cde2b"
format_stamp: "Formatted at 2025-06-26 13:57:46 on dist-test-slave-7p9x"
I20250626 13:57:56.034340 3153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:57:56.052568 2800 raft_consensus.cc:2466] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6abf8cb57c24c17871f9fc9db82be83 in term 2.
I20250626 13:57:56.053268 3169 raft_consensus.cc:2747] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 13:57:56.054003 2603 leader_election.cc:304] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1a2182284f334697becb5cb14da3c22c, a6abf8cb57c24c17871f9fc9db82be83; no voters: aa223e2d12164378942ccb1f6d040b86
I20250626 13:57:56.055022 3170 raft_consensus.cc:2802] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 13:57:56.056835 3170 raft_consensus.cc:695] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [term 2 LEADER]: Becoming Leader. State: Replica: a6abf8cb57c24c17871f9fc9db82be83, State: Running, Role: LEADER
I20250626 13:57:56.057780 3170 consensus_queue.cc:237] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 42753 } } peers { permanent_uuid: "aa223e2d12164378942ccb1f6d040b86" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 36545 } } peers { permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 } }
I20250626 13:57:56.083655 3153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:57:56.085071 3153 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:57:56.085516 3153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:57:56.152776 3153 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.126:41849
I20250626 13:57:56.152804 3235 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.126:41849 every 8 connection(s)
I20250626 13:57:56.155627 3153 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250626 13:57:56.160863 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 3153
I20250626 13:57:56.161477 2477 kudu-admin-test.cc:735] Forcing unsafe config change on tserver a6abf8cb57c24c17871f9fc9db82be83
I20250626 13:57:56.167448 3236 sys_catalog.cc:263] Verifying existing consensus state
I20250626 13:57:56.173105 3236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b: Bootstrap starting.
I20250626 13:57:56.210279 3236 log.cc:826] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b: Log is configured to *not* fsync() on all Append() calls
I20250626 13:57:56.233448 3236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 13:57:56.234431 3236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b: Bootstrap complete.
I20250626 13:57:56.257746 3236 raft_consensus.cc:357] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b0508ef33e44f0850ad667b89cde2b" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 41849 } }
I20250626 13:57:56.259912 3236 raft_consensus.cc:738] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9b0508ef33e44f0850ad667b89cde2b, State: Initialized, Role: FOLLOWER
I20250626 13:57:56.260668 3236 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b0508ef33e44f0850ad667b89cde2b" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 41849 } }
I20250626 13:57:56.261205 3236 raft_consensus.cc:397] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 13:57:56.261461 3236 raft_consensus.cc:491] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 13:57:56.261741 3236 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b [term 1 FOLLOWER]: Advancing to term 2
I20250626 13:57:56.285830 2845 heartbeater.cc:344] Connected to a master server at 127.2.107.126:41849
I20250626 13:57:56.390965 3236 raft_consensus.cc:513] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b0508ef33e44f0850ad667b89cde2b" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 41849 } }
I20250626 13:57:56.391901 3236 leader_election.cc:304] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9b0508ef33e44f0850ad667b89cde2b; no voters:
I20250626 13:57:56.398747 3236 leader_election.cc:290] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b [CANDIDATE]: Term 2 election: Requested vote from peers
I20250626 13:57:56.399248 3241 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b [term 2 FOLLOWER]: Leader election won for term 2
I20250626 13:57:56.402016 3241 raft_consensus.cc:695] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b [term 2 LEADER]: Becoming Leader. State: Replica: d9b0508ef33e44f0850ad667b89cde2b, State: Running, Role: LEADER
I20250626 13:57:56.402922 3241 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b0508ef33e44f0850ad667b89cde2b" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 41849 } }
I20250626 13:57:56.403318 3236 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b [sys.catalog]: configured and running, proceeding with master startup.
I20250626 13:57:56.413412 3242 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d9b0508ef33e44f0850ad667b89cde2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b0508ef33e44f0850ad667b89cde2b" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 41849 } } }
I20250626 13:57:56.414088 3242 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b [sys.catalog]: This master's current role is: LEADER
I20250626 13:57:56.415860 3243 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9b0508ef33e44f0850ad667b89cde2b. Latest consensus state: current_term: 2 leader_uuid: "d9b0508ef33e44f0850ad667b89cde2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b0508ef33e44f0850ad667b89cde2b" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 41849 } } }
I20250626 13:57:56.416438 3243 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b [sys.catalog]: This master's current role is: LEADER
I20250626 13:57:56.422667 3248 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 13:57:56.433647 3248 catalog_manager.cc:671] Loaded metadata for table TestTable [id=0ea0fa958f5649e09bebe81150542504]
I20250626 13:57:56.441064 3248 tablet_loader.cc:96] loaded metadata for tablet 99aedc8b4f7d4290aea59589aa93c419 (table TestTable [id=0ea0fa958f5649e09bebe81150542504])
I20250626 13:57:56.443678 3248 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 13:57:56.448308 3248 catalog_manager.cc:1261] Loaded cluster ID: d239439210ec4bbdb52eec4d4b0ee585
I20250626 13:57:56.449748 3248 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 13:57:56.457746 3248 catalog_manager.cc:1506] Loading token signing keys...
I20250626 13:57:56.462937 3248 catalog_manager.cc:5966] T 00000000000000000000000000000000 P d9b0508ef33e44f0850ad667b89cde2b: Loaded TSK: 0
I20250626 13:57:56.464541 3248 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250626 13:57:56.501392 3238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:57:56.502005 3238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:57:56.533020 3238 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250626 13:57:56.568398 3111 heartbeater.cc:344] Connected to a master server at 127.2.107.126:41849
I20250626 13:57:56.568920 3111 heartbeater.cc:499] Master 127.2.107.126:41849 was elected leader, sending a full tablet report...
I20250626 13:57:56.574265 3201 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7100218b59a049d48f8431324f5bbfd4" instance_seqno: 1750946273474545) as {username='slave'} at 127.2.107.68:37377; Asking this server to re-register.
I20250626 13:57:56.575681 3111 heartbeater.cc:461] Registering TS with master...
I20250626 13:57:56.576293 3111 heartbeater.cc:507] Master 127.2.107.126:41849 requested a full tablet report, sending...
I20250626 13:57:56.578766 3201 ts_manager.cc:194] Registered new tserver with Master: 7100218b59a049d48f8431324f5bbfd4 (127.2.107.68:40935)
I20250626 13:57:56.623240 2800 raft_consensus.cc:1273] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c [term 2 FOLLOWER]: Refusing update from remote peer a6abf8cb57c24c17871f9fc9db82be83: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 13:57:56.625201 3270 consensus_queue.cc:1035] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250626 13:57:56.652988 3201 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "1a2182284f334697becb5cb14da3c22c" instance_seqno: 1750946269995886) as {username='slave'} at 127.2.107.66:48613; Asking this server to re-register.
I20250626 13:57:56.654594 2845 heartbeater.cc:461] Registering TS with master...
I20250626 13:57:56.655226 2845 heartbeater.cc:507] Master 127.2.107.126:41849 requested a full tablet report, sending...
I20250626 13:57:56.657968 3201 ts_manager.cc:194] Registered new tserver with Master: 1a2182284f334697becb5cb14da3c22c (127.2.107.66:42025)
I20250626 13:57:56.661581 3201 catalog_manager.cc:5582] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c reported cstate change: term changed from 1 to 2, leader changed from aa223e2d12164378942ccb1f6d040b86 (127.2.107.67) to a6abf8cb57c24c17871f9fc9db82be83 (127.2.107.65). New cstate: current_term: 2 leader_uuid: "a6abf8cb57c24c17871f9fc9db82be83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 42753 } } peers { permanent_uuid: "aa223e2d12164378942ccb1f6d040b86" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 36545 } } peers { permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 } } }
W20250626 13:57:56.693373 2603 consensus_peers.cc:487] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 -> Peer aa223e2d12164378942ccb1f6d040b86 (127.2.107.67:36545): Couldn't send request to peer aa223e2d12164378942ccb1f6d040b86. Status: Network error: Client connection negotiation failed: client connection to 127.2.107.67:36545: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 13:57:56.699609 2712 heartbeater.cc:344] Connected to a master server at 127.2.107.126:41849
I20250626 13:57:56.703590 3200 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83" instance_seqno: 1750946268267064) as {username='slave'} at 127.2.107.65:59327; Asking this server to re-register.
I20250626 13:57:56.705371 2712 heartbeater.cc:461] Registering TS with master...
I20250626 13:57:56.706040 2712 heartbeater.cc:507] Master 127.2.107.126:41849 requested a full tablet report, sending...
I20250626 13:57:56.709306 3200 ts_manager.cc:194] Registered new tserver with Master: a6abf8cb57c24c17871f9fc9db82be83 (127.2.107.65:42753)
W20250626 13:57:56.913663 2708 debug-util.cc:398] Leaking SignalData structure 0x7b08000b2c20 after lost signal to thread 2584
W20250626 13:57:56.915261 2708 debug-util.cc:398] Leaking SignalData structure 0x7b08000c6740 after lost signal to thread 2711
W20250626 13:57:57.981650 3265 debug-util.cc:398] Leaking SignalData structure 0x7b0800036040 after lost signal to thread 3238
W20250626 13:57:57.982159 3265 kernel_stack_watchdog.cc:198] Thread 3238 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:641 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250626 13:57:58.003875 3238 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.426s user 0.000s sys 0.002s
W20250626 13:57:58.176915 3238 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.599s user 0.000s sys 0.013s
I20250626 13:57:58.239562 2667 tablet_service.cc:1905] Received UnsafeChangeConfig RPC: dest_uuid: "a6abf8cb57c24c17871f9fc9db82be83"
tablet_id: "99aedc8b4f7d4290aea59589aa93c419"
caller_id: "kudu-tools"
new_config {
peers {
permanent_uuid: "1a2182284f334697becb5cb14da3c22c"
}
peers {
permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83"
}
}
from {username='slave'} at 127.0.0.1:40286
W20250626 13:57:58.241273 2667 raft_consensus.cc:2216] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [term 2 LEADER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 42753 } } peers { permanent_uuid: "aa223e2d12164378942ccb1f6d040b86" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 36545 } } peers { permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 } }NEW CONFIG: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 42753 } } peers { permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 } } unsafe_config_change: true
I20250626 13:57:58.242566 2667 raft_consensus.cc:3053] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 13:57:58.242833 2667 raft_consensus.cc:738] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a6abf8cb57c24c17871f9fc9db82be83, State: Running, Role: LEADER
I20250626 13:57:58.243543 2667 consensus_queue.cc:260] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 42753 } } peers { permanent_uuid: "aa223e2d12164378942ccb1f6d040b86" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 36545 } } peers { permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 } }
I20250626 13:57:58.244807 2667 raft_consensus.cc:3058] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [term 2 FOLLOWER]: Advancing to term 3
I20250626 13:57:59.317801 3294 raft_consensus.cc:491] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a6abf8cb57c24c17871f9fc9db82be83)
I20250626 13:57:59.318346 3294 raft_consensus.cc:513] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 42753 } } peers { permanent_uuid: "aa223e2d12164378942ccb1f6d040b86" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 36545 } } peers { permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 } }
I20250626 13:57:59.319912 3294 leader_election.cc:290] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a6abf8cb57c24c17871f9fc9db82be83 (127.2.107.65:42753), aa223e2d12164378942ccb1f6d040b86 (127.2.107.67:36545)
I20250626 13:57:59.320580 2667 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "99aedc8b4f7d4290aea59589aa93c419" candidate_uuid: "1a2182284f334697becb5cb14da3c22c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a6abf8cb57c24c17871f9fc9db82be83" is_pre_election: true
W20250626 13:57:59.324725 2736 leader_election.cc:336] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aa223e2d12164378942ccb1f6d040b86 (127.2.107.67:36545): Network error: Client connection negotiation failed: client connection to 127.2.107.67:36545: connect: Connection refused (error 111)
I20250626 13:57:59.325037 2736 leader_election.cc:304] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1a2182284f334697becb5cb14da3c22c; no voters: a6abf8cb57c24c17871f9fc9db82be83, aa223e2d12164378942ccb1f6d040b86
I20250626 13:57:59.325615 3294 raft_consensus.cc:2747] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 13:57:59.753526 3297 raft_consensus.cc:491] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools)
I20250626 13:57:59.753882 3297 raft_consensus.cc:513] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 42753 } } peers { permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 } } unsafe_config_change: true
I20250626 13:57:59.754889 3297 leader_election.cc:290] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1a2182284f334697becb5cb14da3c22c (127.2.107.66:42025)
I20250626 13:57:59.756050 2800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "99aedc8b4f7d4290aea59589aa93c419" candidate_uuid: "a6abf8cb57c24c17871f9fc9db82be83" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1a2182284f334697becb5cb14da3c22c" is_pre_election: true
I20250626 13:57:59.756579 2800 raft_consensus.cc:2466] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6abf8cb57c24c17871f9fc9db82be83 in term 2.
I20250626 13:57:59.757550 2603 leader_election.cc:304] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1a2182284f334697becb5cb14da3c22c, a6abf8cb57c24c17871f9fc9db82be83; no voters:
I20250626 13:57:59.758082 3297 raft_consensus.cc:2802] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 13:57:59.758311 3297 raft_consensus.cc:491] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [term 3 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools)
I20250626 13:57:59.758553 3297 raft_consensus.cc:3058] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [term 3 FOLLOWER]: Advancing to term 4
I20250626 13:57:59.762637 3297 raft_consensus.cc:513] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 42753 } } peers { permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 } } unsafe_config_change: true
I20250626 13:57:59.763620 3297 leader_election.cc:290] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [CANDIDATE]: Term 4 election: Requested vote from peers 1a2182284f334697becb5cb14da3c22c (127.2.107.66:42025)
I20250626 13:57:59.764436 2800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "99aedc8b4f7d4290aea59589aa93c419" candidate_uuid: "a6abf8cb57c24c17871f9fc9db82be83" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1a2182284f334697becb5cb14da3c22c"
I20250626 13:57:59.764882 2800 raft_consensus.cc:3058] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c [term 2 FOLLOWER]: Advancing to term 4
I20250626 13:57:59.769109 2800 raft_consensus.cc:2466] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6abf8cb57c24c17871f9fc9db82be83 in term 4.
I20250626 13:57:59.769902 2603 leader_election.cc:304] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1a2182284f334697becb5cb14da3c22c, a6abf8cb57c24c17871f9fc9db82be83; no voters:
I20250626 13:57:59.770409 3297 raft_consensus.cc:2802] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 13:57:59.771108 3297 raft_consensus.cc:695] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [term 4 LEADER]: Becoming Leader. State: Replica: a6abf8cb57c24c17871f9fc9db82be83, State: Running, Role: LEADER
I20250626 13:57:59.771667 3297 consensus_queue.cc:237] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 42753 } } peers { permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 } } unsafe_config_change: true
I20250626 13:57:59.776880 3200 catalog_manager.cc:5582] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 reported cstate change: term changed from 2 to 4, now has a pending config: VOTER a6abf8cb57c24c17871f9fc9db82be83 (127.2.107.65), VOTER 1a2182284f334697becb5cb14da3c22c (127.2.107.66). New cstate: current_term: 4 leader_uuid: "a6abf8cb57c24c17871f9fc9db82be83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 42753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa223e2d12164378942ccb1f6d040b86" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 36545 } } peers { permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 42753 } } peers { permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 } } unsafe_config_change: true }
I20250626 13:58:00.381549 2800 raft_consensus.cc:1273] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c [term 4 FOLLOWER]: Refusing update from remote peer a6abf8cb57c24c17871f9fc9db82be83: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250626 13:58:00.382730 3304 consensus_queue.cc:1035] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 13:58:00.390898 3307 raft_consensus.cc:2953] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [term 4 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER aa223e2d12164378942ccb1f6d040b86 (127.2.107.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 42753 } } peers { permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 } } unsafe_config_change: true }
I20250626 13:58:00.392251 2800 raft_consensus.cc:2953] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c [term 4 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER aa223e2d12164378942ccb1f6d040b86 (127.2.107.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 42753 } } peers { permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 } } unsafe_config_change: true }
I20250626 13:58:00.400293 3200 catalog_manager.cc:5582] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 reported cstate change: config changed from index -1 to 3, VOTER aa223e2d12164378942ccb1f6d040b86 (127.2.107.67) evicted, no longer has a pending config: VOTER a6abf8cb57c24c17871f9fc9db82be83 (127.2.107.65), VOTER 1a2182284f334697becb5cb14da3c22c (127.2.107.66). New cstate: current_term: 4 leader_uuid: "a6abf8cb57c24c17871f9fc9db82be83" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 42753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
W20250626 13:58:00.409657 3200 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet 99aedc8b4f7d4290aea59589aa93c419 on TS aa223e2d12164378942ccb1f6d040b86: Not found: failed to reset TS proxy: Could not find TS for UUID aa223e2d12164378942ccb1f6d040b86
I20250626 13:58:00.425302 2667 consensus_queue.cc:237] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 42753 } } peers { permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 } } peers { permanent_uuid: "7100218b59a049d48f8431324f5bbfd4" member_type: NON_VOTER last_known_addr { host: "127.2.107.68" port: 40935 } attrs { promote: true } } unsafe_config_change: true
I20250626 13:58:00.429607 2800 raft_consensus.cc:1273] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c [term 4 FOLLOWER]: Refusing update from remote peer a6abf8cb57c24c17871f9fc9db82be83: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250626 13:58:00.430931 3307 consensus_queue.cc:1035] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 13:58:00.435901 3304 raft_consensus.cc:2953] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER 7100218b59a049d48f8431324f5bbfd4 (127.2.107.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 42753 } } peers { permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 } } peers { permanent_uuid: "7100218b59a049d48f8431324f5bbfd4" member_type: NON_VOTER last_known_addr { host: "127.2.107.68" port: 40935 } attrs { promote: true } } unsafe_config_change: true }
I20250626 13:58:00.437784 2800 raft_consensus.cc:2953] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER 7100218b59a049d48f8431324f5bbfd4 (127.2.107.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 42753 } } peers { permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 } } peers { permanent_uuid: "7100218b59a049d48f8431324f5bbfd4" member_type: NON_VOTER last_known_addr { host: "127.2.107.68" port: 40935 } attrs { promote: true } } unsafe_config_change: true }
W20250626 13:58:00.441105 2600 consensus_peers.cc:487] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 -> Peer 7100218b59a049d48f8431324f5bbfd4 (127.2.107.68:40935): Couldn't send request to peer 7100218b59a049d48f8431324f5bbfd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99aedc8b4f7d4290aea59589aa93c419. This is attempt 1: this message will repeat every 5th retry.
I20250626 13:58:00.446789 3200 catalog_manager.cc:5582] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c reported cstate change: config changed from index 3 to 5, NON_VOTER 7100218b59a049d48f8431324f5bbfd4 (127.2.107.68) added. New cstate: current_term: 4 leader_uuid: "a6abf8cb57c24c17871f9fc9db82be83" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 42753 } } peers { permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 } } peers { permanent_uuid: "7100218b59a049d48f8431324f5bbfd4" member_type: NON_VOTER last_known_addr { host: "127.2.107.68" port: 40935 } attrs { promote: true } } unsafe_config_change: true }
I20250626 13:58:00.448151 3185 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 99aedc8b4f7d4290aea59589aa93c419 with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250626 13:58:00.462050 3186 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 99aedc8b4f7d4290aea59589aa93c419 on TS aa223e2d12164378942ccb1f6d040b86 failed: Not found: failed to reset TS proxy: Could not find TS for UUID aa223e2d12164378942ccb1f6d040b86
I20250626 13:58:01.004434 3319 ts_tablet_manager.cc:927] T 99aedc8b4f7d4290aea59589aa93c419 P 7100218b59a049d48f8431324f5bbfd4: Initiating tablet copy from peer a6abf8cb57c24c17871f9fc9db82be83 (127.2.107.65:42753)
I20250626 13:58:01.006378 3319 tablet_copy_client.cc:323] T 99aedc8b4f7d4290aea59589aa93c419 P 7100218b59a049d48f8431324f5bbfd4: tablet copy: Beginning tablet copy session from remote peer at address 127.2.107.65:42753
I20250626 13:58:01.016714 2687 tablet_copy_service.cc:140] P a6abf8cb57c24c17871f9fc9db82be83: Received BeginTabletCopySession request for tablet 99aedc8b4f7d4290aea59589aa93c419 from peer 7100218b59a049d48f8431324f5bbfd4 ({username='slave'} at 127.2.107.68:43087)
I20250626 13:58:01.017161 2687 tablet_copy_service.cc:161] P a6abf8cb57c24c17871f9fc9db82be83: Beginning new tablet copy session on tablet 99aedc8b4f7d4290aea59589aa93c419 from peer 7100218b59a049d48f8431324f5bbfd4 at {username='slave'} at 127.2.107.68:43087: session id = 7100218b59a049d48f8431324f5bbfd4-99aedc8b4f7d4290aea59589aa93c419
I20250626 13:58:01.021826 2687 tablet_copy_source_session.cc:215] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 13:58:01.026399 3319 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99aedc8b4f7d4290aea59589aa93c419. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:58:01.050612 3319 tablet_copy_client.cc:806] T 99aedc8b4f7d4290aea59589aa93c419 P 7100218b59a049d48f8431324f5bbfd4: tablet copy: Starting download of 0 data blocks...
I20250626 13:58:01.051194 3319 tablet_copy_client.cc:670] T 99aedc8b4f7d4290aea59589aa93c419 P 7100218b59a049d48f8431324f5bbfd4: tablet copy: Starting download of 1 WAL segments...
I20250626 13:58:01.054551 3319 tablet_copy_client.cc:538] T 99aedc8b4f7d4290aea59589aa93c419 P 7100218b59a049d48f8431324f5bbfd4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 13:58:01.062659 3319 tablet_bootstrap.cc:492] T 99aedc8b4f7d4290aea59589aa93c419 P 7100218b59a049d48f8431324f5bbfd4: Bootstrap starting.
I20250626 13:58:01.075366 3319 log.cc:826] T 99aedc8b4f7d4290aea59589aa93c419 P 7100218b59a049d48f8431324f5bbfd4: Log is configured to *not* fsync() on all Append() calls
I20250626 13:58:01.085716 3319 tablet_bootstrap.cc:492] T 99aedc8b4f7d4290aea59589aa93c419 P 7100218b59a049d48f8431324f5bbfd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 13:58:01.086303 3319 tablet_bootstrap.cc:492] T 99aedc8b4f7d4290aea59589aa93c419 P 7100218b59a049d48f8431324f5bbfd4: Bootstrap complete.
I20250626 13:58:01.086798 3319 ts_tablet_manager.cc:1397] T 99aedc8b4f7d4290aea59589aa93c419 P 7100218b59a049d48f8431324f5bbfd4: Time spent bootstrapping tablet: real 0.024s user 0.022s sys 0.001s
I20250626 13:58:01.102808 3319 raft_consensus.cc:357] T 99aedc8b4f7d4290aea59589aa93c419 P 7100218b59a049d48f8431324f5bbfd4 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 42753 } } peers { permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 } } peers { permanent_uuid: "7100218b59a049d48f8431324f5bbfd4" member_type: NON_VOTER last_known_addr { host: "127.2.107.68" port: 40935 } attrs { promote: true } } unsafe_config_change: true
I20250626 13:58:01.103559 3319 raft_consensus.cc:738] T 99aedc8b4f7d4290aea59589aa93c419 P 7100218b59a049d48f8431324f5bbfd4 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 7100218b59a049d48f8431324f5bbfd4, State: Initialized, Role: LEARNER
I20250626 13:58:01.104094 3319 consensus_queue.cc:260] T 99aedc8b4f7d4290aea59589aa93c419 P 7100218b59a049d48f8431324f5bbfd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 42753 } } peers { permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 } } peers { permanent_uuid: "7100218b59a049d48f8431324f5bbfd4" member_type: NON_VOTER last_known_addr { host: "127.2.107.68" port: 40935 } attrs { promote: true } } unsafe_config_change: true
I20250626 13:58:01.107262 3319 ts_tablet_manager.cc:1428] T 99aedc8b4f7d4290aea59589aa93c419 P 7100218b59a049d48f8431324f5bbfd4: Time spent starting tablet: real 0.020s user 0.021s sys 0.000s
I20250626 13:58:01.108877 2687 tablet_copy_service.cc:342] P a6abf8cb57c24c17871f9fc9db82be83: Request end of tablet copy session 7100218b59a049d48f8431324f5bbfd4-99aedc8b4f7d4290aea59589aa93c419 received from {username='slave'} at 127.2.107.68:43087
I20250626 13:58:01.109228 2687 tablet_copy_service.cc:434] P a6abf8cb57c24c17871f9fc9db82be83: ending tablet copy session 7100218b59a049d48f8431324f5bbfd4-99aedc8b4f7d4290aea59589aa93c419 on tablet 99aedc8b4f7d4290aea59589aa93c419 with peer 7100218b59a049d48f8431324f5bbfd4
I20250626 13:58:01.544322 3066 raft_consensus.cc:1215] T 99aedc8b4f7d4290aea59589aa93c419 P 7100218b59a049d48f8431324f5bbfd4 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.4->[4.5-4.5] Dedup: 4.5->[]
W20250626 13:58:01.628651 3186 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 99aedc8b4f7d4290aea59589aa93c419 on TS aa223e2d12164378942ccb1f6d040b86 failed: Not found: failed to reset TS proxy: Could not find TS for UUID aa223e2d12164378942ccb1f6d040b86
I20250626 13:58:02.008124 3326 raft_consensus.cc:1062] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83: attempting to promote NON_VOTER 7100218b59a049d48f8431324f5bbfd4 to VOTER
I20250626 13:58:02.009687 3326 consensus_queue.cc:237] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 42753 } } peers { permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 } } peers { permanent_uuid: "7100218b59a049d48f8431324f5bbfd4" member_type: VOTER last_known_addr { host: "127.2.107.68" port: 40935 } attrs { promote: false } } unsafe_config_change: true
I20250626 13:58:02.013885 2800 raft_consensus.cc:1273] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c [term 4 FOLLOWER]: Refusing update from remote peer a6abf8cb57c24c17871f9fc9db82be83: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250626 13:58:02.013949 3066 raft_consensus.cc:1273] T 99aedc8b4f7d4290aea59589aa93c419 P 7100218b59a049d48f8431324f5bbfd4 [term 4 LEARNER]: Refusing update from remote peer a6abf8cb57c24c17871f9fc9db82be83: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250626 13:58:02.015000 3325 consensus_queue.cc:1035] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250626 13:58:02.015578 3326 consensus_queue.cc:1035] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7100218b59a049d48f8431324f5bbfd4" member_type: VOTER last_known_addr { host: "127.2.107.68" port: 40935 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250626 13:58:02.021025 3325 raft_consensus.cc:2953] T 99aedc8b4f7d4290aea59589aa93c419 P a6abf8cb57c24c17871f9fc9db82be83 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 7100218b59a049d48f8431324f5bbfd4 (127.2.107.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 42753 } } peers { permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 } } peers { permanent_uuid: "7100218b59a049d48f8431324f5bbfd4" member_type: VOTER last_known_addr { host: "127.2.107.68" port: 40935 } attrs { promote: false } } unsafe_config_change: true }
I20250626 13:58:02.022593 2800 raft_consensus.cc:2953] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 7100218b59a049d48f8431324f5bbfd4 (127.2.107.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 42753 } } peers { permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 } } peers { permanent_uuid: "7100218b59a049d48f8431324f5bbfd4" member_type: VOTER last_known_addr { host: "127.2.107.68" port: 40935 } attrs { promote: false } } unsafe_config_change: true }
I20250626 13:58:02.023587 3066 raft_consensus.cc:2953] T 99aedc8b4f7d4290aea59589aa93c419 P 7100218b59a049d48f8431324f5bbfd4 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 7100218b59a049d48f8431324f5bbfd4 (127.2.107.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 42753 } } peers { permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 } } peers { permanent_uuid: "7100218b59a049d48f8431324f5bbfd4" member_type: VOTER last_known_addr { host: "127.2.107.68" port: 40935 } attrs { promote: false } } unsafe_config_change: true }
I20250626 13:58:02.034405 3200 catalog_manager.cc:5582] T 99aedc8b4f7d4290aea59589aa93c419 P 1a2182284f334697becb5cb14da3c22c reported cstate change: config changed from index 5 to 6, 7100218b59a049d48f8431324f5bbfd4 (127.2.107.68) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "a6abf8cb57c24c17871f9fc9db82be83" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 42753 } } peers { permanent_uuid: "1a2182284f334697becb5cb14da3c22c" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 42025 } } peers { permanent_uuid: "7100218b59a049d48f8431324f5bbfd4" member_type: VOTER last_known_addr { host: "127.2.107.68" port: 40935 } attrs { promote: false } } unsafe_config_change: true }
I20250626 13:58:02.036084 2477 kudu-admin-test.cc:751] Waiting for Master to see new config...
I20250626 13:58:02.055051 2477 kudu-admin-test.cc:756] Tablet locations:
tablet_locations {
tablet_id: "99aedc8b4f7d4290aea59589aa93c419"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: LEADER
}
interned_replicas {
ts_info_idx: 1
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "a6abf8cb57c24c17871f9fc9db82be83"
rpc_addresses {
host: "127.2.107.65"
port: 42753
}
}
ts_infos {
permanent_uuid: "1a2182284f334697becb5cb14da3c22c"
rpc_addresses {
host: "127.2.107.66"
port: 42025
}
}
ts_infos {
permanent_uuid: "7100218b59a049d48f8431324f5bbfd4"
rpc_addresses {
host: "127.2.107.68"
port: 40935
}
}
I20250626 13:58:02.057616 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 2583
I20250626 13:58:02.083812 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 2716
I20250626 13:58:02.102403 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 2982
I20250626 13:58:02.120605 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 3153
2025-06-26T13:58:02Z chronyd exiting
[ OK ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes (18605 ms)
[ RUN ] AdminCliTest.TestGracefulSpecificLeaderStepDown
I20250626 13:58:02.180552 2477 test_util.cc:276] Using random seed: -215248991
I20250626 13:58:02.187212 2477 ts_itest-base.cc:115] Starting cluster with:
I20250626 13:58:02.187374 2477 ts_itest-base.cc:116] --------------
I20250626 13:58:02.187486 2477 ts_itest-base.cc:117] 3 tablet servers
I20250626 13:58:02.187605 2477 ts_itest-base.cc:118] 3 replicas per TS
I20250626 13:58:02.187708 2477 ts_itest-base.cc:119] --------------
2025-06-26T13:58:02Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-26T13:58:02Z Disabled control of system clock
I20250626 13:58:02.221031 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.107.126:37415
--webserver_interface=127.2.107.126
--webserver_port=0
--builtin_ntp_servers=127.2.107.84:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.107.126:37415
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250626 13:58:02.512398 3345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:58:02.512943 3345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:58:02.513418 3345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:58:02.544935 3345 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 13:58:02.545241 3345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:58:02.545464 3345 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 13:58:02.545663 3345 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 13:58:02.580967 3345 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.2.107.126:37415
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.107.126:37415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.107.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:58:02.582309 3345 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:58:02.583882 3345 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:58:02.598232 3352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:58:02.598276 3351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:58:02.598371 3354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:58:03.700784 3353 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 13:58:03.700871 3345 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 13:58:03.704962 3345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:58:03.707836 3345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:58:03.709232 3345 hybrid_clock.cc:648] HybridClock initialized: now 1750946283709206 us; error 44 us; skew 500 ppm
I20250626 13:58:03.710045 3345 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:58:03.716317 3345 webserver.cc:469] Webserver started at http://127.2.107.126:37567/ using document root <none> and password file <none>
I20250626 13:58:03.717233 3345 fs_manager.cc:362] Metadata directory not provided
I20250626 13:58:03.717425 3345 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:58:03.717839 3345 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 13:58:03.722182 3345 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "a865e8dffabf45fd8087fc39dfc0f47f"
format_stamp: "Formatted at 2025-06-26 13:58:03 on dist-test-slave-7p9x"
I20250626 13:58:03.723202 3345 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "a865e8dffabf45fd8087fc39dfc0f47f"
format_stamp: "Formatted at 2025-06-26 13:58:03 on dist-test-slave-7p9x"
I20250626 13:58:03.730006 3345 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250626 13:58:03.735394 3362 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:03.736377 3345 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250626 13:58:03.736701 3345 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
uuid: "a865e8dffabf45fd8087fc39dfc0f47f"
format_stamp: "Formatted at 2025-06-26 13:58:03 on dist-test-slave-7p9x"
I20250626 13:58:03.737030 3345 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:58:03.797700 3345 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:58:03.799162 3345 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:58:03.799566 3345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:58:03.866604 3345 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.126:37415
I20250626 13:58:03.866638 3413 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.126:37415 every 8 connection(s)
I20250626 13:58:03.869791 3345 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250626 13:58:03.873265 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 3345
I20250626 13:58:03.873906 2477 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250626 13:58:03.875393 3414 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:58:03.899124 3414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a865e8dffabf45fd8087fc39dfc0f47f: Bootstrap starting.
I20250626 13:58:03.904527 3414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a865e8dffabf45fd8087fc39dfc0f47f: Neither blocks nor log segments found. Creating new log.
I20250626 13:58:03.906208 3414 log.cc:826] T 00000000000000000000000000000000 P a865e8dffabf45fd8087fc39dfc0f47f: Log is configured to *not* fsync() on all Append() calls
I20250626 13:58:03.910707 3414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a865e8dffabf45fd8087fc39dfc0f47f: No bootstrap required, opened a new log
I20250626 13:58:03.927660 3414 raft_consensus.cc:357] T 00000000000000000000000000000000 P a865e8dffabf45fd8087fc39dfc0f47f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a865e8dffabf45fd8087fc39dfc0f47f" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 37415 } }
I20250626 13:58:03.928315 3414 raft_consensus.cc:383] T 00000000000000000000000000000000 P a865e8dffabf45fd8087fc39dfc0f47f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 13:58:03.928506 3414 raft_consensus.cc:738] T 00000000000000000000000000000000 P a865e8dffabf45fd8087fc39dfc0f47f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a865e8dffabf45fd8087fc39dfc0f47f, State: Initialized, Role: FOLLOWER
I20250626 13:58:03.929180 3414 consensus_queue.cc:260] T 00000000000000000000000000000000 P a865e8dffabf45fd8087fc39dfc0f47f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a865e8dffabf45fd8087fc39dfc0f47f" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 37415 } }
I20250626 13:58:03.929669 3414 raft_consensus.cc:397] T 00000000000000000000000000000000 P a865e8dffabf45fd8087fc39dfc0f47f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 13:58:03.929895 3414 raft_consensus.cc:491] T 00000000000000000000000000000000 P a865e8dffabf45fd8087fc39dfc0f47f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 13:58:03.930148 3414 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a865e8dffabf45fd8087fc39dfc0f47f [term 0 FOLLOWER]: Advancing to term 1
I20250626 13:58:03.968326 3414 raft_consensus.cc:513] T 00000000000000000000000000000000 P a865e8dffabf45fd8087fc39dfc0f47f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a865e8dffabf45fd8087fc39dfc0f47f" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 37415 } }
I20250626 13:58:03.969375 3414 leader_election.cc:304] T 00000000000000000000000000000000 P a865e8dffabf45fd8087fc39dfc0f47f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a865e8dffabf45fd8087fc39dfc0f47f; no voters:
I20250626 13:58:03.971240 3414 leader_election.cc:290] T 00000000000000000000000000000000 P a865e8dffabf45fd8087fc39dfc0f47f [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 13:58:03.971983 3419 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a865e8dffabf45fd8087fc39dfc0f47f [term 1 FOLLOWER]: Leader election won for term 1
I20250626 13:58:03.973910 3419 raft_consensus.cc:695] T 00000000000000000000000000000000 P a865e8dffabf45fd8087fc39dfc0f47f [term 1 LEADER]: Becoming Leader. State: Replica: a865e8dffabf45fd8087fc39dfc0f47f, State: Running, Role: LEADER
I20250626 13:58:03.974684 3419 consensus_queue.cc:237] T 00000000000000000000000000000000 P a865e8dffabf45fd8087fc39dfc0f47f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a865e8dffabf45fd8087fc39dfc0f47f" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 37415 } }
I20250626 13:58:03.975726 3414 sys_catalog.cc:564] T 00000000000000000000000000000000 P a865e8dffabf45fd8087fc39dfc0f47f [sys.catalog]: configured and running, proceeding with master startup.
I20250626 13:58:03.983464 3420 sys_catalog.cc:455] T 00000000000000000000000000000000 P a865e8dffabf45fd8087fc39dfc0f47f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a865e8dffabf45fd8087fc39dfc0f47f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a865e8dffabf45fd8087fc39dfc0f47f" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 37415 } } }
I20250626 13:58:03.984386 3420 sys_catalog.cc:458] T 00000000000000000000000000000000 P a865e8dffabf45fd8087fc39dfc0f47f [sys.catalog]: This master's current role is: LEADER
I20250626 13:58:03.986003 3421 sys_catalog.cc:455] T 00000000000000000000000000000000 P a865e8dffabf45fd8087fc39dfc0f47f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a865e8dffabf45fd8087fc39dfc0f47f. Latest consensus state: current_term: 1 leader_uuid: "a865e8dffabf45fd8087fc39dfc0f47f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a865e8dffabf45fd8087fc39dfc0f47f" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 37415 } } }
I20250626 13:58:03.986685 3421 sys_catalog.cc:458] T 00000000000000000000000000000000 P a865e8dffabf45fd8087fc39dfc0f47f [sys.catalog]: This master's current role is: LEADER
I20250626 13:58:03.988154 3426 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 13:58:03.998770 3426 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 13:58:04.014940 3426 catalog_manager.cc:1349] Generated new cluster ID: 1e0c6aad5bbb434d990c699f5fa461ff
I20250626 13:58:04.015188 3426 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 13:58:04.032518 3426 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 13:58:04.034466 3426 catalog_manager.cc:1506] Loading token signing keys...
I20250626 13:58:04.051522 3426 catalog_manager.cc:5955] T 00000000000000000000000000000000 P a865e8dffabf45fd8087fc39dfc0f47f: Generated new TSK 0
I20250626 13:58:04.052553 3426 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 13:58:04.070225 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.107.65:0
--local_ip_for_outbound_sockets=127.2.107.65
--webserver_interface=127.2.107.65
--webserver_port=0
--tserver_master_addrs=127.2.107.126:37415
--builtin_ntp_servers=127.2.107.84:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250626 13:58:04.365080 3438 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250626 13:58:04.365751 3438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:58:04.366004 3438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:58:04.366487 3438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:58:04.398980 3438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:58:04.400221 3438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.107.65
I20250626 13:58:04.435628 3438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.107.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.107.65
--webserver_port=0
--tserver_master_addrs=127.2.107.126:37415
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.107.65
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:58:04.436944 3438 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:58:04.438601 3438 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:58:04.455044 3445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:58:04.455032 3447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:58:04.455032 3444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:58:05.879038 3446 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1421 milliseconds
I20250626 13:58:05.879149 3438 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 13:58:05.880263 3438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:58:05.882800 3438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:58:05.884209 3438 hybrid_clock.cc:648] HybridClock initialized: now 1750946285884184 us; error 46 us; skew 500 ppm
I20250626 13:58:05.884958 3438 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:58:05.891259 3438 webserver.cc:469] Webserver started at http://127.2.107.65:37565/ using document root <none> and password file <none>
I20250626 13:58:05.892195 3438 fs_manager.cc:362] Metadata directory not provided
I20250626 13:58:05.892380 3438 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:58:05.892779 3438 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 13:58:05.897033 3438 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "02552a915508491b9b9d484df84d1d43"
format_stamp: "Formatted at 2025-06-26 13:58:05 on dist-test-slave-7p9x"
I20250626 13:58:05.898077 3438 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "02552a915508491b9b9d484df84d1d43"
format_stamp: "Formatted at 2025-06-26 13:58:05 on dist-test-slave-7p9x"
I20250626 13:58:05.905155 3438 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250626 13:58:05.910516 3454 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:05.911535 3438 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250626 13:58:05.911847 3438 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "02552a915508491b9b9d484df84d1d43"
format_stamp: "Formatted at 2025-06-26 13:58:05 on dist-test-slave-7p9x"
I20250626 13:58:05.912182 3438 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:58:05.963515 3438 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:58:05.964954 3438 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:58:05.965395 3438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:58:05.967818 3438 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 13:58:05.971858 3438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 13:58:05.972070 3438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:05.972306 3438 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 13:58:05.972466 3438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:06.095783 3438 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.65:46615
I20250626 13:58:06.095877 3566 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.65:46615 every 8 connection(s)
I20250626 13:58:06.098261 3438 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250626 13:58:06.103766 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 3438
I20250626 13:58:06.104138 2477 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250626 13:58:06.109871 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.107.66:0
--local_ip_for_outbound_sockets=127.2.107.66
--webserver_interface=127.2.107.66
--webserver_port=0
--tserver_master_addrs=127.2.107.126:37415
--builtin_ntp_servers=127.2.107.84:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250626 13:58:06.118700 3567 heartbeater.cc:344] Connected to a master server at 127.2.107.126:37415
I20250626 13:58:06.119117 3567 heartbeater.cc:461] Registering TS with master...
I20250626 13:58:06.120096 3567 heartbeater.cc:507] Master 127.2.107.126:37415 requested a full tablet report, sending...
I20250626 13:58:06.122515 3379 ts_manager.cc:194] Registered new tserver with Master: 02552a915508491b9b9d484df84d1d43 (127.2.107.65:46615)
I20250626 13:58:06.124459 3379 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.107.65:42781
W20250626 13:58:06.392514 3571 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250626 13:58:06.393127 3571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:58:06.393411 3571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:58:06.393891 3571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:58:06.424206 3571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:58:06.425052 3571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.107.66
I20250626 13:58:06.462092 3571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.107.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.107.66
--webserver_port=0
--tserver_master_addrs=127.2.107.126:37415
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.107.66
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:58:06.463371 3571 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:58:06.465225 3571 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:58:06.479766 3578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:58:07.127861 3567 heartbeater.cc:499] Master 127.2.107.126:37415 was elected leader, sending a full tablet report...
W20250626 13:58:06.480324 3577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:58:06.481515 3580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:58:07.548065 3579 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 13:58:07.548193 3571 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 13:58:07.551990 3571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:58:07.554625 3571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:58:07.556025 3571 hybrid_clock.cc:648] HybridClock initialized: now 1750946287555975 us; error 72 us; skew 500 ppm
I20250626 13:58:07.556771 3571 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:58:07.563014 3571 webserver.cc:469] Webserver started at http://127.2.107.66:37149/ using document root <none> and password file <none>
I20250626 13:58:07.563874 3571 fs_manager.cc:362] Metadata directory not provided
I20250626 13:58:07.564064 3571 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:58:07.564512 3571 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 13:58:07.568786 3571 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "4495e9f7ef3741dca3d605b513944883"
format_stamp: "Formatted at 2025-06-26 13:58:07 on dist-test-slave-7p9x"
I20250626 13:58:07.569860 3571 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "4495e9f7ef3741dca3d605b513944883"
format_stamp: "Formatted at 2025-06-26 13:58:07 on dist-test-slave-7p9x"
I20250626 13:58:07.576573 3571 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250626 13:58:07.581940 3587 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:07.582958 3571 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20250626 13:58:07.583274 3571 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "4495e9f7ef3741dca3d605b513944883"
format_stamp: "Formatted at 2025-06-26 13:58:07 on dist-test-slave-7p9x"
I20250626 13:58:07.583583 3571 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:58:07.648155 3571 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:58:07.649607 3571 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:58:07.650018 3571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:58:07.652459 3571 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 13:58:07.656276 3571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 13:58:07.656469 3571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:07.656713 3571 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 13:58:07.656860 3571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:07.781368 3571 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.66:37143
I20250626 13:58:07.781455 3699 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.66:37143 every 8 connection(s)
I20250626 13:58:07.783864 3571 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250626 13:58:07.792630 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 3571
I20250626 13:58:07.793076 2477 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250626 13:58:07.799482 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.107.67:0
--local_ip_for_outbound_sockets=127.2.107.67
--webserver_interface=127.2.107.67
--webserver_port=0
--tserver_master_addrs=127.2.107.126:37415
--builtin_ntp_servers=127.2.107.84:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250626 13:58:07.804849 3700 heartbeater.cc:344] Connected to a master server at 127.2.107.126:37415
I20250626 13:58:07.805279 3700 heartbeater.cc:461] Registering TS with master...
I20250626 13:58:07.806250 3700 heartbeater.cc:507] Master 127.2.107.126:37415 requested a full tablet report, sending...
I20250626 13:58:07.808395 3379 ts_manager.cc:194] Registered new tserver with Master: 4495e9f7ef3741dca3d605b513944883 (127.2.107.66:37143)
I20250626 13:58:07.809626 3379 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.107.66:57879
W20250626 13:58:08.091540 3704 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250626 13:58:08.092136 3704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:58:08.092382 3704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:58:08.092845 3704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:58:08.123247 3704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:58:08.124099 3704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.107.67
I20250626 13:58:08.158725 3704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.107.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.107.67
--webserver_port=0
--tserver_master_addrs=127.2.107.126:37415
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.107.67
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:58:08.160068 3704 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:58:08.161715 3704 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:58:08.177374 3711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:58:08.812927 3700 heartbeater.cc:499] Master 127.2.107.126:37415 was elected leader, sending a full tablet report...
W20250626 13:58:08.178457 3710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:58:08.177560 3713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:58:09.250226 3712 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 13:58:09.250276 3704 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 13:58:09.254375 3704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:58:09.256842 3704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:58:09.258244 3704 hybrid_clock.cc:648] HybridClock initialized: now 1750946289258219 us; error 44 us; skew 500 ppm
I20250626 13:58:09.259052 3704 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:58:09.265041 3704 webserver.cc:469] Webserver started at http://127.2.107.67:36503/ using document root <none> and password file <none>
I20250626 13:58:09.265944 3704 fs_manager.cc:362] Metadata directory not provided
I20250626 13:58:09.266132 3704 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:58:09.266525 3704 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 13:58:09.270918 3704 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "d9be968f292d47bb9aed877458694e57"
format_stamp: "Formatted at 2025-06-26 13:58:09 on dist-test-slave-7p9x"
I20250626 13:58:09.271945 3704 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "d9be968f292d47bb9aed877458694e57"
format_stamp: "Formatted at 2025-06-26 13:58:09 on dist-test-slave-7p9x"
I20250626 13:58:09.279408 3704 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20250626 13:58:09.284696 3720 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:09.285720 3704 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250626 13:58:09.286096 3704 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "d9be968f292d47bb9aed877458694e57"
format_stamp: "Formatted at 2025-06-26 13:58:09 on dist-test-slave-7p9x"
I20250626 13:58:09.286509 3704 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:58:09.351102 3704 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:58:09.352490 3704 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:58:09.352880 3704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:58:09.355404 3704 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 13:58:09.359205 3704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 13:58:09.359395 3704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:09.359601 3704 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 13:58:09.359731 3704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:09.486229 3704 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.67:34269
I20250626 13:58:09.486320 3832 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.67:34269 every 8 connection(s)
I20250626 13:58:09.488967 3704 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250626 13:58:09.493481 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 3704
I20250626 13:58:09.493925 2477 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250626 13:58:09.508141 3833 heartbeater.cc:344] Connected to a master server at 127.2.107.126:37415
I20250626 13:58:09.508536 3833 heartbeater.cc:461] Registering TS with master...
I20250626 13:58:09.509495 3833 heartbeater.cc:507] Master 127.2.107.126:37415 requested a full tablet report, sending...
I20250626 13:58:09.511555 3379 ts_manager.cc:194] Registered new tserver with Master: d9be968f292d47bb9aed877458694e57 (127.2.107.67:34269)
I20250626 13:58:09.513026 3379 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.107.67:34001
I20250626 13:58:09.513377 2477 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 13:58:09.546588 3379 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59006:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250626 13:58:09.565003 3379 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 13:58:09.610435 3768 tablet_service.cc:1468] Processing CreateTablet for tablet 51d2720a898d47e29cc483a7bad784f9 (DEFAULT_TABLE table=TestTable [id=b3a786cfdb114fa4943299d87a54a4a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 13:58:09.610085 3635 tablet_service.cc:1468] Processing CreateTablet for tablet 51d2720a898d47e29cc483a7bad784f9 (DEFAULT_TABLE table=TestTable [id=b3a786cfdb114fa4943299d87a54a4a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 13:58:09.612322 3635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51d2720a898d47e29cc483a7bad784f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:58:09.612994 3768 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51d2720a898d47e29cc483a7bad784f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:58:09.616109 3502 tablet_service.cc:1468] Processing CreateTablet for tablet 51d2720a898d47e29cc483a7bad784f9 (DEFAULT_TABLE table=TestTable [id=b3a786cfdb114fa4943299d87a54a4a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 13:58:09.617897 3502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51d2720a898d47e29cc483a7bad784f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:58:09.633908 3852 tablet_bootstrap.cc:492] T 51d2720a898d47e29cc483a7bad784f9 P 4495e9f7ef3741dca3d605b513944883: Bootstrap starting.
I20250626 13:58:09.639739 3853 tablet_bootstrap.cc:492] T 51d2720a898d47e29cc483a7bad784f9 P d9be968f292d47bb9aed877458694e57: Bootstrap starting.
I20250626 13:58:09.641970 3852 tablet_bootstrap.cc:654] T 51d2720a898d47e29cc483a7bad784f9 P 4495e9f7ef3741dca3d605b513944883: Neither blocks nor log segments found. Creating new log.
I20250626 13:58:09.643728 3854 tablet_bootstrap.cc:492] T 51d2720a898d47e29cc483a7bad784f9 P 02552a915508491b9b9d484df84d1d43: Bootstrap starting.
I20250626 13:58:09.644918 3852 log.cc:826] T 51d2720a898d47e29cc483a7bad784f9 P 4495e9f7ef3741dca3d605b513944883: Log is configured to *not* fsync() on all Append() calls
I20250626 13:58:09.647040 3853 tablet_bootstrap.cc:654] T 51d2720a898d47e29cc483a7bad784f9 P d9be968f292d47bb9aed877458694e57: Neither blocks nor log segments found. Creating new log.
I20250626 13:58:09.649096 3853 log.cc:826] T 51d2720a898d47e29cc483a7bad784f9 P d9be968f292d47bb9aed877458694e57: Log is configured to *not* fsync() on all Append() calls
I20250626 13:58:09.649677 3854 tablet_bootstrap.cc:654] T 51d2720a898d47e29cc483a7bad784f9 P 02552a915508491b9b9d484df84d1d43: Neither blocks nor log segments found. Creating new log.
I20250626 13:58:09.651654 3854 log.cc:826] T 51d2720a898d47e29cc483a7bad784f9 P 02552a915508491b9b9d484df84d1d43: Log is configured to *not* fsync() on all Append() calls
I20250626 13:58:09.652843 3852 tablet_bootstrap.cc:492] T 51d2720a898d47e29cc483a7bad784f9 P 4495e9f7ef3741dca3d605b513944883: No bootstrap required, opened a new log
I20250626 13:58:09.653430 3852 ts_tablet_manager.cc:1397] T 51d2720a898d47e29cc483a7bad784f9 P 4495e9f7ef3741dca3d605b513944883: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.009s
I20250626 13:58:09.654495 3853 tablet_bootstrap.cc:492] T 51d2720a898d47e29cc483a7bad784f9 P d9be968f292d47bb9aed877458694e57: No bootstrap required, opened a new log
I20250626 13:58:09.654974 3853 ts_tablet_manager.cc:1397] T 51d2720a898d47e29cc483a7bad784f9 P d9be968f292d47bb9aed877458694e57: Time spent bootstrapping tablet: real 0.016s user 0.014s sys 0.000s
I20250626 13:58:09.656167 3854 tablet_bootstrap.cc:492] T 51d2720a898d47e29cc483a7bad784f9 P 02552a915508491b9b9d484df84d1d43: No bootstrap required, opened a new log
I20250626 13:58:09.656535 3854 ts_tablet_manager.cc:1397] T 51d2720a898d47e29cc483a7bad784f9 P 02552a915508491b9b9d484df84d1d43: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20250626 13:58:09.674331 3854 raft_consensus.cc:357] T 51d2720a898d47e29cc483a7bad784f9 P 02552a915508491b9b9d484df84d1d43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9be968f292d47bb9aed877458694e57" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 34269 } } peers { permanent_uuid: "4495e9f7ef3741dca3d605b513944883" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 37143 } } peers { permanent_uuid: "02552a915508491b9b9d484df84d1d43" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 46615 } }
I20250626 13:58:09.675019 3854 raft_consensus.cc:738] T 51d2720a898d47e29cc483a7bad784f9 P 02552a915508491b9b9d484df84d1d43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02552a915508491b9b9d484df84d1d43, State: Initialized, Role: FOLLOWER
I20250626 13:58:09.675653 3854 consensus_queue.cc:260] T 51d2720a898d47e29cc483a7bad784f9 P 02552a915508491b9b9d484df84d1d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9be968f292d47bb9aed877458694e57" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 34269 } } peers { permanent_uuid: "4495e9f7ef3741dca3d605b513944883" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 37143 } } peers { permanent_uuid: "02552a915508491b9b9d484df84d1d43" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 46615 } }
I20250626 13:58:09.679502 3852 raft_consensus.cc:357] T 51d2720a898d47e29cc483a7bad784f9 P 4495e9f7ef3741dca3d605b513944883 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9be968f292d47bb9aed877458694e57" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 34269 } } peers { permanent_uuid: "4495e9f7ef3741dca3d605b513944883" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 37143 } } peers { permanent_uuid: "02552a915508491b9b9d484df84d1d43" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 46615 } }
I20250626 13:58:09.680460 3852 raft_consensus.cc:738] T 51d2720a898d47e29cc483a7bad784f9 P 4495e9f7ef3741dca3d605b513944883 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4495e9f7ef3741dca3d605b513944883, State: Initialized, Role: FOLLOWER
I20250626 13:58:09.680670 3853 raft_consensus.cc:357] T 51d2720a898d47e29cc483a7bad784f9 P d9be968f292d47bb9aed877458694e57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9be968f292d47bb9aed877458694e57" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 34269 } } peers { permanent_uuid: "4495e9f7ef3741dca3d605b513944883" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 37143 } } peers { permanent_uuid: "02552a915508491b9b9d484df84d1d43" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 46615 } }
I20250626 13:58:09.680434 3854 ts_tablet_manager.cc:1428] T 51d2720a898d47e29cc483a7bad784f9 P 02552a915508491b9b9d484df84d1d43: Time spent starting tablet: real 0.024s user 0.020s sys 0.004s
I20250626 13:58:09.681278 3852 consensus_queue.cc:260] T 51d2720a898d47e29cc483a7bad784f9 P 4495e9f7ef3741dca3d605b513944883 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9be968f292d47bb9aed877458694e57" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 34269 } } peers { permanent_uuid: "4495e9f7ef3741dca3d605b513944883" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 37143 } } peers { permanent_uuid: "02552a915508491b9b9d484df84d1d43" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 46615 } }
I20250626 13:58:09.681589 3853 raft_consensus.cc:738] T 51d2720a898d47e29cc483a7bad784f9 P d9be968f292d47bb9aed877458694e57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9be968f292d47bb9aed877458694e57, State: Initialized, Role: FOLLOWER
I20250626 13:58:09.682996 3853 consensus_queue.cc:260] T 51d2720a898d47e29cc483a7bad784f9 P d9be968f292d47bb9aed877458694e57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9be968f292d47bb9aed877458694e57" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 34269 } } peers { permanent_uuid: "4495e9f7ef3741dca3d605b513944883" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 37143 } } peers { permanent_uuid: "02552a915508491b9b9d484df84d1d43" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 46615 } }
I20250626 13:58:09.685508 3852 ts_tablet_manager.cc:1428] T 51d2720a898d47e29cc483a7bad784f9 P 4495e9f7ef3741dca3d605b513944883: Time spent starting tablet: real 0.032s user 0.031s sys 0.000s
I20250626 13:58:09.686043 3833 heartbeater.cc:499] Master 127.2.107.126:37415 was elected leader, sending a full tablet report...
I20250626 13:58:09.686918 3853 ts_tablet_manager.cc:1428] T 51d2720a898d47e29cc483a7bad784f9 P d9be968f292d47bb9aed877458694e57: Time spent starting tablet: real 0.032s user 0.027s sys 0.004s
I20250626 13:58:09.710464 2477 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 13:58:09.713507 2477 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 02552a915508491b9b9d484df84d1d43 to finish bootstrapping
I20250626 13:58:09.725329 2477 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 4495e9f7ef3741dca3d605b513944883 to finish bootstrapping
I20250626 13:58:09.735092 2477 ts_itest-base.cc:246] Waiting for 1 tablets on tserver d9be968f292d47bb9aed877458694e57 to finish bootstrapping
W20250626 13:58:09.743254 3834 tablet.cc:2378] T 51d2720a898d47e29cc483a7bad784f9 P d9be968f292d47bb9aed877458694e57: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 13:58:09.774924 3522 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "51d2720a898d47e29cc483a7bad784f9"
dest_uuid: "02552a915508491b9b9d484df84d1d43"
from {username='slave'} at 127.0.0.1:32890
I20250626 13:58:09.775483 3522 raft_consensus.cc:491] T 51d2720a898d47e29cc483a7bad784f9 P 02552a915508491b9b9d484df84d1d43 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 13:58:09.775789 3522 raft_consensus.cc:3058] T 51d2720a898d47e29cc483a7bad784f9 P 02552a915508491b9b9d484df84d1d43 [term 0 FOLLOWER]: Advancing to term 1
I20250626 13:58:09.780517 3522 raft_consensus.cc:513] T 51d2720a898d47e29cc483a7bad784f9 P 02552a915508491b9b9d484df84d1d43 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9be968f292d47bb9aed877458694e57" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 34269 } } peers { permanent_uuid: "4495e9f7ef3741dca3d605b513944883" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 37143 } } peers { permanent_uuid: "02552a915508491b9b9d484df84d1d43" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 46615 } }
I20250626 13:58:09.782851 3522 leader_election.cc:290] T 51d2720a898d47e29cc483a7bad784f9 P 02552a915508491b9b9d484df84d1d43 [CANDIDATE]: Term 1 election: Requested vote from peers d9be968f292d47bb9aed877458694e57 (127.2.107.67:34269), 4495e9f7ef3741dca3d605b513944883 (127.2.107.66:37143)
W20250626 13:58:09.789916 3701 tablet.cc:2378] T 51d2720a898d47e29cc483a7bad784f9 P 4495e9f7ef3741dca3d605b513944883: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 13:58:09.791230 2477 cluster_itest_util.cc:257] Not converged past 1 yet: 0.0 0.0 0.0
I20250626 13:58:09.805724 3655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "51d2720a898d47e29cc483a7bad784f9" candidate_uuid: "02552a915508491b9b9d484df84d1d43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4495e9f7ef3741dca3d605b513944883"
I20250626 13:58:09.806600 3788 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "51d2720a898d47e29cc483a7bad784f9" candidate_uuid: "02552a915508491b9b9d484df84d1d43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d9be968f292d47bb9aed877458694e57"
I20250626 13:58:09.806838 3655 raft_consensus.cc:3058] T 51d2720a898d47e29cc483a7bad784f9 P 4495e9f7ef3741dca3d605b513944883 [term 0 FOLLOWER]: Advancing to term 1
I20250626 13:58:09.807188 3788 raft_consensus.cc:3058] T 51d2720a898d47e29cc483a7bad784f9 P d9be968f292d47bb9aed877458694e57 [term 0 FOLLOWER]: Advancing to term 1
I20250626 13:58:09.811528 3788 raft_consensus.cc:2466] T 51d2720a898d47e29cc483a7bad784f9 P d9be968f292d47bb9aed877458694e57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02552a915508491b9b9d484df84d1d43 in term 1.
I20250626 13:58:09.812563 3457 leader_election.cc:304] T 51d2720a898d47e29cc483a7bad784f9 P 02552a915508491b9b9d484df84d1d43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02552a915508491b9b9d484df84d1d43, d9be968f292d47bb9aed877458694e57; no voters:
I20250626 13:58:09.813369 3858 raft_consensus.cc:2802] T 51d2720a898d47e29cc483a7bad784f9 P 02552a915508491b9b9d484df84d1d43 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 13:58:09.813416 3655 raft_consensus.cc:2466] T 51d2720a898d47e29cc483a7bad784f9 P 4495e9f7ef3741dca3d605b513944883 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02552a915508491b9b9d484df84d1d43 in term 1.
I20250626 13:58:09.815237 3858 raft_consensus.cc:695] T 51d2720a898d47e29cc483a7bad784f9 P 02552a915508491b9b9d484df84d1d43 [term 1 LEADER]: Becoming Leader. State: Replica: 02552a915508491b9b9d484df84d1d43, State: Running, Role: LEADER
I20250626 13:58:09.815979 3858 consensus_queue.cc:237] T 51d2720a898d47e29cc483a7bad784f9 P 02552a915508491b9b9d484df84d1d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9be968f292d47bb9aed877458694e57" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 34269 } } peers { permanent_uuid: "4495e9f7ef3741dca3d605b513944883" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 37143 } } peers { permanent_uuid: "02552a915508491b9b9d484df84d1d43" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 46615 } }
I20250626 13:58:09.827131 3379 catalog_manager.cc:5582] T 51d2720a898d47e29cc483a7bad784f9 P 02552a915508491b9b9d484df84d1d43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02552a915508491b9b9d484df84d1d43 (127.2.107.65). New cstate: current_term: 1 leader_uuid: "02552a915508491b9b9d484df84d1d43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9be968f292d47bb9aed877458694e57" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 34269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4495e9f7ef3741dca3d605b513944883" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 37143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02552a915508491b9b9d484df84d1d43" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 46615 } health_report { overall_health: HEALTHY } } }
W20250626 13:58:09.855834 3568 tablet.cc:2378] T 51d2720a898d47e29cc483a7bad784f9 P 02552a915508491b9b9d484df84d1d43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 13:58:09.896879 2477 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250626 13:58:10.102246 2477 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250626 13:58:10.327363 3867 consensus_queue.cc:1035] T 51d2720a898d47e29cc483a7bad784f9 P 02552a915508491b9b9d484df84d1d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4495e9f7ef3741dca3d605b513944883" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 13:58:10.335021 3867 consensus_queue.cc:1035] T 51d2720a898d47e29cc483a7bad784f9 P 02552a915508491b9b9d484df84d1d43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9be968f292d47bb9aed877458694e57" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 34269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 13:58:11.962558 3521 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "51d2720a898d47e29cc483a7bad784f9"
dest_uuid: "02552a915508491b9b9d484df84d1d43"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:32904
I20250626 13:58:11.963047 3521 raft_consensus.cc:604] T 51d2720a898d47e29cc483a7bad784f9 P 02552a915508491b9b9d484df84d1d43 [term 1 LEADER]: Received request to transfer leadership
I20250626 13:58:12.306526 3895 raft_consensus.cc:991] T 51d2720a898d47e29cc483a7bad784f9 P 02552a915508491b9b9d484df84d1d43: : Instructing follower d9be968f292d47bb9aed877458694e57 to start an election
I20250626 13:58:12.307003 3870 raft_consensus.cc:1079] T 51d2720a898d47e29cc483a7bad784f9 P 02552a915508491b9b9d484df84d1d43 [term 1 LEADER]: Signalling peer d9be968f292d47bb9aed877458694e57 to start an election
I20250626 13:58:12.308280 3788 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "51d2720a898d47e29cc483a7bad784f9"
dest_uuid: "d9be968f292d47bb9aed877458694e57"
from {username='slave'} at 127.2.107.65:37039
I20250626 13:58:12.308779 3788 raft_consensus.cc:491] T 51d2720a898d47e29cc483a7bad784f9 P d9be968f292d47bb9aed877458694e57 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 13:58:12.309028 3788 raft_consensus.cc:3058] T 51d2720a898d47e29cc483a7bad784f9 P d9be968f292d47bb9aed877458694e57 [term 1 FOLLOWER]: Advancing to term 2
I20250626 13:58:12.313019 3788 raft_consensus.cc:513] T 51d2720a898d47e29cc483a7bad784f9 P d9be968f292d47bb9aed877458694e57 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9be968f292d47bb9aed877458694e57" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 34269 } } peers { permanent_uuid: "4495e9f7ef3741dca3d605b513944883" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 37143 } } peers { permanent_uuid: "02552a915508491b9b9d484df84d1d43" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 46615 } }
I20250626 13:58:12.315127 3788 leader_election.cc:290] T 51d2720a898d47e29cc483a7bad784f9 P d9be968f292d47bb9aed877458694e57 [CANDIDATE]: Term 2 election: Requested vote from peers 4495e9f7ef3741dca3d605b513944883 (127.2.107.66:37143), 02552a915508491b9b9d484df84d1d43 (127.2.107.65:46615)
I20250626 13:58:12.326128 3521 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "51d2720a898d47e29cc483a7bad784f9" candidate_uuid: "d9be968f292d47bb9aed877458694e57" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "02552a915508491b9b9d484df84d1d43"
I20250626 13:58:12.326236 3655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "51d2720a898d47e29cc483a7bad784f9" candidate_uuid: "d9be968f292d47bb9aed877458694e57" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4495e9f7ef3741dca3d605b513944883"
I20250626 13:58:12.326633 3521 raft_consensus.cc:3053] T 51d2720a898d47e29cc483a7bad784f9 P 02552a915508491b9b9d484df84d1d43 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 13:58:12.326658 3655 raft_consensus.cc:3058] T 51d2720a898d47e29cc483a7bad784f9 P 4495e9f7ef3741dca3d605b513944883 [term 1 FOLLOWER]: Advancing to term 2
I20250626 13:58:12.326876 3521 raft_consensus.cc:738] T 51d2720a898d47e29cc483a7bad784f9 P 02552a915508491b9b9d484df84d1d43 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 02552a915508491b9b9d484df84d1d43, State: Running, Role: LEADER
I20250626 13:58:12.327332 3521 consensus_queue.cc:260] T 51d2720a898d47e29cc483a7bad784f9 P 02552a915508491b9b9d484df84d1d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9be968f292d47bb9aed877458694e57" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 34269 } } peers { permanent_uuid: "4495e9f7ef3741dca3d605b513944883" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 37143 } } peers { permanent_uuid: "02552a915508491b9b9d484df84d1d43" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 46615 } }
I20250626 13:58:12.328235 3521 raft_consensus.cc:3058] T 51d2720a898d47e29cc483a7bad784f9 P 02552a915508491b9b9d484df84d1d43 [term 1 FOLLOWER]: Advancing to term 2
I20250626 13:58:12.330863 3655 raft_consensus.cc:2466] T 51d2720a898d47e29cc483a7bad784f9 P 4495e9f7ef3741dca3d605b513944883 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9be968f292d47bb9aed877458694e57 in term 2.
I20250626 13:58:12.331790 3721 leader_election.cc:304] T 51d2720a898d47e29cc483a7bad784f9 P d9be968f292d47bb9aed877458694e57 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4495e9f7ef3741dca3d605b513944883, d9be968f292d47bb9aed877458694e57; no voters:
I20250626 13:58:12.332388 3521 raft_consensus.cc:2466] T 51d2720a898d47e29cc483a7bad784f9 P 02552a915508491b9b9d484df84d1d43 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9be968f292d47bb9aed877458694e57 in term 2.
I20250626 13:58:12.333837 3899 raft_consensus.cc:2802] T 51d2720a898d47e29cc483a7bad784f9 P d9be968f292d47bb9aed877458694e57 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 13:58:12.335011 3899 raft_consensus.cc:695] T 51d2720a898d47e29cc483a7bad784f9 P d9be968f292d47bb9aed877458694e57 [term 2 LEADER]: Becoming Leader. State: Replica: d9be968f292d47bb9aed877458694e57, State: Running, Role: LEADER
I20250626 13:58:12.335727 3899 consensus_queue.cc:237] T 51d2720a898d47e29cc483a7bad784f9 P d9be968f292d47bb9aed877458694e57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9be968f292d47bb9aed877458694e57" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 34269 } } peers { permanent_uuid: "4495e9f7ef3741dca3d605b513944883" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 37143 } } peers { permanent_uuid: "02552a915508491b9b9d484df84d1d43" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 46615 } }
I20250626 13:58:12.342723 3378 catalog_manager.cc:5582] T 51d2720a898d47e29cc483a7bad784f9 P d9be968f292d47bb9aed877458694e57 reported cstate change: term changed from 1 to 2, leader changed from 02552a915508491b9b9d484df84d1d43 (127.2.107.65) to d9be968f292d47bb9aed877458694e57 (127.2.107.67). New cstate: current_term: 2 leader_uuid: "d9be968f292d47bb9aed877458694e57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9be968f292d47bb9aed877458694e57" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 34269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4495e9f7ef3741dca3d605b513944883" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 37143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02552a915508491b9b9d484df84d1d43" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 46615 } health_report { overall_health: UNKNOWN } } }
I20250626 13:58:12.840428 3655 raft_consensus.cc:1273] T 51d2720a898d47e29cc483a7bad784f9 P 4495e9f7ef3741dca3d605b513944883 [term 2 FOLLOWER]: Refusing update from remote peer d9be968f292d47bb9aed877458694e57: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 13:58:12.841583 3899 consensus_queue.cc:1035] T 51d2720a898d47e29cc483a7bad784f9 P d9be968f292d47bb9aed877458694e57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4495e9f7ef3741dca3d605b513944883" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 13:58:12.854133 3521 raft_consensus.cc:1273] T 51d2720a898d47e29cc483a7bad784f9 P 02552a915508491b9b9d484df84d1d43 [term 2 FOLLOWER]: Refusing update from remote peer d9be968f292d47bb9aed877458694e57: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 13:58:12.856623 3909 consensus_queue.cc:1035] T 51d2720a898d47e29cc483a7bad784f9 P d9be968f292d47bb9aed877458694e57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02552a915508491b9b9d484df84d1d43" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 46615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250626 13:58:14.533998 3521 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "51d2720a898d47e29cc483a7bad784f9"
dest_uuid: "02552a915508491b9b9d484df84d1d43"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57266
I20250626 13:58:14.534598 3521 raft_consensus.cc:604] T 51d2720a898d47e29cc483a7bad784f9 P 02552a915508491b9b9d484df84d1d43 [term 2 FOLLOWER]: Received request to transfer leadership
I20250626 13:58:14.534906 3521 raft_consensus.cc:612] T 51d2720a898d47e29cc483a7bad784f9 P 02552a915508491b9b9d484df84d1d43 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250626 13:58:15.566486 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 3438
I20250626 13:58:15.587090 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 3571
I20250626 13:58:15.609280 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 3704
I20250626 13:58:15.631268 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 3345
2025-06-26T13:58:15Z chronyd exiting
[ OK ] AdminCliTest.TestGracefulSpecificLeaderStepDown (13501 ms)
[ RUN ] AdminCliTest.TestDescribeTableColumnFlags
I20250626 13:58:15.681910 2477 test_util.cc:276] Using random seed: -201747621
I20250626 13:58:15.685940 2477 ts_itest-base.cc:115] Starting cluster with:
I20250626 13:58:15.686106 2477 ts_itest-base.cc:116] --------------
I20250626 13:58:15.686259 2477 ts_itest-base.cc:117] 3 tablet servers
I20250626 13:58:15.686398 2477 ts_itest-base.cc:118] 3 replicas per TS
I20250626 13:58:15.686538 2477 ts_itest-base.cc:119] --------------
2025-06-26T13:58:15Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-26T13:58:15Z Disabled control of system clock
I20250626 13:58:15.721020 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.107.126:45887
--webserver_interface=127.2.107.126
--webserver_port=0
--builtin_ntp_servers=127.2.107.84:33199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.107.126:45887 with env {}
W20250626 13:58:16.009706 3947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:58:16.010259 3947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:58:16.010668 3947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:58:16.040971 3947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 13:58:16.041272 3947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:58:16.041476 3947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 13:58:16.041668 3947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 13:58:16.075901 3947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:33199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.107.126:45887
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.107.126:45887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.107.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:58:16.077168 3947 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:58:16.078740 3947 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:58:16.094252 3953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:58:16.094377 3956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:58:16.095566 3947 server_base.cc:1048] running on GCE node
W20250626 13:58:16.094295 3954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:58:17.190824 3947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:58:17.193787 3947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:58:17.195286 3947 hybrid_clock.cc:648] HybridClock initialized: now 1750946297195249 us; error 52 us; skew 500 ppm
I20250626 13:58:17.196439 3947 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:58:17.203511 3947 webserver.cc:469] Webserver started at http://127.2.107.126:32921/ using document root <none> and password file <none>
I20250626 13:58:17.204438 3947 fs_manager.cc:362] Metadata directory not provided
I20250626 13:58:17.204649 3947 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:58:17.205082 3947 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 13:58:17.209439 3947 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "4f9704ba86fd475daeddd42652c7daa3"
format_stamp: "Formatted at 2025-06-26 13:58:17 on dist-test-slave-7p9x"
I20250626 13:58:17.210515 3947 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "4f9704ba86fd475daeddd42652c7daa3"
format_stamp: "Formatted at 2025-06-26 13:58:17 on dist-test-slave-7p9x"
I20250626 13:58:17.217492 3947 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250626 13:58:17.222731 3963 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:17.223733 3947 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20250626 13:58:17.224064 3947 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
uuid: "4f9704ba86fd475daeddd42652c7daa3"
format_stamp: "Formatted at 2025-06-26 13:58:17 on dist-test-slave-7p9x"
I20250626 13:58:17.224416 3947 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:58:17.282907 3947 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:58:17.284435 3947 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:58:17.284871 3947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:58:17.351882 3947 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.126:45887
I20250626 13:58:17.351953 4014 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.126:45887 every 8 connection(s)
I20250626 13:58:17.354499 3947 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250626 13:58:17.356076 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 3947
I20250626 13:58:17.356477 2477 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250626 13:58:17.360755 4015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:58:17.384279 4015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f9704ba86fd475daeddd42652c7daa3: Bootstrap starting.
I20250626 13:58:17.389878 4015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f9704ba86fd475daeddd42652c7daa3: Neither blocks nor log segments found. Creating new log.
I20250626 13:58:17.391554 4015 log.cc:826] T 00000000000000000000000000000000 P 4f9704ba86fd475daeddd42652c7daa3: Log is configured to *not* fsync() on all Append() calls
I20250626 13:58:17.395907 4015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f9704ba86fd475daeddd42652c7daa3: No bootstrap required, opened a new log
I20250626 13:58:17.415222 4015 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4f9704ba86fd475daeddd42652c7daa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f9704ba86fd475daeddd42652c7daa3" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 45887 } }
I20250626 13:58:17.415887 4015 raft_consensus.cc:383] T 00000000000000000000000000000000 P 4f9704ba86fd475daeddd42652c7daa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 13:58:17.416110 4015 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4f9704ba86fd475daeddd42652c7daa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f9704ba86fd475daeddd42652c7daa3, State: Initialized, Role: FOLLOWER
I20250626 13:58:17.416764 4015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f9704ba86fd475daeddd42652c7daa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f9704ba86fd475daeddd42652c7daa3" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 45887 } }
I20250626 13:58:17.417276 4015 raft_consensus.cc:397] T 00000000000000000000000000000000 P 4f9704ba86fd475daeddd42652c7daa3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 13:58:17.417536 4015 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4f9704ba86fd475daeddd42652c7daa3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 13:58:17.417814 4015 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4f9704ba86fd475daeddd42652c7daa3 [term 0 FOLLOWER]: Advancing to term 1
I20250626 13:58:17.422400 4015 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4f9704ba86fd475daeddd42652c7daa3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f9704ba86fd475daeddd42652c7daa3" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 45887 } }
I20250626 13:58:17.423081 4015 leader_election.cc:304] T 00000000000000000000000000000000 P 4f9704ba86fd475daeddd42652c7daa3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f9704ba86fd475daeddd42652c7daa3; no voters:
I20250626 13:58:17.424675 4015 leader_election.cc:290] T 00000000000000000000000000000000 P 4f9704ba86fd475daeddd42652c7daa3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 13:58:17.425324 4020 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4f9704ba86fd475daeddd42652c7daa3 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 13:58:17.427201 4020 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4f9704ba86fd475daeddd42652c7daa3 [term 1 LEADER]: Becoming Leader. State: Replica: 4f9704ba86fd475daeddd42652c7daa3, State: Running, Role: LEADER
I20250626 13:58:17.427935 4020 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f9704ba86fd475daeddd42652c7daa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f9704ba86fd475daeddd42652c7daa3" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 45887 } }
I20250626 13:58:17.428879 4015 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f9704ba86fd475daeddd42652c7daa3 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 13:58:17.437640 4021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f9704ba86fd475daeddd42652c7daa3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f9704ba86fd475daeddd42652c7daa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f9704ba86fd475daeddd42652c7daa3" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 45887 } } }
I20250626 13:58:17.438336 4021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f9704ba86fd475daeddd42652c7daa3 [sys.catalog]: This master's current role is: LEADER
I20250626 13:58:17.439780 4022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f9704ba86fd475daeddd42652c7daa3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f9704ba86fd475daeddd42652c7daa3. Latest consensus state: current_term: 1 leader_uuid: "4f9704ba86fd475daeddd42652c7daa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f9704ba86fd475daeddd42652c7daa3" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 45887 } } }
I20250626 13:58:17.440387 4022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f9704ba86fd475daeddd42652c7daa3 [sys.catalog]: This master's current role is: LEADER
I20250626 13:58:17.445156 4030 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 13:58:17.456378 4030 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 13:58:17.471100 4030 catalog_manager.cc:1349] Generated new cluster ID: 8086dc93c3ba49eb8912293b026862e1
I20250626 13:58:17.471423 4030 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 13:58:17.502730 4030 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 13:58:17.504172 4030 catalog_manager.cc:1506] Loading token signing keys...
I20250626 13:58:17.524552 4030 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 4f9704ba86fd475daeddd42652c7daa3: Generated new TSK 0
I20250626 13:58:17.525465 4030 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 13:58:17.538255 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.107.65:0
--local_ip_for_outbound_sockets=127.2.107.65
--webserver_interface=127.2.107.65
--webserver_port=0
--tserver_master_addrs=127.2.107.126:45887
--builtin_ntp_servers=127.2.107.84:33199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250626 13:58:17.827230 4039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:58:17.827716 4039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:58:17.828207 4039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:58:17.858932 4039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:58:17.859772 4039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.107.65
I20250626 13:58:17.894707 4039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:33199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.107.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.107.65
--webserver_port=0
--tserver_master_addrs=127.2.107.126:45887
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.107.65
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:58:17.895968 4039 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:58:17.897616 4039 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:58:17.914759 4045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:58:17.914911 4046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:58:17.916545 4039 server_base.cc:1048] running on GCE node
W20250626 13:58:17.915004 4048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:58:18.999439 4039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:58:19.002028 4039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:58:19.003450 4039 hybrid_clock.cc:648] HybridClock initialized: now 1750946299003393 us; error 74 us; skew 500 ppm
I20250626 13:58:19.004264 4039 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:58:19.010823 4039 webserver.cc:469] Webserver started at http://127.2.107.65:43577/ using document root <none> and password file <none>
I20250626 13:58:19.011718 4039 fs_manager.cc:362] Metadata directory not provided
I20250626 13:58:19.011914 4039 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:58:19.012359 4039 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 13:58:19.016649 4039 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "3debe32907bf4023a01f1c9986a30d5b"
format_stamp: "Formatted at 2025-06-26 13:58:19 on dist-test-slave-7p9x"
I20250626 13:58:19.017724 4039 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "3debe32907bf4023a01f1c9986a30d5b"
format_stamp: "Formatted at 2025-06-26 13:58:19 on dist-test-slave-7p9x"
I20250626 13:58:19.024816 4039 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250626 13:58:19.030300 4055 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:19.031414 4039 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.005s
I20250626 13:58:19.031713 4039 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "3debe32907bf4023a01f1c9986a30d5b"
format_stamp: "Formatted at 2025-06-26 13:58:19 on dist-test-slave-7p9x"
I20250626 13:58:19.032030 4039 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:58:19.081514 4039 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:58:19.082926 4039 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:58:19.083366 4039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:58:19.086160 4039 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 13:58:19.090261 4039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 13:58:19.090459 4039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:19.090692 4039 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 13:58:19.090849 4039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:19.229264 4039 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.65:43533
I20250626 13:58:19.229355 4167 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.65:43533 every 8 connection(s)
I20250626 13:58:19.231806 4039 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250626 13:58:19.241346 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 4039
I20250626 13:58:19.241730 2477 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250626 13:58:19.247232 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.107.66:0
--local_ip_for_outbound_sockets=127.2.107.66
--webserver_interface=127.2.107.66
--webserver_port=0
--tserver_master_addrs=127.2.107.126:45887
--builtin_ntp_servers=127.2.107.84:33199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 13:58:19.251438 4168 heartbeater.cc:344] Connected to a master server at 127.2.107.126:45887
I20250626 13:58:19.251937 4168 heartbeater.cc:461] Registering TS with master...
I20250626 13:58:19.253152 4168 heartbeater.cc:507] Master 127.2.107.126:45887 requested a full tablet report, sending...
I20250626 13:58:19.256033 3980 ts_manager.cc:194] Registered new tserver with Master: 3debe32907bf4023a01f1c9986a30d5b (127.2.107.65:43533)
I20250626 13:58:19.258641 3980 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.107.65:45415
W20250626 13:58:19.535094 4172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:58:19.535542 4172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:58:19.535996 4172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:58:19.566946 4172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:58:19.567732 4172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.107.66
I20250626 13:58:19.601708 4172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:33199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.107.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.107.66
--webserver_port=0
--tserver_master_addrs=127.2.107.126:45887
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.107.66
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:58:19.602952 4172 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:58:19.604526 4172 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:58:19.618835 4179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:58:20.262013 4168 heartbeater.cc:499] Master 127.2.107.126:45887 was elected leader, sending a full tablet report...
W20250626 13:58:19.618924 4178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:58:19.620532 4181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:58:20.681010 4180 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 13:58:20.681053 4172 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 13:58:20.685665 4172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:58:20.687851 4172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:58:20.689231 4172 hybrid_clock.cc:648] HybridClock initialized: now 1750946300689195 us; error 45 us; skew 500 ppm
I20250626 13:58:20.690076 4172 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:58:20.699322 4172 webserver.cc:469] Webserver started at http://127.2.107.66:37541/ using document root <none> and password file <none>
I20250626 13:58:20.700261 4172 fs_manager.cc:362] Metadata directory not provided
I20250626 13:58:20.700495 4172 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:58:20.700949 4172 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 13:58:20.705622 4172 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "b647e7fffe884a90850d540cb1a10ee5"
format_stamp: "Formatted at 2025-06-26 13:58:20 on dist-test-slave-7p9x"
I20250626 13:58:20.707000 4172 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "b647e7fffe884a90850d540cb1a10ee5"
format_stamp: "Formatted at 2025-06-26 13:58:20 on dist-test-slave-7p9x"
I20250626 13:58:20.714280 4172 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20250626 13:58:20.720698 4188 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:20.721923 4172 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250626 13:58:20.722270 4172 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "b647e7fffe884a90850d540cb1a10ee5"
format_stamp: "Formatted at 2025-06-26 13:58:20 on dist-test-slave-7p9x"
I20250626 13:58:20.722659 4172 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:58:20.783519 4172 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:58:20.784963 4172 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:58:20.785398 4172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:58:20.787784 4172 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 13:58:20.791926 4172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 13:58:20.792115 4172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:20.792372 4172 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 13:58:20.792519 4172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:20.923367 4172 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.66:39263
I20250626 13:58:20.923485 4300 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.66:39263 every 8 connection(s)
I20250626 13:58:20.925863 4172 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250626 13:58:20.934325 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 4172
I20250626 13:58:20.934793 2477 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250626 13:58:20.941411 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.107.67:0
--local_ip_for_outbound_sockets=127.2.107.67
--webserver_interface=127.2.107.67
--webserver_port=0
--tserver_master_addrs=127.2.107.126:45887
--builtin_ntp_servers=127.2.107.84:33199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 13:58:20.947407 4301 heartbeater.cc:344] Connected to a master server at 127.2.107.126:45887
I20250626 13:58:20.947824 4301 heartbeater.cc:461] Registering TS with master...
I20250626 13:58:20.948787 4301 heartbeater.cc:507] Master 127.2.107.126:45887 requested a full tablet report, sending...
I20250626 13:58:20.950810 3980 ts_manager.cc:194] Registered new tserver with Master: b647e7fffe884a90850d540cb1a10ee5 (127.2.107.66:39263)
I20250626 13:58:20.952013 3980 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.107.66:33155
W20250626 13:58:21.225312 4305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:58:21.225754 4305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:58:21.226191 4305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:58:21.262916 4305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:58:21.263844 4305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.107.67
I20250626 13:58:21.298218 4305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:33199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.107.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.107.67
--webserver_port=0
--tserver_master_addrs=127.2.107.126:45887
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.107.67
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:58:21.299453 4305 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:58:21.301021 4305 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:58:21.315560 4311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:58:21.954944 4301 heartbeater.cc:499] Master 127.2.107.126:45887 was elected leader, sending a full tablet report...
W20250626 13:58:21.315712 4312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:58:21.317399 4314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:58:21.317701 4305 server_base.cc:1048] running on GCE node
I20250626 13:58:22.389917 4305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:58:22.392024 4305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:58:22.393394 4305 hybrid_clock.cc:648] HybridClock initialized: now 1750946302393351 us; error 61 us; skew 500 ppm
I20250626 13:58:22.394167 4305 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:58:22.399936 4305 webserver.cc:469] Webserver started at http://127.2.107.67:36347/ using document root <none> and password file <none>
I20250626 13:58:22.400821 4305 fs_manager.cc:362] Metadata directory not provided
I20250626 13:58:22.401031 4305 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:58:22.401499 4305 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 13:58:22.405874 4305 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "5b6980c46ce8424fb743e0bec091e2a6"
format_stamp: "Formatted at 2025-06-26 13:58:22 on dist-test-slave-7p9x"
I20250626 13:58:22.406936 4305 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "5b6980c46ce8424fb743e0bec091e2a6"
format_stamp: "Formatted at 2025-06-26 13:58:22 on dist-test-slave-7p9x"
I20250626 13:58:22.413635 4305 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250626 13:58:22.418826 4321 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:22.419791 4305 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250626 13:58:22.420081 4305 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "5b6980c46ce8424fb743e0bec091e2a6"
format_stamp: "Formatted at 2025-06-26 13:58:22 on dist-test-slave-7p9x"
I20250626 13:58:22.420409 4305 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:58:22.468981 4305 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:58:22.470438 4305 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:58:22.470852 4305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:58:22.473237 4305 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 13:58:22.477033 4305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 13:58:22.477277 4305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250626 13:58:22.477499 4305 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 13:58:22.477653 4305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:22.600708 4305 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.67:40277
I20250626 13:58:22.600808 4433 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.67:40277 every 8 connection(s)
I20250626 13:58:22.603269 4305 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250626 13:58:22.606737 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 4305
I20250626 13:58:22.607208 2477 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250626 13:58:22.623837 4434 heartbeater.cc:344] Connected to a master server at 127.2.107.126:45887
I20250626 13:58:22.624331 4434 heartbeater.cc:461] Registering TS with master...
I20250626 13:58:22.625481 4434 heartbeater.cc:507] Master 127.2.107.126:45887 requested a full tablet report, sending...
I20250626 13:58:22.627609 3980 ts_manager.cc:194] Registered new tserver with Master: 5b6980c46ce8424fb743e0bec091e2a6 (127.2.107.67:40277)
I20250626 13:58:22.628741 3980 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.107.67:51693
I20250626 13:58:22.641327 2477 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 13:58:22.669315 3980 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59540:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250626 13:58:22.687471 3980 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 13:58:22.735581 4236 tablet_service.cc:1468] Processing CreateTablet for tablet fe77b00feedd471dade12d9d31d6e3fa (DEFAULT_TABLE table=TestTable [id=1cd1dea7e4194ff6ab4ef232251d0372]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 13:58:22.736464 4103 tablet_service.cc:1468] Processing CreateTablet for tablet fe77b00feedd471dade12d9d31d6e3fa (DEFAULT_TABLE table=TestTable [id=1cd1dea7e4194ff6ab4ef232251d0372]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 13:58:22.737589 4236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe77b00feedd471dade12d9d31d6e3fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:58:22.738166 4103 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe77b00feedd471dade12d9d31d6e3fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:58:22.746685 4369 tablet_service.cc:1468] Processing CreateTablet for tablet fe77b00feedd471dade12d9d31d6e3fa (DEFAULT_TABLE table=TestTable [id=1cd1dea7e4194ff6ab4ef232251d0372]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 13:58:22.748167 4369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe77b00feedd471dade12d9d31d6e3fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:58:22.764956 4453 tablet_bootstrap.cc:492] T fe77b00feedd471dade12d9d31d6e3fa P b647e7fffe884a90850d540cb1a10ee5: Bootstrap starting.
I20250626 13:58:22.766638 4454 tablet_bootstrap.cc:492] T fe77b00feedd471dade12d9d31d6e3fa P 3debe32907bf4023a01f1c9986a30d5b: Bootstrap starting.
I20250626 13:58:22.770994 4455 tablet_bootstrap.cc:492] T fe77b00feedd471dade12d9d31d6e3fa P 5b6980c46ce8424fb743e0bec091e2a6: Bootstrap starting.
I20250626 13:58:22.772083 4453 tablet_bootstrap.cc:654] T fe77b00feedd471dade12d9d31d6e3fa P b647e7fffe884a90850d540cb1a10ee5: Neither blocks nor log segments found. Creating new log.
I20250626 13:58:22.773207 4454 tablet_bootstrap.cc:654] T fe77b00feedd471dade12d9d31d6e3fa P 3debe32907bf4023a01f1c9986a30d5b: Neither blocks nor log segments found. Creating new log.
I20250626 13:58:22.774403 4453 log.cc:826] T fe77b00feedd471dade12d9d31d6e3fa P b647e7fffe884a90850d540cb1a10ee5: Log is configured to *not* fsync() on all Append() calls
I20250626 13:58:22.775468 4454 log.cc:826] T fe77b00feedd471dade12d9d31d6e3fa P 3debe32907bf4023a01f1c9986a30d5b: Log is configured to *not* fsync() on all Append() calls
I20250626 13:58:22.778183 4455 tablet_bootstrap.cc:654] T fe77b00feedd471dade12d9d31d6e3fa P 5b6980c46ce8424fb743e0bec091e2a6: Neither blocks nor log segments found. Creating new log.
I20250626 13:58:22.780068 4453 tablet_bootstrap.cc:492] T fe77b00feedd471dade12d9d31d6e3fa P b647e7fffe884a90850d540cb1a10ee5: No bootstrap required, opened a new log
I20250626 13:58:22.780316 4455 log.cc:826] T fe77b00feedd471dade12d9d31d6e3fa P 5b6980c46ce8424fb743e0bec091e2a6: Log is configured to *not* fsync() on all Append() calls
I20250626 13:58:22.780565 4453 ts_tablet_manager.cc:1397] T fe77b00feedd471dade12d9d31d6e3fa P b647e7fffe884a90850d540cb1a10ee5: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.004s
I20250626 13:58:22.780815 4454 tablet_bootstrap.cc:492] T fe77b00feedd471dade12d9d31d6e3fa P 3debe32907bf4023a01f1c9986a30d5b: No bootstrap required, opened a new log
I20250626 13:58:22.781240 4454 ts_tablet_manager.cc:1397] T fe77b00feedd471dade12d9d31d6e3fa P 3debe32907bf4023a01f1c9986a30d5b: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.004s
I20250626 13:58:22.785621 4455 tablet_bootstrap.cc:492] T fe77b00feedd471dade12d9d31d6e3fa P 5b6980c46ce8424fb743e0bec091e2a6: No bootstrap required, opened a new log
I20250626 13:58:22.786103 4455 ts_tablet_manager.cc:1397] T fe77b00feedd471dade12d9d31d6e3fa P 5b6980c46ce8424fb743e0bec091e2a6: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.005s
I20250626 13:58:22.799485 4454 raft_consensus.cc:357] T fe77b00feedd471dade12d9d31d6e3fa P 3debe32907bf4023a01f1c9986a30d5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b647e7fffe884a90850d540cb1a10ee5" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 39263 } } peers { permanent_uuid: "3debe32907bf4023a01f1c9986a30d5b" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 43533 } } peers { permanent_uuid: "5b6980c46ce8424fb743e0bec091e2a6" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 40277 } }
I20250626 13:58:22.800179 4454 raft_consensus.cc:383] T fe77b00feedd471dade12d9d31d6e3fa P 3debe32907bf4023a01f1c9986a30d5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 13:58:22.800442 4454 raft_consensus.cc:738] T fe77b00feedd471dade12d9d31d6e3fa P 3debe32907bf4023a01f1c9986a30d5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3debe32907bf4023a01f1c9986a30d5b, State: Initialized, Role: FOLLOWER
I20250626 13:58:22.801231 4454 consensus_queue.cc:260] T fe77b00feedd471dade12d9d31d6e3fa P 3debe32907bf4023a01f1c9986a30d5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b647e7fffe884a90850d540cb1a10ee5" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 39263 } } peers { permanent_uuid: "3debe32907bf4023a01f1c9986a30d5b" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 43533 } } peers { permanent_uuid: "5b6980c46ce8424fb743e0bec091e2a6" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 40277 } }
I20250626 13:58:22.805097 4454 ts_tablet_manager.cc:1428] T fe77b00feedd471dade12d9d31d6e3fa P 3debe32907bf4023a01f1c9986a30d5b: Time spent starting tablet: real 0.024s user 0.017s sys 0.004s
I20250626 13:58:22.806284 4453 raft_consensus.cc:357] T fe77b00feedd471dade12d9d31d6e3fa P b647e7fffe884a90850d540cb1a10ee5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b647e7fffe884a90850d540cb1a10ee5" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 39263 } } peers { permanent_uuid: "3debe32907bf4023a01f1c9986a30d5b" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 43533 } } peers { permanent_uuid: "5b6980c46ce8424fb743e0bec091e2a6" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 40277 } }
I20250626 13:58:22.807243 4453 raft_consensus.cc:383] T fe77b00feedd471dade12d9d31d6e3fa P b647e7fffe884a90850d540cb1a10ee5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 13:58:22.807570 4453 raft_consensus.cc:738] T fe77b00feedd471dade12d9d31d6e3fa P b647e7fffe884a90850d540cb1a10ee5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b647e7fffe884a90850d540cb1a10ee5, State: Initialized, Role: FOLLOWER
I20250626 13:58:22.809057 4453 consensus_queue.cc:260] T fe77b00feedd471dade12d9d31d6e3fa P b647e7fffe884a90850d540cb1a10ee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b647e7fffe884a90850d540cb1a10ee5" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 39263 } } peers { permanent_uuid: "3debe32907bf4023a01f1c9986a30d5b" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 43533 } } peers { permanent_uuid: "5b6980c46ce8424fb743e0bec091e2a6" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 40277 } }
I20250626 13:58:22.811414 4455 raft_consensus.cc:357] T fe77b00feedd471dade12d9d31d6e3fa P 5b6980c46ce8424fb743e0bec091e2a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b647e7fffe884a90850d540cb1a10ee5" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 39263 } } peers { permanent_uuid: "3debe32907bf4023a01f1c9986a30d5b" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 43533 } } peers { permanent_uuid: "5b6980c46ce8424fb743e0bec091e2a6" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 40277 } }
I20250626 13:58:22.812330 4455 raft_consensus.cc:383] T fe77b00feedd471dade12d9d31d6e3fa P 5b6980c46ce8424fb743e0bec091e2a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 13:58:22.812757 4455 raft_consensus.cc:738] T fe77b00feedd471dade12d9d31d6e3fa P 5b6980c46ce8424fb743e0bec091e2a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b6980c46ce8424fb743e0bec091e2a6, State: Initialized, Role: FOLLOWER
I20250626 13:58:22.813169 4453 ts_tablet_manager.cc:1428] T fe77b00feedd471dade12d9d31d6e3fa P b647e7fffe884a90850d540cb1a10ee5: Time spent starting tablet: real 0.032s user 0.028s sys 0.004s
I20250626 13:58:22.813808 4455 consensus_queue.cc:260] T fe77b00feedd471dade12d9d31d6e3fa P 5b6980c46ce8424fb743e0bec091e2a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b647e7fffe884a90850d540cb1a10ee5" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 39263 } } peers { permanent_uuid: "3debe32907bf4023a01f1c9986a30d5b" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 43533 } } peers { permanent_uuid: "5b6980c46ce8424fb743e0bec091e2a6" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 40277 } }
I20250626 13:58:22.816704 4434 heartbeater.cc:499] Master 127.2.107.126:45887 was elected leader, sending a full tablet report...
I20250626 13:58:22.817498 4455 ts_tablet_manager.cc:1428] T fe77b00feedd471dade12d9d31d6e3fa P 5b6980c46ce8424fb743e0bec091e2a6: Time spent starting tablet: real 0.031s user 0.027s sys 0.005s
W20250626 13:58:22.857509 4435 tablet.cc:2378] T fe77b00feedd471dade12d9d31d6e3fa P 5b6980c46ce8424fb743e0bec091e2a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250626 13:58:22.931828 4302 tablet.cc:2378] T fe77b00feedd471dade12d9d31d6e3fa P b647e7fffe884a90850d540cb1a10ee5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250626 13:58:22.989707 4169 tablet.cc:2378] T fe77b00feedd471dade12d9d31d6e3fa P 3debe32907bf4023a01f1c9986a30d5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 13:58:23.038424 4461 raft_consensus.cc:491] T fe77b00feedd471dade12d9d31d6e3fa P 5b6980c46ce8424fb743e0bec091e2a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 13:58:23.038908 4461 raft_consensus.cc:513] T fe77b00feedd471dade12d9d31d6e3fa P 5b6980c46ce8424fb743e0bec091e2a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b647e7fffe884a90850d540cb1a10ee5" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 39263 } } peers { permanent_uuid: "3debe32907bf4023a01f1c9986a30d5b" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 43533 } } peers { permanent_uuid: "5b6980c46ce8424fb743e0bec091e2a6" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 40277 } }
I20250626 13:58:23.041205 4461 leader_election.cc:290] T fe77b00feedd471dade12d9d31d6e3fa P 5b6980c46ce8424fb743e0bec091e2a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b647e7fffe884a90850d540cb1a10ee5 (127.2.107.66:39263), 3debe32907bf4023a01f1c9986a30d5b (127.2.107.65:43533)
I20250626 13:58:23.049605 4256 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe77b00feedd471dade12d9d31d6e3fa" candidate_uuid: "5b6980c46ce8424fb743e0bec091e2a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b647e7fffe884a90850d540cb1a10ee5" is_pre_election: true
I20250626 13:58:23.050442 4256 raft_consensus.cc:2466] T fe77b00feedd471dade12d9d31d6e3fa P b647e7fffe884a90850d540cb1a10ee5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b6980c46ce8424fb743e0bec091e2a6 in term 0.
I20250626 13:58:23.050935 4123 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe77b00feedd471dade12d9d31d6e3fa" candidate_uuid: "5b6980c46ce8424fb743e0bec091e2a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3debe32907bf4023a01f1c9986a30d5b" is_pre_election: true
I20250626 13:58:23.051573 4324 leader_election.cc:304] T fe77b00feedd471dade12d9d31d6e3fa P 5b6980c46ce8424fb743e0bec091e2a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b6980c46ce8424fb743e0bec091e2a6, b647e7fffe884a90850d540cb1a10ee5; no voters:
I20250626 13:58:23.051688 4123 raft_consensus.cc:2466] T fe77b00feedd471dade12d9d31d6e3fa P 3debe32907bf4023a01f1c9986a30d5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b6980c46ce8424fb743e0bec091e2a6 in term 0.
I20250626 13:58:23.052292 4461 raft_consensus.cc:2802] T fe77b00feedd471dade12d9d31d6e3fa P 5b6980c46ce8424fb743e0bec091e2a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 13:58:23.052533 4461 raft_consensus.cc:491] T fe77b00feedd471dade12d9d31d6e3fa P 5b6980c46ce8424fb743e0bec091e2a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 13:58:23.052774 4461 raft_consensus.cc:3058] T fe77b00feedd471dade12d9d31d6e3fa P 5b6980c46ce8424fb743e0bec091e2a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 13:58:23.057065 4461 raft_consensus.cc:513] T fe77b00feedd471dade12d9d31d6e3fa P 5b6980c46ce8424fb743e0bec091e2a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b647e7fffe884a90850d540cb1a10ee5" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 39263 } } peers { permanent_uuid: "3debe32907bf4023a01f1c9986a30d5b" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 43533 } } peers { permanent_uuid: "5b6980c46ce8424fb743e0bec091e2a6" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 40277 } }
I20250626 13:58:23.058481 4461 leader_election.cc:290] T fe77b00feedd471dade12d9d31d6e3fa P 5b6980c46ce8424fb743e0bec091e2a6 [CANDIDATE]: Term 1 election: Requested vote from peers b647e7fffe884a90850d540cb1a10ee5 (127.2.107.66:39263), 3debe32907bf4023a01f1c9986a30d5b (127.2.107.65:43533)
I20250626 13:58:23.059218 4256 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe77b00feedd471dade12d9d31d6e3fa" candidate_uuid: "5b6980c46ce8424fb743e0bec091e2a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b647e7fffe884a90850d540cb1a10ee5"
I20250626 13:58:23.059419 4123 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe77b00feedd471dade12d9d31d6e3fa" candidate_uuid: "5b6980c46ce8424fb743e0bec091e2a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3debe32907bf4023a01f1c9986a30d5b"
I20250626 13:58:23.059616 4256 raft_consensus.cc:3058] T fe77b00feedd471dade12d9d31d6e3fa P b647e7fffe884a90850d540cb1a10ee5 [term 0 FOLLOWER]: Advancing to term 1
I20250626 13:58:23.059836 4123 raft_consensus.cc:3058] T fe77b00feedd471dade12d9d31d6e3fa P 3debe32907bf4023a01f1c9986a30d5b [term 0 FOLLOWER]: Advancing to term 1
I20250626 13:58:23.064039 4256 raft_consensus.cc:2466] T fe77b00feedd471dade12d9d31d6e3fa P b647e7fffe884a90850d540cb1a10ee5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b6980c46ce8424fb743e0bec091e2a6 in term 1.
I20250626 13:58:23.064183 4123 raft_consensus.cc:2466] T fe77b00feedd471dade12d9d31d6e3fa P 3debe32907bf4023a01f1c9986a30d5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b6980c46ce8424fb743e0bec091e2a6 in term 1.
I20250626 13:58:23.064850 4324 leader_election.cc:304] T fe77b00feedd471dade12d9d31d6e3fa P 5b6980c46ce8424fb743e0bec091e2a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b6980c46ce8424fb743e0bec091e2a6, b647e7fffe884a90850d540cb1a10ee5; no voters:
I20250626 13:58:23.065455 4461 raft_consensus.cc:2802] T fe77b00feedd471dade12d9d31d6e3fa P 5b6980c46ce8424fb743e0bec091e2a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 13:58:23.066871 4461 raft_consensus.cc:695] T fe77b00feedd471dade12d9d31d6e3fa P 5b6980c46ce8424fb743e0bec091e2a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5b6980c46ce8424fb743e0bec091e2a6, State: Running, Role: LEADER
I20250626 13:58:23.067561 4461 consensus_queue.cc:237] T fe77b00feedd471dade12d9d31d6e3fa P 5b6980c46ce8424fb743e0bec091e2a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b647e7fffe884a90850d540cb1a10ee5" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 39263 } } peers { permanent_uuid: "3debe32907bf4023a01f1c9986a30d5b" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 43533 } } peers { permanent_uuid: "5b6980c46ce8424fb743e0bec091e2a6" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 40277 } }
I20250626 13:58:23.078429 3980 catalog_manager.cc:5582] T fe77b00feedd471dade12d9d31d6e3fa P 5b6980c46ce8424fb743e0bec091e2a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b6980c46ce8424fb743e0bec091e2a6 (127.2.107.67). New cstate: current_term: 1 leader_uuid: "5b6980c46ce8424fb743e0bec091e2a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b647e7fffe884a90850d540cb1a10ee5" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 39263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3debe32907bf4023a01f1c9986a30d5b" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 43533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b6980c46ce8424fb743e0bec091e2a6" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 40277 } health_report { overall_health: HEALTHY } } }
I20250626 13:58:23.107928 2477 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 13:58:23.111100 2477 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 3debe32907bf4023a01f1c9986a30d5b to finish bootstrapping
I20250626 13:58:23.123055 2477 ts_itest-base.cc:246] Waiting for 1 tablets on tserver b647e7fffe884a90850d540cb1a10ee5 to finish bootstrapping
I20250626 13:58:23.133615 2477 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 5b6980c46ce8424fb743e0bec091e2a6 to finish bootstrapping
I20250626 13:58:23.145807 3980 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59540:
name: "TestAnotherTable"
schema {
columns {
name: "foo"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "bar"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
comment: "comment for bar"
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "foo"
}
}
}
W20250626 13:58:23.147434 3980 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAnotherTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 13:58:23.163334 4103 tablet_service.cc:1468] Processing CreateTablet for tablet bf75d6d25d4842e3a51ea76c6d0d4728 (DEFAULT_TABLE table=TestAnotherTable [id=7fa868f10e53415fa5dc48430904054e]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250626 13:58:23.164086 4236 tablet_service.cc:1468] Processing CreateTablet for tablet bf75d6d25d4842e3a51ea76c6d0d4728 (DEFAULT_TABLE table=TestAnotherTable [id=7fa868f10e53415fa5dc48430904054e]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250626 13:58:23.164489 4103 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf75d6d25d4842e3a51ea76c6d0d4728. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:58:23.164834 4369 tablet_service.cc:1468] Processing CreateTablet for tablet bf75d6d25d4842e3a51ea76c6d0d4728 (DEFAULT_TABLE table=TestAnotherTable [id=7fa868f10e53415fa5dc48430904054e]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250626 13:58:23.165112 4236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf75d6d25d4842e3a51ea76c6d0d4728. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:58:23.165901 4369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf75d6d25d4842e3a51ea76c6d0d4728. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:58:23.176223 4455 tablet_bootstrap.cc:492] T bf75d6d25d4842e3a51ea76c6d0d4728 P 5b6980c46ce8424fb743e0bec091e2a6: Bootstrap starting.
I20250626 13:58:23.178175 4454 tablet_bootstrap.cc:492] T bf75d6d25d4842e3a51ea76c6d0d4728 P 3debe32907bf4023a01f1c9986a30d5b: Bootstrap starting.
I20250626 13:58:23.178625 4453 tablet_bootstrap.cc:492] T bf75d6d25d4842e3a51ea76c6d0d4728 P b647e7fffe884a90850d540cb1a10ee5: Bootstrap starting.
I20250626 13:58:23.181597 4455 tablet_bootstrap.cc:654] T bf75d6d25d4842e3a51ea76c6d0d4728 P 5b6980c46ce8424fb743e0bec091e2a6: Neither blocks nor log segments found. Creating new log.
I20250626 13:58:23.183903 4454 tablet_bootstrap.cc:654] T bf75d6d25d4842e3a51ea76c6d0d4728 P 3debe32907bf4023a01f1c9986a30d5b: Neither blocks nor log segments found. Creating new log.
I20250626 13:58:23.184026 4453 tablet_bootstrap.cc:654] T bf75d6d25d4842e3a51ea76c6d0d4728 P b647e7fffe884a90850d540cb1a10ee5: Neither blocks nor log segments found. Creating new log.
I20250626 13:58:23.190341 4455 tablet_bootstrap.cc:492] T bf75d6d25d4842e3a51ea76c6d0d4728 P 5b6980c46ce8424fb743e0bec091e2a6: No bootstrap required, opened a new log
I20250626 13:58:23.190752 4455 ts_tablet_manager.cc:1397] T bf75d6d25d4842e3a51ea76c6d0d4728 P 5b6980c46ce8424fb743e0bec091e2a6: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.005s
I20250626 13:58:23.190850 4453 tablet_bootstrap.cc:492] T bf75d6d25d4842e3a51ea76c6d0d4728 P b647e7fffe884a90850d540cb1a10ee5: No bootstrap required, opened a new log
I20250626 13:58:23.191231 4453 ts_tablet_manager.cc:1397] T bf75d6d25d4842e3a51ea76c6d0d4728 P b647e7fffe884a90850d540cb1a10ee5: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20250626 13:58:23.192040 4454 tablet_bootstrap.cc:492] T bf75d6d25d4842e3a51ea76c6d0d4728 P 3debe32907bf4023a01f1c9986a30d5b: No bootstrap required, opened a new log
I20250626 13:58:23.192456 4454 ts_tablet_manager.cc:1397] T bf75d6d25d4842e3a51ea76c6d0d4728 P 3debe32907bf4023a01f1c9986a30d5b: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20250626 13:58:23.192752 4455 raft_consensus.cc:357] T bf75d6d25d4842e3a51ea76c6d0d4728 P 5b6980c46ce8424fb743e0bec091e2a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3debe32907bf4023a01f1c9986a30d5b" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 43533 } } peers { permanent_uuid: "b647e7fffe884a90850d540cb1a10ee5" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 39263 } } peers { permanent_uuid: "5b6980c46ce8424fb743e0bec091e2a6" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 40277 } }
I20250626 13:58:23.193459 4455 raft_consensus.cc:383] T bf75d6d25d4842e3a51ea76c6d0d4728 P 5b6980c46ce8424fb743e0bec091e2a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 13:58:23.193742 4455 raft_consensus.cc:738] T bf75d6d25d4842e3a51ea76c6d0d4728 P 5b6980c46ce8424fb743e0bec091e2a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b6980c46ce8424fb743e0bec091e2a6, State: Initialized, Role: FOLLOWER
I20250626 13:58:23.193677 4453 raft_consensus.cc:357] T bf75d6d25d4842e3a51ea76c6d0d4728 P b647e7fffe884a90850d540cb1a10ee5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3debe32907bf4023a01f1c9986a30d5b" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 43533 } } peers { permanent_uuid: "b647e7fffe884a90850d540cb1a10ee5" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 39263 } } peers { permanent_uuid: "5b6980c46ce8424fb743e0bec091e2a6" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 40277 } }
I20250626 13:58:23.194347 4453 raft_consensus.cc:383] T bf75d6d25d4842e3a51ea76c6d0d4728 P b647e7fffe884a90850d540cb1a10ee5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 13:58:23.194659 4453 raft_consensus.cc:738] T bf75d6d25d4842e3a51ea76c6d0d4728 P b647e7fffe884a90850d540cb1a10ee5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b647e7fffe884a90850d540cb1a10ee5, State: Initialized, Role: FOLLOWER
I20250626 13:58:23.194459 4455 consensus_queue.cc:260] T bf75d6d25d4842e3a51ea76c6d0d4728 P 5b6980c46ce8424fb743e0bec091e2a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3debe32907bf4023a01f1c9986a30d5b" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 43533 } } peers { permanent_uuid: "b647e7fffe884a90850d540cb1a10ee5" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 39263 } } peers { permanent_uuid: "5b6980c46ce8424fb743e0bec091e2a6" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 40277 } }
I20250626 13:58:23.194941 4454 raft_consensus.cc:357] T bf75d6d25d4842e3a51ea76c6d0d4728 P 3debe32907bf4023a01f1c9986a30d5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3debe32907bf4023a01f1c9986a30d5b" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 43533 } } peers { permanent_uuid: "b647e7fffe884a90850d540cb1a10ee5" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 39263 } } peers { permanent_uuid: "5b6980c46ce8424fb743e0bec091e2a6" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 40277 } }
I20250626 13:58:23.195642 4454 raft_consensus.cc:383] T bf75d6d25d4842e3a51ea76c6d0d4728 P 3debe32907bf4023a01f1c9986a30d5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 13:58:23.195442 4453 consensus_queue.cc:260] T bf75d6d25d4842e3a51ea76c6d0d4728 P b647e7fffe884a90850d540cb1a10ee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3debe32907bf4023a01f1c9986a30d5b" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 43533 } } peers { permanent_uuid: "b647e7fffe884a90850d540cb1a10ee5" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 39263 } } peers { permanent_uuid: "5b6980c46ce8424fb743e0bec091e2a6" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 40277 } }
I20250626 13:58:23.195940 4454 raft_consensus.cc:738] T bf75d6d25d4842e3a51ea76c6d0d4728 P 3debe32907bf4023a01f1c9986a30d5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3debe32907bf4023a01f1c9986a30d5b, State: Initialized, Role: FOLLOWER
I20250626 13:58:23.196677 4454 consensus_queue.cc:260] T bf75d6d25d4842e3a51ea76c6d0d4728 P 3debe32907bf4023a01f1c9986a30d5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3debe32907bf4023a01f1c9986a30d5b" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 43533 } } peers { permanent_uuid: "b647e7fffe884a90850d540cb1a10ee5" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 39263 } } peers { permanent_uuid: "5b6980c46ce8424fb743e0bec091e2a6" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 40277 } }
I20250626 13:58:23.197592 4453 ts_tablet_manager.cc:1428] T bf75d6d25d4842e3a51ea76c6d0d4728 P b647e7fffe884a90850d540cb1a10ee5: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20250626 13:58:23.198681 4454 ts_tablet_manager.cc:1428] T bf75d6d25d4842e3a51ea76c6d0d4728 P 3debe32907bf4023a01f1c9986a30d5b: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250626 13:58:23.199504 4455 ts_tablet_manager.cc:1428] T bf75d6d25d4842e3a51ea76c6d0d4728 P 5b6980c46ce8424fb743e0bec091e2a6: Time spent starting tablet: real 0.009s user 0.004s sys 0.003s
I20250626 13:58:23.387223 4459 raft_consensus.cc:491] T bf75d6d25d4842e3a51ea76c6d0d4728 P 3debe32907bf4023a01f1c9986a30d5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 13:58:23.387645 4459 raft_consensus.cc:513] T bf75d6d25d4842e3a51ea76c6d0d4728 P 3debe32907bf4023a01f1c9986a30d5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3debe32907bf4023a01f1c9986a30d5b" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 43533 } } peers { permanent_uuid: "b647e7fffe884a90850d540cb1a10ee5" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 39263 } } peers { permanent_uuid: "5b6980c46ce8424fb743e0bec091e2a6" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 40277 } }
I20250626 13:58:23.390007 4459 leader_election.cc:290] T bf75d6d25d4842e3a51ea76c6d0d4728 P 3debe32907bf4023a01f1c9986a30d5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b647e7fffe884a90850d540cb1a10ee5 (127.2.107.66:39263), 5b6980c46ce8424fb743e0bec091e2a6 (127.2.107.67:40277)
I20250626 13:58:23.399448 4256 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf75d6d25d4842e3a51ea76c6d0d4728" candidate_uuid: "3debe32907bf4023a01f1c9986a30d5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b647e7fffe884a90850d540cb1a10ee5" is_pre_election: true
I20250626 13:58:23.399976 4256 raft_consensus.cc:2466] T bf75d6d25d4842e3a51ea76c6d0d4728 P b647e7fffe884a90850d540cb1a10ee5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3debe32907bf4023a01f1c9986a30d5b in term 0.
I20250626 13:58:23.400893 4058 leader_election.cc:304] T bf75d6d25d4842e3a51ea76c6d0d4728 P 3debe32907bf4023a01f1c9986a30d5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3debe32907bf4023a01f1c9986a30d5b, b647e7fffe884a90850d540cb1a10ee5; no voters:
I20250626 13:58:23.401217 4389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf75d6d25d4842e3a51ea76c6d0d4728" candidate_uuid: "3debe32907bf4023a01f1c9986a30d5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b6980c46ce8424fb743e0bec091e2a6" is_pre_election: true
I20250626 13:58:23.401561 4459 raft_consensus.cc:2802] T bf75d6d25d4842e3a51ea76c6d0d4728 P 3debe32907bf4023a01f1c9986a30d5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 13:58:23.401834 4459 raft_consensus.cc:491] T bf75d6d25d4842e3a51ea76c6d0d4728 P 3debe32907bf4023a01f1c9986a30d5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 13:58:23.401806 4389 raft_consensus.cc:2466] T bf75d6d25d4842e3a51ea76c6d0d4728 P 5b6980c46ce8424fb743e0bec091e2a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3debe32907bf4023a01f1c9986a30d5b in term 0.
I20250626 13:58:23.402077 4459 raft_consensus.cc:3058] T bf75d6d25d4842e3a51ea76c6d0d4728 P 3debe32907bf4023a01f1c9986a30d5b [term 0 FOLLOWER]: Advancing to term 1
I20250626 13:58:23.406471 4459 raft_consensus.cc:513] T bf75d6d25d4842e3a51ea76c6d0d4728 P 3debe32907bf4023a01f1c9986a30d5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3debe32907bf4023a01f1c9986a30d5b" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 43533 } } peers { permanent_uuid: "b647e7fffe884a90850d540cb1a10ee5" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 39263 } } peers { permanent_uuid: "5b6980c46ce8424fb743e0bec091e2a6" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 40277 } }
I20250626 13:58:23.407907 4459 leader_election.cc:290] T bf75d6d25d4842e3a51ea76c6d0d4728 P 3debe32907bf4023a01f1c9986a30d5b [CANDIDATE]: Term 1 election: Requested vote from peers b647e7fffe884a90850d540cb1a10ee5 (127.2.107.66:39263), 5b6980c46ce8424fb743e0bec091e2a6 (127.2.107.67:40277)
I20250626 13:58:23.408557 4256 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf75d6d25d4842e3a51ea76c6d0d4728" candidate_uuid: "3debe32907bf4023a01f1c9986a30d5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b647e7fffe884a90850d540cb1a10ee5"
I20250626 13:58:23.408891 4389 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf75d6d25d4842e3a51ea76c6d0d4728" candidate_uuid: "3debe32907bf4023a01f1c9986a30d5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b6980c46ce8424fb743e0bec091e2a6"
I20250626 13:58:23.409081 4256 raft_consensus.cc:3058] T bf75d6d25d4842e3a51ea76c6d0d4728 P b647e7fffe884a90850d540cb1a10ee5 [term 0 FOLLOWER]: Advancing to term 1
I20250626 13:58:23.409413 4389 raft_consensus.cc:3058] T bf75d6d25d4842e3a51ea76c6d0d4728 P 5b6980c46ce8424fb743e0bec091e2a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 13:58:23.415169 4256 raft_consensus.cc:2466] T bf75d6d25d4842e3a51ea76c6d0d4728 P b647e7fffe884a90850d540cb1a10ee5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3debe32907bf4023a01f1c9986a30d5b in term 1.
I20250626 13:58:23.415594 4389 raft_consensus.cc:2466] T bf75d6d25d4842e3a51ea76c6d0d4728 P 5b6980c46ce8424fb743e0bec091e2a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3debe32907bf4023a01f1c9986a30d5b in term 1.
I20250626 13:58:23.416158 4058 leader_election.cc:304] T bf75d6d25d4842e3a51ea76c6d0d4728 P 3debe32907bf4023a01f1c9986a30d5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3debe32907bf4023a01f1c9986a30d5b, b647e7fffe884a90850d540cb1a10ee5; no voters:
I20250626 13:58:23.416853 4459 raft_consensus.cc:2802] T bf75d6d25d4842e3a51ea76c6d0d4728 P 3debe32907bf4023a01f1c9986a30d5b [term 1 FOLLOWER]: Leader election won for term 1
I20250626 13:58:23.418450 4459 raft_consensus.cc:695] T bf75d6d25d4842e3a51ea76c6d0d4728 P 3debe32907bf4023a01f1c9986a30d5b [term 1 LEADER]: Becoming Leader. State: Replica: 3debe32907bf4023a01f1c9986a30d5b, State: Running, Role: LEADER
I20250626 13:58:23.419255 4459 consensus_queue.cc:237] T bf75d6d25d4842e3a51ea76c6d0d4728 P 3debe32907bf4023a01f1c9986a30d5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3debe32907bf4023a01f1c9986a30d5b" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 43533 } } peers { permanent_uuid: "b647e7fffe884a90850d540cb1a10ee5" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 39263 } } peers { permanent_uuid: "5b6980c46ce8424fb743e0bec091e2a6" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 40277 } }
I20250626 13:58:23.430068 3977 catalog_manager.cc:5582] T bf75d6d25d4842e3a51ea76c6d0d4728 P 3debe32907bf4023a01f1c9986a30d5b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3debe32907bf4023a01f1c9986a30d5b (127.2.107.65). New cstate: current_term: 1 leader_uuid: "3debe32907bf4023a01f1c9986a30d5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3debe32907bf4023a01f1c9986a30d5b" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 43533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b647e7fffe884a90850d540cb1a10ee5" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 39263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b6980c46ce8424fb743e0bec091e2a6" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 40277 } health_report { overall_health: UNKNOWN } } }
I20250626 13:58:23.480265 4461 consensus_queue.cc:1035] T fe77b00feedd471dade12d9d31d6e3fa P 5b6980c46ce8424fb743e0bec091e2a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b647e7fffe884a90850d540cb1a10ee5" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 39263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 13:58:23.504060 4461 consensus_queue.cc:1035] T fe77b00feedd471dade12d9d31d6e3fa P 5b6980c46ce8424fb743e0bec091e2a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3debe32907bf4023a01f1c9986a30d5b" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 13:58:23.774446 4473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:58:23.775041 4473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250626 13:58:23.804911 4459 consensus_queue.cc:1035] T bf75d6d25d4842e3a51ea76c6d0d4728 P 3debe32907bf4023a01f1c9986a30d5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b647e7fffe884a90850d540cb1a10ee5" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 39263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 13:58:23.807505 4473 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250626 13:58:23.816431 4486 consensus_queue.cc:1035] T bf75d6d25d4842e3a51ea76c6d0d4728 P 3debe32907bf4023a01f1c9986a30d5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b6980c46ce8424fb743e0bec091e2a6" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 40277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 13:58:25.008869 4473 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.157s user 0.419s sys 0.735s
W20250626 13:58:25.009272 4473 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.157s user 0.422s sys 0.735s
W20250626 13:58:25.445163 4430 debug-util.cc:398] Leaking SignalData structure 0x7b08000ac460 after lost signal to thread 4306
W20250626 13:58:25.445909 4430 debug-util.cc:398] Leaking SignalData structure 0x7b08000b2180 after lost signal to thread 4433
W20250626 13:58:26.386423 4498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:58:26.386999 4498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:58:26.417366 4498 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250626 13:58:27.759025 4498 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.301s user 0.543s sys 0.757s
W20250626 13:58:27.759402 4498 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.302s user 0.543s sys 0.757s
W20250626 13:58:29.120235 4512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:58:29.120852 4512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:58:29.151221 4512 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250626 13:58:30.320289 4512 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.129s user 0.524s sys 0.603s
W20250626 13:58:30.320685 4512 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.130s user 0.524s sys 0.603s
W20250626 13:58:31.682895 4530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:58:31.683470 4530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:58:31.714250 4530 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250626 13:58:32.890556 4530 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.136s user 0.445s sys 0.691s
W20250626 13:58:32.890951 4530 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.137s user 0.445s sys 0.691s
I20250626 13:58:33.966933 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 4039
I20250626 13:58:33.990506 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 4172
I20250626 13:58:34.012188 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 4305
I20250626 13:58:34.033990 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 3947
2025-06-26T13:58:34Z chronyd exiting
[ OK ] AdminCliTest.TestDescribeTableColumnFlags (18403 ms)
[ RUN ] AdminCliTest.TestAuthzResetCacheNotAuthorized
I20250626 13:58:34.085913 2477 test_util.cc:276] Using random seed: -183343618
I20250626 13:58:34.089994 2477 ts_itest-base.cc:115] Starting cluster with:
I20250626 13:58:34.090164 2477 ts_itest-base.cc:116] --------------
I20250626 13:58:34.090283 2477 ts_itest-base.cc:117] 3 tablet servers
I20250626 13:58:34.090390 2477 ts_itest-base.cc:118] 3 replicas per TS
I20250626 13:58:34.090502 2477 ts_itest-base.cc:119] --------------
2025-06-26T13:58:34Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-26T13:58:34Z Disabled control of system clock
I20250626 13:58:34.123373 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.107.126:40521
--webserver_interface=127.2.107.126
--webserver_port=0
--builtin_ntp_servers=127.2.107.84:41683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.107.126:40521
--superuser_acl=no-such-user with env {}
W20250626 13:58:34.411589 4552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:58:34.412354 4552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:58:34.412875 4552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:58:34.443255 4552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 13:58:34.443547 4552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:58:34.443758 4552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 13:58:34.443951 4552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 13:58:34.478699 4552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:41683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.107.126:40521
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.107.126:40521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.107.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:58:34.479959 4552 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:58:34.481611 4552 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:58:34.495091 4558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:58:34.495823 4559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:58:34.496549 4561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:58:34.496866 4552 server_base.cc:1048] running on GCE node
I20250626 13:58:35.587604 4552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:58:35.590597 4552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:58:35.592015 4552 hybrid_clock.cc:648] HybridClock initialized: now 1750946315591959 us; error 74 us; skew 500 ppm
I20250626 13:58:35.592787 4552 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:58:35.599192 4552 webserver.cc:469] Webserver started at http://127.2.107.126:36177/ using document root <none> and password file <none>
I20250626 13:58:35.600121 4552 fs_manager.cc:362] Metadata directory not provided
I20250626 13:58:35.600340 4552 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:58:35.600805 4552 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 13:58:35.605280 4552 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "b1403136ef92487e93699c3ad918f2ff"
format_stamp: "Formatted at 2025-06-26 13:58:35 on dist-test-slave-7p9x"
I20250626 13:58:35.606444 4552 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "b1403136ef92487e93699c3ad918f2ff"
format_stamp: "Formatted at 2025-06-26 13:58:35 on dist-test-slave-7p9x"
I20250626 13:58:35.613543 4552 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250626 13:58:35.618993 4568 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:35.620074 4552 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250626 13:58:35.620396 4552 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
uuid: "b1403136ef92487e93699c3ad918f2ff"
format_stamp: "Formatted at 2025-06-26 13:58:35 on dist-test-slave-7p9x"
I20250626 13:58:35.620748 4552 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:58:35.687456 4552 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:58:35.688961 4552 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:58:35.689395 4552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:58:35.755671 4552 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.126:40521
I20250626 13:58:35.755741 4619 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.126:40521 every 8 connection(s)
I20250626 13:58:35.758350 4552 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250626 13:58:35.763295 4620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:58:35.768568 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 4552
I20250626 13:58:35.768952 2477 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250626 13:58:35.781672 4620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1403136ef92487e93699c3ad918f2ff: Bootstrap starting.
I20250626 13:58:35.788233 4620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1403136ef92487e93699c3ad918f2ff: Neither blocks nor log segments found. Creating new log.
I20250626 13:58:35.790246 4620 log.cc:826] T 00000000000000000000000000000000 P b1403136ef92487e93699c3ad918f2ff: Log is configured to *not* fsync() on all Append() calls
I20250626 13:58:35.795017 4620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1403136ef92487e93699c3ad918f2ff: No bootstrap required, opened a new log
I20250626 13:58:35.812004 4620 raft_consensus.cc:357] T 00000000000000000000000000000000 P b1403136ef92487e93699c3ad918f2ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1403136ef92487e93699c3ad918f2ff" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 40521 } }
I20250626 13:58:35.812669 4620 raft_consensus.cc:383] T 00000000000000000000000000000000 P b1403136ef92487e93699c3ad918f2ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 13:58:35.812866 4620 raft_consensus.cc:738] T 00000000000000000000000000000000 P b1403136ef92487e93699c3ad918f2ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1403136ef92487e93699c3ad918f2ff, State: Initialized, Role: FOLLOWER
I20250626 13:58:35.813537 4620 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1403136ef92487e93699c3ad918f2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1403136ef92487e93699c3ad918f2ff" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 40521 } }
I20250626 13:58:35.814021 4620 raft_consensus.cc:397] T 00000000000000000000000000000000 P b1403136ef92487e93699c3ad918f2ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 13:58:35.814239 4620 raft_consensus.cc:491] T 00000000000000000000000000000000 P b1403136ef92487e93699c3ad918f2ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 13:58:35.814496 4620 raft_consensus.cc:3058] T 00000000000000000000000000000000 P b1403136ef92487e93699c3ad918f2ff [term 0 FOLLOWER]: Advancing to term 1
I20250626 13:58:35.818439 4620 raft_consensus.cc:513] T 00000000000000000000000000000000 P b1403136ef92487e93699c3ad918f2ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1403136ef92487e93699c3ad918f2ff" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 40521 } }
I20250626 13:58:35.819114 4620 leader_election.cc:304] T 00000000000000000000000000000000 P b1403136ef92487e93699c3ad918f2ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1403136ef92487e93699c3ad918f2ff; no voters:
I20250626 13:58:35.820721 4620 leader_election.cc:290] T 00000000000000000000000000000000 P b1403136ef92487e93699c3ad918f2ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 13:58:35.821507 4625 raft_consensus.cc:2802] T 00000000000000000000000000000000 P b1403136ef92487e93699c3ad918f2ff [term 1 FOLLOWER]: Leader election won for term 1
I20250626 13:58:35.823390 4625 raft_consensus.cc:695] T 00000000000000000000000000000000 P b1403136ef92487e93699c3ad918f2ff [term 1 LEADER]: Becoming Leader. State: Replica: b1403136ef92487e93699c3ad918f2ff, State: Running, Role: LEADER
I20250626 13:58:35.824129 4625 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1403136ef92487e93699c3ad918f2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1403136ef92487e93699c3ad918f2ff" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 40521 } }
I20250626 13:58:35.825068 4620 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1403136ef92487e93699c3ad918f2ff [sys.catalog]: configured and running, proceeding with master startup.
I20250626 13:58:35.833657 4626 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1403136ef92487e93699c3ad918f2ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1403136ef92487e93699c3ad918f2ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1403136ef92487e93699c3ad918f2ff" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 40521 } } }
I20250626 13:58:35.833815 4627 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1403136ef92487e93699c3ad918f2ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1403136ef92487e93699c3ad918f2ff. Latest consensus state: current_term: 1 leader_uuid: "b1403136ef92487e93699c3ad918f2ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1403136ef92487e93699c3ad918f2ff" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 40521 } } }
I20250626 13:58:35.834563 4626 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1403136ef92487e93699c3ad918f2ff [sys.catalog]: This master's current role is: LEADER
I20250626 13:58:35.834586 4627 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1403136ef92487e93699c3ad918f2ff [sys.catalog]: This master's current role is: LEADER
I20250626 13:58:35.837713 4634 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 13:58:35.848809 4634 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 13:58:35.867316 4634 catalog_manager.cc:1349] Generated new cluster ID: 4fcd4dfa3b534dd7bb32ba1574dcd00d
I20250626 13:58:35.867563 4634 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 13:58:35.901502 4634 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 13:58:35.903493 4634 catalog_manager.cc:1506] Loading token signing keys...
I20250626 13:58:35.919698 4634 catalog_manager.cc:5955] T 00000000000000000000000000000000 P b1403136ef92487e93699c3ad918f2ff: Generated new TSK 0
I20250626 13:58:35.920593 4634 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 13:58:35.932578 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.107.65:0
--local_ip_for_outbound_sockets=127.2.107.65
--webserver_interface=127.2.107.65
--webserver_port=0
--tserver_master_addrs=127.2.107.126:40521
--builtin_ntp_servers=127.2.107.84:41683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250626 13:58:36.247677 4644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:58:36.248184 4644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:58:36.248689 4644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:58:36.279932 4644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:58:36.280829 4644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.107.65
I20250626 13:58:36.316023 4644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:41683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.107.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.107.65
--webserver_port=0
--tserver_master_addrs=127.2.107.126:40521
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.107.65
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:58:36.317368 4644 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:58:36.319015 4644 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:58:36.335546 4651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:58:36.336666 4650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:58:36.335563 4653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:58:37.435048 4652 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 13:58:37.435226 4644 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 13:58:37.439358 4644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:58:37.442082 4644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:58:37.443569 4644 hybrid_clock.cc:648] HybridClock initialized: now 1750946317443520 us; error 41 us; skew 500 ppm
I20250626 13:58:37.444619 4644 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:58:37.452229 4644 webserver.cc:469] Webserver started at http://127.2.107.65:45327/ using document root <none> and password file <none>
I20250626 13:58:37.453541 4644 fs_manager.cc:362] Metadata directory not provided
I20250626 13:58:37.453827 4644 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:58:37.454402 4644 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 13:58:37.461035 4644 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "02fca4dfb1ef4ed8a2d1332728020efa"
format_stamp: "Formatted at 2025-06-26 13:58:37 on dist-test-slave-7p9x"
I20250626 13:58:37.462558 4644 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "02fca4dfb1ef4ed8a2d1332728020efa"
format_stamp: "Formatted at 2025-06-26 13:58:37 on dist-test-slave-7p9x"
I20250626 13:58:37.471635 4644 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20250626 13:58:37.478704 4660 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:37.479877 4644 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250626 13:58:37.480266 4644 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "02fca4dfb1ef4ed8a2d1332728020efa"
format_stamp: "Formatted at 2025-06-26 13:58:37 on dist-test-slave-7p9x"
I20250626 13:58:37.480690 4644 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:58:37.551033 4644 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:58:37.552613 4644 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:58:37.553022 4644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:58:37.555601 4644 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 13:58:37.559707 4644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 13:58:37.559913 4644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:37.560132 4644 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 13:58:37.560310 4644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:37.703719 4644 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.65:33637
I20250626 13:58:37.703800 4772 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.65:33637 every 8 connection(s)
I20250626 13:58:37.706506 4644 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250626 13:58:37.713733 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 4644
I20250626 13:58:37.714115 2477 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250626 13:58:37.719771 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.107.66:0
--local_ip_for_outbound_sockets=127.2.107.66
--webserver_interface=127.2.107.66
--webserver_port=0
--tserver_master_addrs=127.2.107.126:40521
--builtin_ntp_servers=127.2.107.84:41683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 13:58:37.727530 4773 heartbeater.cc:344] Connected to a master server at 127.2.107.126:40521
I20250626 13:58:37.728022 4773 heartbeater.cc:461] Registering TS with master...
I20250626 13:58:37.729301 4773 heartbeater.cc:507] Master 127.2.107.126:40521 requested a full tablet report, sending...
I20250626 13:58:37.731894 4585 ts_manager.cc:194] Registered new tserver with Master: 02fca4dfb1ef4ed8a2d1332728020efa (127.2.107.65:33637)
I20250626 13:58:37.733896 4585 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.107.65:58299
W20250626 13:58:38.014242 4777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:58:38.014747 4777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:58:38.015254 4777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:58:38.046002 4777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:58:38.046854 4777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.107.66
I20250626 13:58:38.081463 4777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:41683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.107.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.107.66
--webserver_port=0
--tserver_master_addrs=127.2.107.126:40521
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.107.66
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:58:38.082728 4777 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:58:38.084342 4777 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:58:38.099400 4783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:58:38.737257 4773 heartbeater.cc:499] Master 127.2.107.126:40521 was elected leader, sending a full tablet report...
W20250626 13:58:38.099406 4786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:58:38.099547 4784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:58:39.169710 4785 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 13:58:39.169857 4777 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 13:58:39.174202 4777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:58:39.176800 4777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:58:39.178254 4777 hybrid_clock.cc:648] HybridClock initialized: now 1750946319178222 us; error 61 us; skew 500 ppm
I20250626 13:58:39.179090 4777 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:58:39.189364 4777 webserver.cc:469] Webserver started at http://127.2.107.66:36889/ using document root <none> and password file <none>
I20250626 13:58:39.190270 4777 fs_manager.cc:362] Metadata directory not provided
I20250626 13:58:39.190485 4777 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:58:39.190920 4777 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 13:58:39.195417 4777 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "3c9ec6bdd59442fb95e5457cc48f3300"
format_stamp: "Formatted at 2025-06-26 13:58:39 on dist-test-slave-7p9x"
I20250626 13:58:39.196491 4777 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "3c9ec6bdd59442fb95e5457cc48f3300"
format_stamp: "Formatted at 2025-06-26 13:58:39 on dist-test-slave-7p9x"
I20250626 13:58:39.203460 4777 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20250626 13:58:39.209218 4793 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:39.210306 4777 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250626 13:58:39.210625 4777 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "3c9ec6bdd59442fb95e5457cc48f3300"
format_stamp: "Formatted at 2025-06-26 13:58:39 on dist-test-slave-7p9x"
I20250626 13:58:39.210947 4777 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:58:39.258973 4777 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:58:39.260432 4777 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:58:39.260843 4777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:58:39.263296 4777 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 13:58:39.267287 4777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 13:58:39.267490 4777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:39.267720 4777 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 13:58:39.267880 4777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:39.393934 4777 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.66:33377
I20250626 13:58:39.394025 4905 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.66:33377 every 8 connection(s)
I20250626 13:58:39.396430 4777 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250626 13:58:39.398264 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 4777
I20250626 13:58:39.398756 2477 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250626 13:58:39.406556 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.107.67:0
--local_ip_for_outbound_sockets=127.2.107.67
--webserver_interface=127.2.107.67
--webserver_port=0
--tserver_master_addrs=127.2.107.126:40521
--builtin_ntp_servers=127.2.107.84:41683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 13:58:39.420703 4906 heartbeater.cc:344] Connected to a master server at 127.2.107.126:40521
I20250626 13:58:39.421283 4906 heartbeater.cc:461] Registering TS with master...
I20250626 13:58:39.422614 4906 heartbeater.cc:507] Master 127.2.107.126:40521 requested a full tablet report, sending...
I20250626 13:58:39.425107 4585 ts_manager.cc:194] Registered new tserver with Master: 3c9ec6bdd59442fb95e5457cc48f3300 (127.2.107.66:33377)
I20250626 13:58:39.426406 4585 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.107.66:33037
W20250626 13:58:39.698866 4910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:58:39.699384 4910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:58:39.699874 4910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:58:39.731740 4910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:58:39.732589 4910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.107.67
I20250626 13:58:39.768126 4910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:41683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.107.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.107.67
--webserver_port=0
--tserver_master_addrs=127.2.107.126:40521
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.107.67
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:58:39.769498 4910 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:58:39.771108 4910 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:58:39.786355 4919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:58:40.430190 4906 heartbeater.cc:499] Master 127.2.107.126:40521 was elected leader, sending a full tablet report...
W20250626 13:58:39.786375 4916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:58:39.787551 4917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:58:40.858690 4918 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 13:58:40.858747 4910 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 13:58:40.862814 4910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:58:40.865303 4910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:58:40.866730 4910 hybrid_clock.cc:648] HybridClock initialized: now 1750946320866631 us; error 91 us; skew 500 ppm
I20250626 13:58:40.867619 4910 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:58:40.874428 4910 webserver.cc:469] Webserver started at http://127.2.107.67:32897/ using document root <none> and password file <none>
I20250626 13:58:40.875314 4910 fs_manager.cc:362] Metadata directory not provided
I20250626 13:58:40.875519 4910 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:58:40.875953 4910 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 13:58:40.880301 4910 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "d399d51260634042b24021ea865b15ec"
format_stamp: "Formatted at 2025-06-26 13:58:40 on dist-test-slave-7p9x"
I20250626 13:58:40.881608 4910 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "d399d51260634042b24021ea865b15ec"
format_stamp: "Formatted at 2025-06-26 13:58:40 on dist-test-slave-7p9x"
I20250626 13:58:40.888617 4910 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20250626 13:58:40.893973 4926 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:40.894989 4910 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250626 13:58:40.895262 4910 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "d399d51260634042b24021ea865b15ec"
format_stamp: "Formatted at 2025-06-26 13:58:40 on dist-test-slave-7p9x"
I20250626 13:58:40.895565 4910 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:58:40.949841 4910 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:58:40.951243 4910 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:58:40.951627 4910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:58:40.954159 4910 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 13:58:40.958083 4910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 13:58:40.958274 4910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:40.958464 4910 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 13:58:40.958616 4910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:41.084179 4910 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.67:32829
I20250626 13:58:41.084290 5038 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.67:32829 every 8 connection(s)
I20250626 13:58:41.086642 4910 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250626 13:58:41.096285 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 4910
I20250626 13:58:41.096652 2477 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250626 13:58:41.106307 5039 heartbeater.cc:344] Connected to a master server at 127.2.107.126:40521
I20250626 13:58:41.106721 5039 heartbeater.cc:461] Registering TS with master...
I20250626 13:58:41.107671 5039 heartbeater.cc:507] Master 127.2.107.126:40521 requested a full tablet report, sending...
I20250626 13:58:41.109630 4585 ts_manager.cc:194] Registered new tserver with Master: d399d51260634042b24021ea865b15ec (127.2.107.67:32829)
I20250626 13:58:41.110772 4585 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.107.67:44963
I20250626 13:58:41.116209 2477 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 13:58:41.145191 4585 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36558:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250626 13:58:41.164078 4585 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 13:58:41.219178 4708 tablet_service.cc:1468] Processing CreateTablet for tablet dfd6f1fb60c8438090dbec623dfddedd (DEFAULT_TABLE table=TestTable [id=200dcfc52def4ce5914805822f100ce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 13:58:41.220022 4841 tablet_service.cc:1468] Processing CreateTablet for tablet dfd6f1fb60c8438090dbec623dfddedd (DEFAULT_TABLE table=TestTable [id=200dcfc52def4ce5914805822f100ce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 13:58:41.221096 4708 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfd6f1fb60c8438090dbec623dfddedd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:58:41.221462 4841 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfd6f1fb60c8438090dbec623dfddedd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:58:41.221482 4974 tablet_service.cc:1468] Processing CreateTablet for tablet dfd6f1fb60c8438090dbec623dfddedd (DEFAULT_TABLE table=TestTable [id=200dcfc52def4ce5914805822f100ce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 13:58:41.223198 4974 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfd6f1fb60c8438090dbec623dfddedd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:58:41.245180 5058 tablet_bootstrap.cc:492] T dfd6f1fb60c8438090dbec623dfddedd P 3c9ec6bdd59442fb95e5457cc48f3300: Bootstrap starting.
I20250626 13:58:41.247975 5059 tablet_bootstrap.cc:492] T dfd6f1fb60c8438090dbec623dfddedd P 02fca4dfb1ef4ed8a2d1332728020efa: Bootstrap starting.
I20250626 13:58:41.250306 5060 tablet_bootstrap.cc:492] T dfd6f1fb60c8438090dbec623dfddedd P d399d51260634042b24021ea865b15ec: Bootstrap starting.
I20250626 13:58:41.253687 5058 tablet_bootstrap.cc:654] T dfd6f1fb60c8438090dbec623dfddedd P 3c9ec6bdd59442fb95e5457cc48f3300: Neither blocks nor log segments found. Creating new log.
I20250626 13:58:41.255300 5059 tablet_bootstrap.cc:654] T dfd6f1fb60c8438090dbec623dfddedd P 02fca4dfb1ef4ed8a2d1332728020efa: Neither blocks nor log segments found. Creating new log.
I20250626 13:58:41.256026 5058 log.cc:826] T dfd6f1fb60c8438090dbec623dfddedd P 3c9ec6bdd59442fb95e5457cc48f3300: Log is configured to *not* fsync() on all Append() calls
I20250626 13:58:41.256273 5060 tablet_bootstrap.cc:654] T dfd6f1fb60c8438090dbec623dfddedd P d399d51260634042b24021ea865b15ec: Neither blocks nor log segments found. Creating new log.
I20250626 13:58:41.257557 5059 log.cc:826] T dfd6f1fb60c8438090dbec623dfddedd P 02fca4dfb1ef4ed8a2d1332728020efa: Log is configured to *not* fsync() on all Append() calls
I20250626 13:58:41.258262 5060 log.cc:826] T dfd6f1fb60c8438090dbec623dfddedd P d399d51260634042b24021ea865b15ec: Log is configured to *not* fsync() on all Append() calls
I20250626 13:58:41.261682 5058 tablet_bootstrap.cc:492] T dfd6f1fb60c8438090dbec623dfddedd P 3c9ec6bdd59442fb95e5457cc48f3300: No bootstrap required, opened a new log
I20250626 13:58:41.262198 5058 ts_tablet_manager.cc:1397] T dfd6f1fb60c8438090dbec623dfddedd P 3c9ec6bdd59442fb95e5457cc48f3300: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.009s
I20250626 13:58:41.263731 5059 tablet_bootstrap.cc:492] T dfd6f1fb60c8438090dbec623dfddedd P 02fca4dfb1ef4ed8a2d1332728020efa: No bootstrap required, opened a new log
I20250626 13:58:41.263803 5060 tablet_bootstrap.cc:492] T dfd6f1fb60c8438090dbec623dfddedd P d399d51260634042b24021ea865b15ec: No bootstrap required, opened a new log
I20250626 13:58:41.264233 5059 ts_tablet_manager.cc:1397] T dfd6f1fb60c8438090dbec623dfddedd P 02fca4dfb1ef4ed8a2d1332728020efa: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.013s
I20250626 13:58:41.264312 5060 ts_tablet_manager.cc:1397] T dfd6f1fb60c8438090dbec623dfddedd P d399d51260634042b24021ea865b15ec: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.005s
I20250626 13:58:41.282106 5059 raft_consensus.cc:357] T dfd6f1fb60c8438090dbec623dfddedd P 02fca4dfb1ef4ed8a2d1332728020efa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fca4dfb1ef4ed8a2d1332728020efa" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 33637 } } peers { permanent_uuid: "3c9ec6bdd59442fb95e5457cc48f3300" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 33377 } } peers { permanent_uuid: "d399d51260634042b24021ea865b15ec" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32829 } }
I20250626 13:58:41.282794 5059 raft_consensus.cc:383] T dfd6f1fb60c8438090dbec623dfddedd P 02fca4dfb1ef4ed8a2d1332728020efa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 13:58:41.283046 5059 raft_consensus.cc:738] T dfd6f1fb60c8438090dbec623dfddedd P 02fca4dfb1ef4ed8a2d1332728020efa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fca4dfb1ef4ed8a2d1332728020efa, State: Initialized, Role: FOLLOWER
I20250626 13:58:41.283828 5059 consensus_queue.cc:260] T dfd6f1fb60c8438090dbec623dfddedd P 02fca4dfb1ef4ed8a2d1332728020efa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fca4dfb1ef4ed8a2d1332728020efa" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 33637 } } peers { permanent_uuid: "3c9ec6bdd59442fb95e5457cc48f3300" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 33377 } } peers { permanent_uuid: "d399d51260634042b24021ea865b15ec" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32829 } }
I20250626 13:58:41.287739 5059 ts_tablet_manager.cc:1428] T dfd6f1fb60c8438090dbec623dfddedd P 02fca4dfb1ef4ed8a2d1332728020efa: Time spent starting tablet: real 0.023s user 0.020s sys 0.003s
I20250626 13:58:41.288995 5058 raft_consensus.cc:357] T dfd6f1fb60c8438090dbec623dfddedd P 3c9ec6bdd59442fb95e5457cc48f3300 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fca4dfb1ef4ed8a2d1332728020efa" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 33637 } } peers { permanent_uuid: "3c9ec6bdd59442fb95e5457cc48f3300" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 33377 } } peers { permanent_uuid: "d399d51260634042b24021ea865b15ec" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32829 } }
I20250626 13:58:41.289885 5058 raft_consensus.cc:383] T dfd6f1fb60c8438090dbec623dfddedd P 3c9ec6bdd59442fb95e5457cc48f3300 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 13:58:41.290195 5058 raft_consensus.cc:738] T dfd6f1fb60c8438090dbec623dfddedd P 3c9ec6bdd59442fb95e5457cc48f3300 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9ec6bdd59442fb95e5457cc48f3300, State: Initialized, Role: FOLLOWER
I20250626 13:58:41.289954 5060 raft_consensus.cc:357] T dfd6f1fb60c8438090dbec623dfddedd P d399d51260634042b24021ea865b15ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fca4dfb1ef4ed8a2d1332728020efa" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 33637 } } peers { permanent_uuid: "3c9ec6bdd59442fb95e5457cc48f3300" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 33377 } } peers { permanent_uuid: "d399d51260634042b24021ea865b15ec" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32829 } }
I20250626 13:58:41.290767 5060 raft_consensus.cc:383] T dfd6f1fb60c8438090dbec623dfddedd P d399d51260634042b24021ea865b15ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 13:58:41.291071 5060 raft_consensus.cc:738] T dfd6f1fb60c8438090dbec623dfddedd P d399d51260634042b24021ea865b15ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d399d51260634042b24021ea865b15ec, State: Initialized, Role: FOLLOWER
I20250626 13:58:41.291188 5058 consensus_queue.cc:260] T dfd6f1fb60c8438090dbec623dfddedd P 3c9ec6bdd59442fb95e5457cc48f3300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fca4dfb1ef4ed8a2d1332728020efa" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 33637 } } peers { permanent_uuid: "3c9ec6bdd59442fb95e5457cc48f3300" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 33377 } } peers { permanent_uuid: "d399d51260634042b24021ea865b15ec" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32829 } }
I20250626 13:58:41.291900 5060 consensus_queue.cc:260] T dfd6f1fb60c8438090dbec623dfddedd P d399d51260634042b24021ea865b15ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fca4dfb1ef4ed8a2d1332728020efa" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 33637 } } peers { permanent_uuid: "3c9ec6bdd59442fb95e5457cc48f3300" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 33377 } } peers { permanent_uuid: "d399d51260634042b24021ea865b15ec" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32829 } }
I20250626 13:58:41.295336 5058 ts_tablet_manager.cc:1428] T dfd6f1fb60c8438090dbec623dfddedd P 3c9ec6bdd59442fb95e5457cc48f3300: Time spent starting tablet: real 0.033s user 0.027s sys 0.004s
I20250626 13:58:41.297941 5060 ts_tablet_manager.cc:1428] T dfd6f1fb60c8438090dbec623dfddedd P d399d51260634042b24021ea865b15ec: Time spent starting tablet: real 0.033s user 0.021s sys 0.011s
I20250626 13:58:41.298573 5039 heartbeater.cc:499] Master 127.2.107.126:40521 was elected leader, sending a full tablet report...
W20250626 13:58:41.340835 5040 tablet.cc:2378] T dfd6f1fb60c8438090dbec623dfddedd P d399d51260634042b24021ea865b15ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 13:58:41.390417 5064 raft_consensus.cc:491] T dfd6f1fb60c8438090dbec623dfddedd P 02fca4dfb1ef4ed8a2d1332728020efa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 13:58:41.390894 5064 raft_consensus.cc:513] T dfd6f1fb60c8438090dbec623dfddedd P 02fca4dfb1ef4ed8a2d1332728020efa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fca4dfb1ef4ed8a2d1332728020efa" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 33637 } } peers { permanent_uuid: "3c9ec6bdd59442fb95e5457cc48f3300" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 33377 } } peers { permanent_uuid: "d399d51260634042b24021ea865b15ec" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32829 } }
I20250626 13:58:41.394026 5064 leader_election.cc:290] T dfd6f1fb60c8438090dbec623dfddedd P 02fca4dfb1ef4ed8a2d1332728020efa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c9ec6bdd59442fb95e5457cc48f3300 (127.2.107.66:33377), d399d51260634042b24021ea865b15ec (127.2.107.67:32829)
W20250626 13:58:41.402868 4907 tablet.cc:2378] T dfd6f1fb60c8438090dbec623dfddedd P 3c9ec6bdd59442fb95e5457cc48f3300: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 13:58:41.411811 4861 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfd6f1fb60c8438090dbec623dfddedd" candidate_uuid: "02fca4dfb1ef4ed8a2d1332728020efa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9ec6bdd59442fb95e5457cc48f3300" is_pre_election: true
I20250626 13:58:41.412726 4861 raft_consensus.cc:2466] T dfd6f1fb60c8438090dbec623dfddedd P 3c9ec6bdd59442fb95e5457cc48f3300 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fca4dfb1ef4ed8a2d1332728020efa in term 0.
I20250626 13:58:41.414139 4661 leader_election.cc:304] T dfd6f1fb60c8438090dbec623dfddedd P 02fca4dfb1ef4ed8a2d1332728020efa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fca4dfb1ef4ed8a2d1332728020efa, 3c9ec6bdd59442fb95e5457cc48f3300; no voters:
I20250626 13:58:41.415066 5064 raft_consensus.cc:2802] T dfd6f1fb60c8438090dbec623dfddedd P 02fca4dfb1ef4ed8a2d1332728020efa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 13:58:41.415364 5064 raft_consensus.cc:491] T dfd6f1fb60c8438090dbec623dfddedd P 02fca4dfb1ef4ed8a2d1332728020efa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 13:58:41.415679 5064 raft_consensus.cc:3058] T dfd6f1fb60c8438090dbec623dfddedd P 02fca4dfb1ef4ed8a2d1332728020efa [term 0 FOLLOWER]: Advancing to term 1
I20250626 13:58:41.415906 4994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfd6f1fb60c8438090dbec623dfddedd" candidate_uuid: "02fca4dfb1ef4ed8a2d1332728020efa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d399d51260634042b24021ea865b15ec" is_pre_election: true
I20250626 13:58:41.416736 4994 raft_consensus.cc:2466] T dfd6f1fb60c8438090dbec623dfddedd P d399d51260634042b24021ea865b15ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fca4dfb1ef4ed8a2d1332728020efa in term 0.
I20250626 13:58:41.420676 5064 raft_consensus.cc:513] T dfd6f1fb60c8438090dbec623dfddedd P 02fca4dfb1ef4ed8a2d1332728020efa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fca4dfb1ef4ed8a2d1332728020efa" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 33637 } } peers { permanent_uuid: "3c9ec6bdd59442fb95e5457cc48f3300" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 33377 } } peers { permanent_uuid: "d399d51260634042b24021ea865b15ec" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32829 } }
I20250626 13:58:41.422130 5064 leader_election.cc:290] T dfd6f1fb60c8438090dbec623dfddedd P 02fca4dfb1ef4ed8a2d1332728020efa [CANDIDATE]: Term 1 election: Requested vote from peers 3c9ec6bdd59442fb95e5457cc48f3300 (127.2.107.66:33377), d399d51260634042b24021ea865b15ec (127.2.107.67:32829)
I20250626 13:58:41.422871 4861 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfd6f1fb60c8438090dbec623dfddedd" candidate_uuid: "02fca4dfb1ef4ed8a2d1332728020efa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9ec6bdd59442fb95e5457cc48f3300"
I20250626 13:58:41.423081 4994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfd6f1fb60c8438090dbec623dfddedd" candidate_uuid: "02fca4dfb1ef4ed8a2d1332728020efa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d399d51260634042b24021ea865b15ec"
I20250626 13:58:41.423405 4861 raft_consensus.cc:3058] T dfd6f1fb60c8438090dbec623dfddedd P 3c9ec6bdd59442fb95e5457cc48f3300 [term 0 FOLLOWER]: Advancing to term 1
I20250626 13:58:41.423550 4994 raft_consensus.cc:3058] T dfd6f1fb60c8438090dbec623dfddedd P d399d51260634042b24021ea865b15ec [term 0 FOLLOWER]: Advancing to term 1
I20250626 13:58:41.430239 4861 raft_consensus.cc:2466] T dfd6f1fb60c8438090dbec623dfddedd P 3c9ec6bdd59442fb95e5457cc48f3300 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fca4dfb1ef4ed8a2d1332728020efa in term 1.
I20250626 13:58:41.430239 4994 raft_consensus.cc:2466] T dfd6f1fb60c8438090dbec623dfddedd P d399d51260634042b24021ea865b15ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fca4dfb1ef4ed8a2d1332728020efa in term 1.
I20250626 13:58:41.431442 4662 leader_election.cc:304] T dfd6f1fb60c8438090dbec623dfddedd P 02fca4dfb1ef4ed8a2d1332728020efa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fca4dfb1ef4ed8a2d1332728020efa, d399d51260634042b24021ea865b15ec; no voters:
I20250626 13:58:41.432049 5064 raft_consensus.cc:2802] T dfd6f1fb60c8438090dbec623dfddedd P 02fca4dfb1ef4ed8a2d1332728020efa [term 1 FOLLOWER]: Leader election won for term 1
I20250626 13:58:41.433840 5064 raft_consensus.cc:695] T dfd6f1fb60c8438090dbec623dfddedd P 02fca4dfb1ef4ed8a2d1332728020efa [term 1 LEADER]: Becoming Leader. State: Replica: 02fca4dfb1ef4ed8a2d1332728020efa, State: Running, Role: LEADER
I20250626 13:58:41.434643 5064 consensus_queue.cc:237] T dfd6f1fb60c8438090dbec623dfddedd P 02fca4dfb1ef4ed8a2d1332728020efa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fca4dfb1ef4ed8a2d1332728020efa" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 33637 } } peers { permanent_uuid: "3c9ec6bdd59442fb95e5457cc48f3300" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 33377 } } peers { permanent_uuid: "d399d51260634042b24021ea865b15ec" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32829 } }
I20250626 13:58:41.445689 4584 catalog_manager.cc:5582] T dfd6f1fb60c8438090dbec623dfddedd P 02fca4dfb1ef4ed8a2d1332728020efa reported cstate change: term changed from 0 to 1, leader changed from <none> to 02fca4dfb1ef4ed8a2d1332728020efa (127.2.107.65). New cstate: current_term: 1 leader_uuid: "02fca4dfb1ef4ed8a2d1332728020efa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fca4dfb1ef4ed8a2d1332728020efa" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 33637 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c9ec6bdd59442fb95e5457cc48f3300" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 33377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d399d51260634042b24021ea865b15ec" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32829 } health_report { overall_health: UNKNOWN } } }
W20250626 13:58:41.464200 4774 tablet.cc:2378] T dfd6f1fb60c8438090dbec623dfddedd P 02fca4dfb1ef4ed8a2d1332728020efa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 13:58:41.468708 2477 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 13:58:41.471894 2477 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 02fca4dfb1ef4ed8a2d1332728020efa to finish bootstrapping
I20250626 13:58:41.484265 2477 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 3c9ec6bdd59442fb95e5457cc48f3300 to finish bootstrapping
I20250626 13:58:41.494342 2477 ts_itest-base.cc:246] Waiting for 1 tablets on tserver d399d51260634042b24021ea865b15ec to finish bootstrapping
I20250626 13:58:41.837085 5064 consensus_queue.cc:1035] T dfd6f1fb60c8438090dbec623dfddedd P 02fca4dfb1ef4ed8a2d1332728020efa [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9ec6bdd59442fb95e5457cc48f3300" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 33377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 13:58:41.860392 5069 consensus_queue.cc:1035] T dfd6f1fb60c8438090dbec623dfddedd P 02fca4dfb1ef4ed8a2d1332728020efa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d399d51260634042b24021ea865b15ec" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 13:58:43.020941 4584 server_base.cc:1130] Unauthorized access attempt to method kudu.master.MasterService.RefreshAuthzCache from {username='slave'} at 127.0.0.1:36568
I20250626 13:58:44.051044 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 4644
I20250626 13:58:44.071806 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 4777
I20250626 13:58:44.093920 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 4910
I20250626 13:58:44.112943 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 4552
2025-06-26T13:58:44Z chronyd exiting
[ OK ] AdminCliTest.TestAuthzResetCacheNotAuthorized (10079 ms)
[ RUN ] AdminCliTest.TestRebuildTables
I20250626 13:58:44.165896 2477 test_util.cc:276] Using random seed: -173263633
I20250626 13:58:44.169831 2477 ts_itest-base.cc:115] Starting cluster with:
I20250626 13:58:44.169984 2477 ts_itest-base.cc:116] --------------
I20250626 13:58:44.170096 2477 ts_itest-base.cc:117] 3 tablet servers
I20250626 13:58:44.170202 2477 ts_itest-base.cc:118] 3 replicas per TS
I20250626 13:58:44.170316 2477 ts_itest-base.cc:119] --------------
2025-06-26T13:58:44Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-26T13:58:44Z Disabled control of system clock
I20250626 13:58:44.202508 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.107.126:39915
--webserver_interface=127.2.107.126
--webserver_port=0
--builtin_ntp_servers=127.2.107.84:40679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.107.126:39915 with env {}
W20250626 13:58:44.487236 5108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:58:44.487787 5108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:58:44.488186 5108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:58:44.518484 5108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 13:58:44.518766 5108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:58:44.518965 5108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 13:58:44.519155 5108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 13:58:44.553793 5108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:40679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.107.126:39915
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.107.126:39915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.107.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:58:44.555058 5108 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:58:44.556625 5108 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:58:44.570497 5117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:58:44.570640 5115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:58:44.571659 5108 server_base.cc:1048] running on GCE node
W20250626 13:58:44.571273 5114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:58:45.662213 5108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:58:45.664749 5108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:58:45.666131 5108 hybrid_clock.cc:648] HybridClock initialized: now 1750946325666091 us; error 51 us; skew 500 ppm
I20250626 13:58:45.666982 5108 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:58:45.676280 5108 webserver.cc:469] Webserver started at http://127.2.107.126:45001/ using document root <none> and password file <none>
I20250626 13:58:45.677242 5108 fs_manager.cc:362] Metadata directory not provided
I20250626 13:58:45.677461 5108 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:58:45.677944 5108 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 13:58:45.682444 5108 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "dff2ce1fa5a3439a92429a6dd10c6097"
format_stamp: "Formatted at 2025-06-26 13:58:45 on dist-test-slave-7p9x"
I20250626 13:58:45.683516 5108 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "dff2ce1fa5a3439a92429a6dd10c6097"
format_stamp: "Formatted at 2025-06-26 13:58:45 on dist-test-slave-7p9x"
I20250626 13:58:45.690534 5108 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20250626 13:58:45.695987 5124 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:45.697041 5108 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250626 13:58:45.697409 5108 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
uuid: "dff2ce1fa5a3439a92429a6dd10c6097"
format_stamp: "Formatted at 2025-06-26 13:58:45 on dist-test-slave-7p9x"
I20250626 13:58:45.697754 5108 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:58:45.747334 5108 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:58:45.748792 5108 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:58:45.749253 5108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:58:45.815083 5108 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.126:39915
I20250626 13:58:45.815169 5175 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.126:39915 every 8 connection(s)
I20250626 13:58:45.817688 5108 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250626 13:58:45.822619 5176 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:58:45.824086 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 5108
I20250626 13:58:45.824548 2477 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250626 13:58:45.842890 5176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097: Bootstrap starting.
I20250626 13:58:45.849020 5176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097: Neither blocks nor log segments found. Creating new log.
I20250626 13:58:45.850713 5176 log.cc:826] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097: Log is configured to *not* fsync() on all Append() calls
I20250626 13:58:45.855221 5176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097: No bootstrap required, opened a new log
I20250626 13:58:45.872833 5176 raft_consensus.cc:357] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff2ce1fa5a3439a92429a6dd10c6097" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 39915 } }
I20250626 13:58:45.873569 5176 raft_consensus.cc:383] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 13:58:45.873800 5176 raft_consensus.cc:738] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff2ce1fa5a3439a92429a6dd10c6097, State: Initialized, Role: FOLLOWER
I20250626 13:58:45.874475 5176 consensus_queue.cc:260] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff2ce1fa5a3439a92429a6dd10c6097" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 39915 } }
I20250626 13:58:45.874971 5176 raft_consensus.cc:397] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 13:58:45.875226 5176 raft_consensus.cc:491] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 13:58:45.875515 5176 raft_consensus.cc:3058] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [term 0 FOLLOWER]: Advancing to term 1
I20250626 13:58:45.880048 5176 raft_consensus.cc:513] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff2ce1fa5a3439a92429a6dd10c6097" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 39915 } }
I20250626 13:58:45.880730 5176 leader_election.cc:304] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dff2ce1fa5a3439a92429a6dd10c6097; no voters:
I20250626 13:58:45.882455 5176 leader_election.cc:290] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 13:58:45.883291 5181 raft_consensus.cc:2802] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 13:58:45.885613 5181 raft_consensus.cc:695] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [term 1 LEADER]: Becoming Leader. State: Replica: dff2ce1fa5a3439a92429a6dd10c6097, State: Running, Role: LEADER
I20250626 13:58:45.886503 5181 consensus_queue.cc:237] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff2ce1fa5a3439a92429a6dd10c6097" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 39915 } }
I20250626 13:58:45.887144 5176 sys_catalog.cc:564] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 13:58:45.896167 5183 sys_catalog.cc:455] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dff2ce1fa5a3439a92429a6dd10c6097. Latest consensus state: current_term: 1 leader_uuid: "dff2ce1fa5a3439a92429a6dd10c6097" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff2ce1fa5a3439a92429a6dd10c6097" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 39915 } } }
I20250626 13:58:45.896440 5182 sys_catalog.cc:455] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dff2ce1fa5a3439a92429a6dd10c6097" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff2ce1fa5a3439a92429a6dd10c6097" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 39915 } } }
I20250626 13:58:45.897118 5182 sys_catalog.cc:458] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [sys.catalog]: This master's current role is: LEADER
I20250626 13:58:45.897106 5183 sys_catalog.cc:458] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [sys.catalog]: This master's current role is: LEADER
I20250626 13:58:45.902034 5189 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 13:58:45.915871 5189 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 13:58:45.930620 5189 catalog_manager.cc:1349] Generated new cluster ID: fdd2a3c7727c496ca7c541014bb33fe4
I20250626 13:58:45.930943 5189 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 13:58:45.947448 5189 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 13:58:45.948915 5189 catalog_manager.cc:1506] Loading token signing keys...
I20250626 13:58:45.969059 5189 catalog_manager.cc:5955] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097: Generated new TSK 0
I20250626 13:58:45.970247 5189 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 13:58:45.992609 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.107.65:0
--local_ip_for_outbound_sockets=127.2.107.65
--webserver_interface=127.2.107.65
--webserver_port=0
--tserver_master_addrs=127.2.107.126:39915
--builtin_ntp_servers=127.2.107.84:40679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250626 13:58:46.286370 5200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:58:46.286873 5200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:58:46.287356 5200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:58:46.318693 5200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:58:46.319586 5200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.107.65
I20250626 13:58:46.354339 5200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:40679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.107.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.107.65
--webserver_port=0
--tserver_master_addrs=127.2.107.126:39915
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.107.65
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:58:46.355672 5200 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:58:46.357326 5200 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:58:46.375128 5207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:58:46.375926 5206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:58:46.376813 5200 server_base.cc:1048] running on GCE node
W20250626 13:58:46.376677 5209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:58:47.496253 5200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:58:47.498852 5200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:58:47.500281 5200 hybrid_clock.cc:648] HybridClock initialized: now 1750946327500210 us; error 89 us; skew 500 ppm
I20250626 13:58:47.501065 5200 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:58:47.507910 5200 webserver.cc:469] Webserver started at http://127.2.107.65:42549/ using document root <none> and password file <none>
I20250626 13:58:47.508802 5200 fs_manager.cc:362] Metadata directory not provided
I20250626 13:58:47.509024 5200 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:58:47.509635 5200 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 13:58:47.514097 5200 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "adfc75fd62a24908b775fe41af11b077"
format_stamp: "Formatted at 2025-06-26 13:58:47 on dist-test-slave-7p9x"
I20250626 13:58:47.515200 5200 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "adfc75fd62a24908b775fe41af11b077"
format_stamp: "Formatted at 2025-06-26 13:58:47 on dist-test-slave-7p9x"
I20250626 13:58:47.522440 5200 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20250626 13:58:47.528148 5216 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:47.529459 5200 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250626 13:58:47.529853 5200 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "adfc75fd62a24908b775fe41af11b077"
format_stamp: "Formatted at 2025-06-26 13:58:47 on dist-test-slave-7p9x"
I20250626 13:58:47.530231 5200 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:58:47.589587 5200 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:58:47.591044 5200 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:58:47.591471 5200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:58:47.594065 5200 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 13:58:47.598733 5200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 13:58:47.598953 5200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:47.599218 5200 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 13:58:47.599378 5200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:47.737921 5200 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.65:41115
I20250626 13:58:47.738039 5328 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.65:41115 every 8 connection(s)
I20250626 13:58:47.740406 5200 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250626 13:58:47.744822 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 5200
I20250626 13:58:47.745311 2477 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250626 13:58:47.751170 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.107.66:0
--local_ip_for_outbound_sockets=127.2.107.66
--webserver_interface=127.2.107.66
--webserver_port=0
--tserver_master_addrs=127.2.107.126:39915
--builtin_ntp_servers=127.2.107.84:40679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 13:58:47.762249 5329 heartbeater.cc:344] Connected to a master server at 127.2.107.126:39915
I20250626 13:58:47.762701 5329 heartbeater.cc:461] Registering TS with master...
I20250626 13:58:47.763700 5329 heartbeater.cc:507] Master 127.2.107.126:39915 requested a full tablet report, sending...
I20250626 13:58:47.766264 5141 ts_manager.cc:194] Registered new tserver with Master: adfc75fd62a24908b775fe41af11b077 (127.2.107.65:41115)
I20250626 13:58:47.768262 5141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.107.65:49793
W20250626 13:58:48.041199 5333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:58:48.041687 5333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:58:48.042174 5333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:58:48.073526 5333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:58:48.074406 5333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.107.66
I20250626 13:58:48.108652 5333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:40679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.107.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.107.66
--webserver_port=0
--tserver_master_addrs=127.2.107.126:39915
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.107.66
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:58:48.109998 5333 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:58:48.111635 5333 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:58:48.126808 5340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:58:48.771556 5329 heartbeater.cc:499] Master 127.2.107.126:39915 was elected leader, sending a full tablet report...
W20250626 13:58:48.126845 5342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:58:48.126797 5339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:58:48.129628 5333 server_base.cc:1048] running on GCE node
I20250626 13:58:49.203703 5333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:58:49.205907 5333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:58:49.207262 5333 hybrid_clock.cc:648] HybridClock initialized: now 1750946329207216 us; error 57 us; skew 500 ppm
I20250626 13:58:49.208045 5333 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:58:49.214058 5333 webserver.cc:469] Webserver started at http://127.2.107.66:42749/ using document root <none> and password file <none>
I20250626 13:58:49.214943 5333 fs_manager.cc:362] Metadata directory not provided
I20250626 13:58:49.215147 5333 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:58:49.215579 5333 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 13:58:49.221410 5333 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "f7c67fd060a84947a1abd37e5b54d1c9"
format_stamp: "Formatted at 2025-06-26 13:58:49 on dist-test-slave-7p9x"
I20250626 13:58:49.222474 5333 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "f7c67fd060a84947a1abd37e5b54d1c9"
format_stamp: "Formatted at 2025-06-26 13:58:49 on dist-test-slave-7p9x"
I20250626 13:58:49.229322 5333 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20250626 13:58:49.234733 5349 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:49.235750 5333 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250626 13:58:49.236052 5333 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "f7c67fd060a84947a1abd37e5b54d1c9"
format_stamp: "Formatted at 2025-06-26 13:58:49 on dist-test-slave-7p9x"
I20250626 13:58:49.236388 5333 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:58:49.284693 5333 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:58:49.286160 5333 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:58:49.286583 5333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:58:49.288998 5333 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 13:58:49.292822 5333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 13:58:49.293015 5333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:49.293238 5333 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 13:58:49.293382 5333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:49.418061 5333 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.66:40681
I20250626 13:58:49.418188 5461 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.66:40681 every 8 connection(s)
I20250626 13:58:49.420498 5333 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250626 13:58:49.424876 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 5333
I20250626 13:58:49.425367 2477 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250626 13:58:49.432418 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.107.67:0
--local_ip_for_outbound_sockets=127.2.107.67
--webserver_interface=127.2.107.67
--webserver_port=0
--tserver_master_addrs=127.2.107.126:39915
--builtin_ntp_servers=127.2.107.84:40679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 13:58:49.444316 5462 heartbeater.cc:344] Connected to a master server at 127.2.107.126:39915
I20250626 13:58:49.444732 5462 heartbeater.cc:461] Registering TS with master...
I20250626 13:58:49.445858 5462 heartbeater.cc:507] Master 127.2.107.126:39915 requested a full tablet report, sending...
I20250626 13:58:49.448032 5141 ts_manager.cc:194] Registered new tserver with Master: f7c67fd060a84947a1abd37e5b54d1c9 (127.2.107.66:40681)
I20250626 13:58:49.449301 5141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.107.66:35363
W20250626 13:58:49.720793 5466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:58:49.721321 5466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:58:49.721805 5466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:58:49.753245 5466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:58:49.754077 5466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.107.67
I20250626 13:58:49.789748 5466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:40679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.107.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.107.67
--webserver_port=0
--tserver_master_addrs=127.2.107.126:39915
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.107.67
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:58:49.791023 5466 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:58:49.792631 5466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:58:49.807627 5473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:58:50.452332 5462 heartbeater.cc:499] Master 127.2.107.126:39915 was elected leader, sending a full tablet report...
W20250626 13:58:49.808290 5472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:58:49.810107 5466 server_base.cc:1048] running on GCE node
W20250626 13:58:49.809751 5475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:58:50.889822 5466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:58:50.891925 5466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:58:50.893255 5466 hybrid_clock.cc:648] HybridClock initialized: now 1750946330893220 us; error 54 us; skew 500 ppm
I20250626 13:58:50.894023 5466 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:58:50.904940 5466 webserver.cc:469] Webserver started at http://127.2.107.67:34703/ using document root <none> and password file <none>
I20250626 13:58:50.906185 5466 fs_manager.cc:362] Metadata directory not provided
I20250626 13:58:50.906455 5466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:58:50.906996 5466 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 13:58:50.912421 5466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "8e2735d1eb124fceae9b6b3c5d5ab261"
format_stamp: "Formatted at 2025-06-26 13:58:50 on dist-test-slave-7p9x"
I20250626 13:58:50.913558 5466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "8e2735d1eb124fceae9b6b3c5d5ab261"
format_stamp: "Formatted at 2025-06-26 13:58:50 on dist-test-slave-7p9x"
I20250626 13:58:50.920745 5466 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.002s sys 0.004s
I20250626 13:58:50.926195 5482 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:50.927196 5466 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250626 13:58:50.927486 5466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "8e2735d1eb124fceae9b6b3c5d5ab261"
format_stamp: "Formatted at 2025-06-26 13:58:50 on dist-test-slave-7p9x"
I20250626 13:58:50.927800 5466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:58:50.975906 5466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:58:50.977371 5466 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:58:50.977777 5466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:58:50.980170 5466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 13:58:50.984117 5466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 13:58:50.984318 5466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:50.984549 5466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 13:58:50.984709 5466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:51.110715 5466 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.67:32943
I20250626 13:58:51.110831 5594 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.67:32943 every 8 connection(s)
I20250626 13:58:51.113221 5466 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250626 13:58:51.114158 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 5466
I20250626 13:58:51.114665 2477 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250626 13:58:51.133539 5595 heartbeater.cc:344] Connected to a master server at 127.2.107.126:39915
I20250626 13:58:51.133991 5595 heartbeater.cc:461] Registering TS with master...
I20250626 13:58:51.135262 5595 heartbeater.cc:507] Master 127.2.107.126:39915 requested a full tablet report, sending...
I20250626 13:58:51.137442 5140 ts_manager.cc:194] Registered new tserver with Master: 8e2735d1eb124fceae9b6b3c5d5ab261 (127.2.107.67:32943)
I20250626 13:58:51.138725 5140 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.107.67:34055
I20250626 13:58:51.149438 2477 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 13:58:51.179774 2477 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 13:58:51.180099 2477 test_util.cc:276] Using random seed: -166249425
I20250626 13:58:51.219677 5140 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48128:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 13:58:51.260054 5264 tablet_service.cc:1468] Processing CreateTablet for tablet 23a90d8fe96e47eaa5971c96f177b60a (DEFAULT_TABLE table=TestTable [id=a935c167041e4654b52c321592e5e301]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 13:58:51.261709 5264 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23a90d8fe96e47eaa5971c96f177b60a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:58:51.280699 5615 tablet_bootstrap.cc:492] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077: Bootstrap starting.
I20250626 13:58:51.286082 5615 tablet_bootstrap.cc:654] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077: Neither blocks nor log segments found. Creating new log.
I20250626 13:58:51.287752 5615 log.cc:826] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077: Log is configured to *not* fsync() on all Append() calls
I20250626 13:58:51.292376 5615 tablet_bootstrap.cc:492] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077: No bootstrap required, opened a new log
I20250626 13:58:51.292765 5615 ts_tablet_manager.cc:1397] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20250626 13:58:51.309695 5615 raft_consensus.cc:357] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } }
I20250626 13:58:51.310263 5615 raft_consensus.cc:383] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 13:58:51.310467 5615 raft_consensus.cc:738] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adfc75fd62a24908b775fe41af11b077, State: Initialized, Role: FOLLOWER
I20250626 13:58:51.311087 5615 consensus_queue.cc:260] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } }
I20250626 13:58:51.311731 5615 raft_consensus.cc:397] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 13:58:51.311982 5615 raft_consensus.cc:491] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 13:58:51.312238 5615 raft_consensus.cc:3058] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [term 0 FOLLOWER]: Advancing to term 1
I20250626 13:58:51.316823 5615 raft_consensus.cc:513] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } }
I20250626 13:58:51.317580 5615 leader_election.cc:304] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adfc75fd62a24908b775fe41af11b077; no voters:
I20250626 13:58:51.319291 5615 leader_election.cc:290] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 13:58:51.319648 5617 raft_consensus.cc:2802] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 13:58:51.321645 5617 raft_consensus.cc:695] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [term 1 LEADER]: Becoming Leader. State: Replica: adfc75fd62a24908b775fe41af11b077, State: Running, Role: LEADER
I20250626 13:58:51.322580 5617 consensus_queue.cc:237] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } }
I20250626 13:58:51.323432 5615 ts_tablet_manager.cc:1428] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077: Time spent starting tablet: real 0.030s user 0.023s sys 0.008s
I20250626 13:58:51.335475 5141 catalog_manager.cc:5582] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 reported cstate change: term changed from 0 to 1, leader changed from <none> to adfc75fd62a24908b775fe41af11b077 (127.2.107.65). New cstate: current_term: 1 leader_uuid: "adfc75fd62a24908b775fe41af11b077" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } health_report { overall_health: HEALTHY } } }
I20250626 13:58:51.539886 2477 test_util.cc:276] Using random seed: -165889650
I20250626 13:58:51.560700 5140 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48130:
name: "TestTable1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 13:58:51.588245 5397 tablet_service.cc:1468] Processing CreateTablet for tablet 0d6f39a2629e4584a8bbb8a5a40fe5ed (DEFAULT_TABLE table=TestTable1 [id=e9d5148750b14db8bd3db6a6c876a398]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 13:58:51.589746 5397 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d6f39a2629e4584a8bbb8a5a40fe5ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:58:51.609229 5636 tablet_bootstrap.cc:492] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9: Bootstrap starting.
I20250626 13:58:51.614840 5636 tablet_bootstrap.cc:654] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9: Neither blocks nor log segments found. Creating new log.
I20250626 13:58:51.616475 5636 log.cc:826] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9: Log is configured to *not* fsync() on all Append() calls
I20250626 13:58:51.620666 5636 tablet_bootstrap.cc:492] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9: No bootstrap required, opened a new log
I20250626 13:58:51.621169 5636 ts_tablet_manager.cc:1397] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20250626 13:58:51.638598 5636 raft_consensus.cc:357] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } }
I20250626 13:58:51.639194 5636 raft_consensus.cc:383] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 13:58:51.639394 5636 raft_consensus.cc:738] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c67fd060a84947a1abd37e5b54d1c9, State: Initialized, Role: FOLLOWER
I20250626 13:58:51.640024 5636 consensus_queue.cc:260] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } }
I20250626 13:58:51.640494 5636 raft_consensus.cc:397] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 13:58:51.640724 5636 raft_consensus.cc:491] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 13:58:51.640987 5636 raft_consensus.cc:3058] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [term 0 FOLLOWER]: Advancing to term 1
I20250626 13:58:51.644974 5636 raft_consensus.cc:513] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } }
I20250626 13:58:51.645674 5636 leader_election.cc:304] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7c67fd060a84947a1abd37e5b54d1c9; no voters:
I20250626 13:58:51.647564 5636 leader_election.cc:290] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 13:58:51.647883 5638 raft_consensus.cc:2802] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 13:58:51.650247 5638 raft_consensus.cc:695] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [term 1 LEADER]: Becoming Leader. State: Replica: f7c67fd060a84947a1abd37e5b54d1c9, State: Running, Role: LEADER
I20250626 13:58:51.651132 5638 consensus_queue.cc:237] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } }
I20250626 13:58:51.652421 5636 ts_tablet_manager.cc:1428] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9: Time spent starting tablet: real 0.031s user 0.025s sys 0.004s
I20250626 13:58:51.661892 5140 catalog_manager.cc:5582] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7c67fd060a84947a1abd37e5b54d1c9 (127.2.107.66). New cstate: current_term: 1 leader_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } health_report { overall_health: HEALTHY } } }
I20250626 13:58:51.842751 2477 test_util.cc:276] Using random seed: -165586779
I20250626 13:58:51.866320 5140 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48132:
name: "TestTable2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 13:58:51.895473 5530 tablet_service.cc:1468] Processing CreateTablet for tablet 2d6e309ecdf64c5aaffdc5b24aff9086 (DEFAULT_TABLE table=TestTable2 [id=216035caa9b64b13a7bc7dc18a6eed4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 13:58:51.896839 5530 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d6e309ecdf64c5aaffdc5b24aff9086. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:58:51.915642 5657 tablet_bootstrap.cc:492] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261: Bootstrap starting.
I20250626 13:58:51.921295 5657 tablet_bootstrap.cc:654] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261: Neither blocks nor log segments found. Creating new log.
I20250626 13:58:51.922947 5657 log.cc:826] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261: Log is configured to *not* fsync() on all Append() calls
I20250626 13:58:51.927155 5657 tablet_bootstrap.cc:492] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261: No bootstrap required, opened a new log
I20250626 13:58:51.927572 5657 ts_tablet_manager.cc:1397] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20250626 13:58:51.944262 5657 raft_consensus.cc:357] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } }
I20250626 13:58:51.944868 5657 raft_consensus.cc:383] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 13:58:51.945087 5657 raft_consensus.cc:738] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e2735d1eb124fceae9b6b3c5d5ab261, State: Initialized, Role: FOLLOWER
I20250626 13:58:51.945744 5657 consensus_queue.cc:260] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } }
I20250626 13:58:51.946221 5657 raft_consensus.cc:397] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 13:58:51.946444 5657 raft_consensus.cc:491] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 13:58:51.946708 5657 raft_consensus.cc:3058] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 0 FOLLOWER]: Advancing to term 1
I20250626 13:58:51.950672 5657 raft_consensus.cc:513] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } }
I20250626 13:58:51.951305 5657 leader_election.cc:304] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e2735d1eb124fceae9b6b3c5d5ab261; no voters:
I20250626 13:58:51.953079 5657 leader_election.cc:290] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 13:58:51.953627 5659 raft_consensus.cc:2802] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 13:58:51.956022 5595 heartbeater.cc:499] Master 127.2.107.126:39915 was elected leader, sending a full tablet report...
I20250626 13:58:51.956446 5659 raft_consensus.cc:695] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 1 LEADER]: Becoming Leader. State: Replica: 8e2735d1eb124fceae9b6b3c5d5ab261, State: Running, Role: LEADER
I20250626 13:58:51.957094 5657 ts_tablet_manager.cc:1428] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261: Time spent starting tablet: real 0.029s user 0.021s sys 0.009s
I20250626 13:58:51.957242 5659 consensus_queue.cc:237] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } }
I20250626 13:58:51.968391 5140 catalog_manager.cc:5582] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e2735d1eb124fceae9b6b3c5d5ab261 (127.2.107.67). New cstate: current_term: 1 leader_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } health_report { overall_health: HEALTHY } } }
I20250626 13:58:52.179867 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 5108
W20250626 13:58:52.350852 5329 heartbeater.cc:646] Failed to heartbeat to 127.2.107.126:39915 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.107.126:39915: connect: Connection refused (error 111)
W20250626 13:58:52.677071 5462 heartbeater.cc:646] Failed to heartbeat to 127.2.107.126:39915 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.107.126:39915: connect: Connection refused (error 111)
W20250626 13:58:52.985545 5595 heartbeater.cc:646] Failed to heartbeat to 127.2.107.126:39915 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.107.126:39915: connect: Connection refused (error 111)
I20250626 13:58:56.711392 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 5200
I20250626 13:58:56.730859 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 5333
I20250626 13:58:56.753654 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 5466
I20250626 13:58:56.777081 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.107.126:39915
--webserver_interface=127.2.107.126
--webserver_port=45001
--builtin_ntp_servers=127.2.107.84:40679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.107.126:39915 with env {}
W20250626 13:58:57.067762 5736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:58:57.068392 5736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:58:57.068876 5736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:58:57.099589 5736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 13:58:57.099913 5736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:58:57.100157 5736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 13:58:57.100394 5736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 13:58:57.135502 5736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:40679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.107.126:39915
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.107.126:39915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.107.126
--webserver_port=45001
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:58:57.136845 5736 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:58:57.138537 5736 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:58:57.154165 5743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:58:57.154274 5745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:58:57.156345 5736 server_base.cc:1048] running on GCE node
W20250626 13:58:57.154152 5742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:58:58.258656 5736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:58:58.261379 5736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:58:58.262774 5736 hybrid_clock.cc:648] HybridClock initialized: now 1750946338262732 us; error 54 us; skew 500 ppm
I20250626 13:58:58.263669 5736 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:58:58.269675 5736 webserver.cc:469] Webserver started at http://127.2.107.126:45001/ using document root <none> and password file <none>
I20250626 13:58:58.270566 5736 fs_manager.cc:362] Metadata directory not provided
I20250626 13:58:58.270773 5736 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:58:58.278502 5736 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20250626 13:58:58.282930 5752 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:58.283923 5736 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250626 13:58:58.284240 5736 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
uuid: "dff2ce1fa5a3439a92429a6dd10c6097"
format_stamp: "Formatted at 2025-06-26 13:58:45 on dist-test-slave-7p9x"
I20250626 13:58:58.286196 5736 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:58:58.332145 5736 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:58:58.333633 5736 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:58:58.334045 5736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:58:58.400887 5736 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.126:39915
I20250626 13:58:58.400972 5803 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.126:39915 every 8 connection(s)
I20250626 13:58:58.403673 5736 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250626 13:58:58.406994 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 5736
I20250626 13:58:58.408839 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.107.65:41115
--local_ip_for_outbound_sockets=127.2.107.65
--tserver_master_addrs=127.2.107.126:39915
--webserver_port=42549
--webserver_interface=127.2.107.65
--builtin_ntp_servers=127.2.107.84:40679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 13:58:58.416678 5804 sys_catalog.cc:263] Verifying existing consensus state
I20250626 13:58:58.421756 5804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097: Bootstrap starting.
I20250626 13:58:58.431396 5804 log.cc:826] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097: Log is configured to *not* fsync() on all Append() calls
I20250626 13:58:58.478937 5804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=13 ignored=0} mutations{seen=10 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 13:58:58.479720 5804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097: Bootstrap complete.
I20250626 13:58:58.500422 5804 raft_consensus.cc:357] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff2ce1fa5a3439a92429a6dd10c6097" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 39915 } }
I20250626 13:58:58.502532 5804 raft_consensus.cc:738] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff2ce1fa5a3439a92429a6dd10c6097, State: Initialized, Role: FOLLOWER
I20250626 13:58:58.503324 5804 consensus_queue.cc:260] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff2ce1fa5a3439a92429a6dd10c6097" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 39915 } }
I20250626 13:58:58.503832 5804 raft_consensus.cc:397] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 13:58:58.504087 5804 raft_consensus.cc:491] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 13:58:58.504372 5804 raft_consensus.cc:3058] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [term 2 FOLLOWER]: Advancing to term 3
I20250626 13:58:58.510007 5804 raft_consensus.cc:513] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff2ce1fa5a3439a92429a6dd10c6097" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 39915 } }
I20250626 13:58:58.510620 5804 leader_election.cc:304] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dff2ce1fa5a3439a92429a6dd10c6097; no voters:
I20250626 13:58:58.512501 5804 leader_election.cc:290] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250626 13:58:58.512974 5808 raft_consensus.cc:2802] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 13:58:58.515908 5808 raft_consensus.cc:695] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [term 3 LEADER]: Becoming Leader. State: Replica: dff2ce1fa5a3439a92429a6dd10c6097, State: Running, Role: LEADER
I20250626 13:58:58.517081 5804 sys_catalog.cc:564] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 13:58:58.516754 5808 consensus_queue.cc:237] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff2ce1fa5a3439a92429a6dd10c6097" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 39915 } }
I20250626 13:58:58.527370 5810 sys_catalog.cc:455] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dff2ce1fa5a3439a92429a6dd10c6097. Latest consensus state: current_term: 3 leader_uuid: "dff2ce1fa5a3439a92429a6dd10c6097" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff2ce1fa5a3439a92429a6dd10c6097" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 39915 } } }
I20250626 13:58:58.528700 5809 sys_catalog.cc:455] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "dff2ce1fa5a3439a92429a6dd10c6097" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff2ce1fa5a3439a92429a6dd10c6097" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 39915 } } }
I20250626 13:58:58.529412 5809 sys_catalog.cc:458] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [sys.catalog]: This master's current role is: LEADER
I20250626 13:58:58.532133 5810 sys_catalog.cc:458] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [sys.catalog]: This master's current role is: LEADER
I20250626 13:58:58.539515 5815 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 13:58:58.552861 5815 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=216035caa9b64b13a7bc7dc18a6eed4d]
I20250626 13:58:58.554565 5815 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=a310ba57cf9c410394fdd12e608ba840]
I20250626 13:58:58.556325 5815 catalog_manager.cc:671] Loaded metadata for table TestTable [id=d15e5999205b4f7f858e9372c5d96cf3]
I20250626 13:58:58.564567 5815 tablet_loader.cc:96] loaded metadata for tablet 0d6f39a2629e4584a8bbb8a5a40fe5ed (table TestTable1 [id=a310ba57cf9c410394fdd12e608ba840])
I20250626 13:58:58.565944 5815 tablet_loader.cc:96] loaded metadata for tablet 23a90d8fe96e47eaa5971c96f177b60a (table TestTable [id=d15e5999205b4f7f858e9372c5d96cf3])
I20250626 13:58:58.567456 5815 tablet_loader.cc:96] loaded metadata for tablet 2d6e309ecdf64c5aaffdc5b24aff9086 (table TestTable2 [id=216035caa9b64b13a7bc7dc18a6eed4d])
I20250626 13:58:58.568917 5815 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 13:58:58.574496 5815 catalog_manager.cc:1261] Loaded cluster ID: fdd2a3c7727c496ca7c541014bb33fe4
I20250626 13:58:58.574818 5815 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 13:58:58.583323 5815 catalog_manager.cc:1506] Loading token signing keys...
I20250626 13:58:58.588871 5815 catalog_manager.cc:5966] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097: Loaded TSK: 0
I20250626 13:58:58.590543 5815 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 13:58:58.591642 5825 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20250626 13:58:58.739009 5806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:58:58.739502 5806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:58:58.739993 5806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:58:58.771703 5806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:58:58.772576 5806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.107.65
I20250626 13:58:58.808081 5806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:40679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.107.65:41115
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.107.65
--webserver_port=42549
--tserver_master_addrs=127.2.107.126:39915
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.107.65
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:58:58.809422 5806 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:58:58.811064 5806 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:58:58.827514 5832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:58:58.829396 5806 server_base.cc:1048] running on GCE node
W20250626 13:58:58.829510 5834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:58:58.828868 5831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:58:59.929241 5806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:58:59.931787 5806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:58:59.933223 5806 hybrid_clock.cc:648] HybridClock initialized: now 1750946339933164 us; error 73 us; skew 500 ppm
I20250626 13:58:59.934019 5806 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:58:59.941010 5806 webserver.cc:469] Webserver started at http://127.2.107.65:42549/ using document root <none> and password file <none>
I20250626 13:58:59.941937 5806 fs_manager.cc:362] Metadata directory not provided
I20250626 13:58:59.942150 5806 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:58:59.950114 5806 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.001s
I20250626 13:58:59.954718 5841 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:59.955762 5806 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20250626 13:58:59.956049 5806 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "adfc75fd62a24908b775fe41af11b077"
format_stamp: "Formatted at 2025-06-26 13:58:47 on dist-test-slave-7p9x"
I20250626 13:58:59.957954 5806 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:59:00.012166 5806 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:59:00.013651 5806 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:59:00.014063 5806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:59:00.016849 5806 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 13:59:00.023175 5848 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250626 13:59:00.034240 5806 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250626 13:59:00.034492 5806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.002s sys 0.000s
I20250626 13:59:00.034760 5806 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250626 13:59:00.039332 5806 ts_tablet_manager.cc:610] Registered 1 tablets
I20250626 13:59:00.039523 5806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.000s sys 0.003s
I20250626 13:59:00.039952 5848 tablet_bootstrap.cc:492] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077: Bootstrap starting.
I20250626 13:59:00.111176 5848 log.cc:826] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077: Log is configured to *not* fsync() on all Append() calls
I20250626 13:59:00.222295 5806 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.65:41115
I20250626 13:59:00.222456 5955 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.65:41115 every 8 connection(s)
I20250626 13:59:00.224917 5806 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250626 13:59:00.225893 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 5806
I20250626 13:59:00.227758 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.107.66:40681
--local_ip_for_outbound_sockets=127.2.107.66
--tserver_master_addrs=127.2.107.126:39915
--webserver_port=42749
--webserver_interface=127.2.107.66
--builtin_ntp_servers=127.2.107.84:40679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 13:59:00.239960 5848 tablet_bootstrap.cc:492] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 13:59:00.241015 5848 tablet_bootstrap.cc:492] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077: Bootstrap complete.
I20250626 13:59:00.242667 5848 ts_tablet_manager.cc:1397] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077: Time spent bootstrapping tablet: real 0.203s user 0.167s sys 0.033s
I20250626 13:59:00.264163 5848 raft_consensus.cc:357] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } }
I20250626 13:59:00.267064 5848 raft_consensus.cc:738] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: adfc75fd62a24908b775fe41af11b077, State: Initialized, Role: FOLLOWER
I20250626 13:59:00.268153 5848 consensus_queue.cc:260] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } }
I20250626 13:59:00.268849 5848 raft_consensus.cc:397] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 13:59:00.269224 5848 raft_consensus.cc:491] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 13:59:00.269621 5848 raft_consensus.cc:3058] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [term 1 FOLLOWER]: Advancing to term 2
I20250626 13:59:00.277884 5848 raft_consensus.cc:513] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } }
I20250626 13:59:00.278935 5848 leader_election.cc:304] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adfc75fd62a24908b775fe41af11b077; no voters:
I20250626 13:59:00.283552 5956 heartbeater.cc:344] Connected to a master server at 127.2.107.126:39915
I20250626 13:59:00.284013 5956 heartbeater.cc:461] Registering TS with master...
I20250626 13:59:00.285272 5956 heartbeater.cc:507] Master 127.2.107.126:39915 requested a full tablet report, sending...
I20250626 13:59:00.289712 5848 leader_election.cc:290] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250626 13:59:00.289757 5769 ts_manager.cc:194] Registered new tserver with Master: adfc75fd62a24908b775fe41af11b077 (127.2.107.65:41115)
I20250626 13:59:00.290493 5961 raft_consensus.cc:2802] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 13:59:00.297592 5769 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.107.65:57733
I20250626 13:59:00.301985 5961 raft_consensus.cc:695] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [term 2 LEADER]: Becoming Leader. State: Replica: adfc75fd62a24908b775fe41af11b077, State: Running, Role: LEADER
I20250626 13:59:00.302937 5961 consensus_queue.cc:237] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } }
I20250626 13:59:00.304255 5956 heartbeater.cc:499] Master 127.2.107.126:39915 was elected leader, sending a full tablet report...
I20250626 13:59:00.313248 5848 ts_tablet_manager.cc:1428] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077: Time spent starting tablet: real 0.070s user 0.037s sys 0.015s
I20250626 13:59:00.314508 5769 catalog_manager.cc:5582] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 reported cstate change: term changed from 0 to 2, leader changed from <none> to adfc75fd62a24908b775fe41af11b077 (127.2.107.65), VOTER adfc75fd62a24908b775fe41af11b077 (127.2.107.65) added. New cstate: current_term: 2 leader_uuid: "adfc75fd62a24908b775fe41af11b077" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } health_report { overall_health: HEALTHY } } }
W20250626 13:59:00.358767 5769 catalog_manager.cc:5260] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 23a90d8fe96e47eaa5971c96f177b60a with cas_config_opid_index -1: no extra replica candidate found for tablet 23a90d8fe96e47eaa5971c96f177b60a (table TestTable [id=d15e5999205b4f7f858e9372c5d96cf3]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (1) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
W20250626 13:59:00.572010 5957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:59:00.572506 5957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:59:00.572994 5957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:59:00.604104 5957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:59:00.604992 5957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.107.66
I20250626 13:59:00.639074 5957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:40679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.107.66:40681
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.107.66
--webserver_port=42749
--tserver_master_addrs=127.2.107.126:39915
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.107.66
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:59:00.640393 5957 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:59:00.642078 5957 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:59:00.658474 5975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:59:00.779070 5800 debug-util.cc:398] Leaking SignalData structure 0x7b080006f0a0 after lost signal to thread 5737
W20250626 13:59:00.659920 5976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:59:00.658629 5978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:59:01.736116 5957 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250626 13:59:01.736078 5977 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 13:59:01.740103 5957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:59:01.742692 5957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:59:01.744102 5957 hybrid_clock.cc:648] HybridClock initialized: now 1750946341744073 us; error 45 us; skew 500 ppm
I20250626 13:59:01.744880 5957 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:59:01.751135 5957 webserver.cc:469] Webserver started at http://127.2.107.66:42749/ using document root <none> and password file <none>
I20250626 13:59:01.752012 5957 fs_manager.cc:362] Metadata directory not provided
I20250626 13:59:01.752244 5957 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:59:01.760354 5957 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.002s sys 0.004s
I20250626 13:59:01.765077 5985 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:59:01.766160 5957 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250626 13:59:01.766501 5957 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "f7c67fd060a84947a1abd37e5b54d1c9"
format_stamp: "Formatted at 2025-06-26 13:58:49 on dist-test-slave-7p9x"
I20250626 13:59:01.768417 5957 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:59:01.820621 5957 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:59:01.822069 5957 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:59:01.822502 5957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:59:01.825297 5957 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 13:59:01.831159 5992 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250626 13:59:01.838502 5957 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250626 13:59:01.838716 5957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.002s sys 0.000s
I20250626 13:59:01.838984 5957 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250626 13:59:01.843439 5957 ts_tablet_manager.cc:610] Registered 1 tablets
I20250626 13:59:01.843676 5957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.003s sys 0.000s
I20250626 13:59:01.844041 5992 tablet_bootstrap.cc:492] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9: Bootstrap starting.
I20250626 13:59:01.917999 5992 log.cc:826] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9: Log is configured to *not* fsync() on all Append() calls
I20250626 13:59:02.020561 5957 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.66:40681
I20250626 13:59:02.020697 6099 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.66:40681 every 8 connection(s)
I20250626 13:59:02.024075 5957 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250626 13:59:02.027959 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 5957
I20250626 13:59:02.029698 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.107.67:32943
--local_ip_for_outbound_sockets=127.2.107.67
--tserver_master_addrs=127.2.107.126:39915
--webserver_port=34703
--webserver_interface=127.2.107.67
--builtin_ntp_servers=127.2.107.84:40679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 13:59:02.044979 5992 tablet_bootstrap.cc:492] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 13:59:02.046135 5992 tablet_bootstrap.cc:492] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9: Bootstrap complete.
I20250626 13:59:02.047925 5992 ts_tablet_manager.cc:1397] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9: Time spent bootstrapping tablet: real 0.204s user 0.157s sys 0.040s
I20250626 13:59:02.071588 5992 raft_consensus.cc:357] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } }
I20250626 13:59:02.074472 5992 raft_consensus.cc:738] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c67fd060a84947a1abd37e5b54d1c9, State: Initialized, Role: FOLLOWER
I20250626 13:59:02.075454 5992 consensus_queue.cc:260] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } }
I20250626 13:59:02.076113 5992 raft_consensus.cc:397] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 13:59:02.076453 5992 raft_consensus.cc:491] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 13:59:02.077292 5992 raft_consensus.cc:3058] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [term 1 FOLLOWER]: Advancing to term 2
I20250626 13:59:02.084843 6100 heartbeater.cc:344] Connected to a master server at 127.2.107.126:39915
I20250626 13:59:02.085369 6100 heartbeater.cc:461] Registering TS with master...
I20250626 13:59:02.085191 5992 raft_consensus.cc:513] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } }
I20250626 13:59:02.086054 5992 leader_election.cc:304] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7c67fd060a84947a1abd37e5b54d1c9; no voters:
I20250626 13:59:02.086609 6100 heartbeater.cc:507] Master 127.2.107.126:39915 requested a full tablet report, sending...
I20250626 13:59:02.090197 5992 leader_election.cc:290] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250626 13:59:02.090461 5769 ts_manager.cc:194] Registered new tserver with Master: f7c67fd060a84947a1abd37e5b54d1c9 (127.2.107.66:40681)
I20250626 13:59:02.090623 6105 raft_consensus.cc:2802] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 13:59:02.093732 5769 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.107.66:38493
I20250626 13:59:02.098603 6105 raft_consensus.cc:695] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [term 2 LEADER]: Becoming Leader. State: Replica: f7c67fd060a84947a1abd37e5b54d1c9, State: Running, Role: LEADER
I20250626 13:59:02.099509 6105 consensus_queue.cc:237] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } }
I20250626 13:59:02.102500 6100 heartbeater.cc:499] Master 127.2.107.126:39915 was elected leader, sending a full tablet report...
I20250626 13:59:02.111485 5992 ts_tablet_manager.cc:1428] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9: Time spent starting tablet: real 0.063s user 0.037s sys 0.019s
I20250626 13:59:02.116286 5769 catalog_manager.cc:5582] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 reported cstate change: term changed from 0 to 2, leader changed from <none> to f7c67fd060a84947a1abd37e5b54d1c9 (127.2.107.66), VOTER f7c67fd060a84947a1abd37e5b54d1c9 (127.2.107.66) added. New cstate: current_term: 2 leader_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } health_report { overall_health: HEALTHY } } }
I20250626 13:59:02.158345 6055 consensus_queue.cc:237] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } } peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: NON_VOTER last_known_addr { host: "127.2.107.65" port: 41115 } attrs { promote: true } }
I20250626 13:59:02.161758 6106 raft_consensus.cc:2953] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index -1 to 9, NON_VOTER adfc75fd62a24908b775fe41af11b077 (127.2.107.65) added. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } } peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: NON_VOTER last_known_addr { host: "127.2.107.65" port: 41115 } attrs { promote: true } } }
I20250626 13:59:02.174578 5756 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0d6f39a2629e4584a8bbb8a5a40fe5ed with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250626 13:59:02.179663 5769 catalog_manager.cc:5582] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 reported cstate change: config changed from index -1 to 9, NON_VOTER adfc75fd62a24908b775fe41af11b077 (127.2.107.65) added. New cstate: current_term: 2 leader_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: NON_VOTER last_known_addr { host: "127.2.107.65" port: 41115 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250626 13:59:02.194352 5988 consensus_peers.cc:487] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 -> Peer adfc75fd62a24908b775fe41af11b077 (127.2.107.65:41115): Couldn't send request to peer adfc75fd62a24908b775fe41af11b077. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0d6f39a2629e4584a8bbb8a5a40fe5ed. This is attempt 1: this message will repeat every 5th retry.
W20250626 13:59:02.519809 6104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:59:02.520495 6104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:59:02.521317 6104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:59:02.578346 6104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:59:02.579722 6104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.107.67
I20250626 13:59:02.620515 6104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:40679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.107.67:32943
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.107.67
--webserver_port=34703
--tserver_master_addrs=127.2.107.126:39915
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.107.67
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:59:02.621793 6104 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:59:02.623382 6104 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:59:02.641235 6123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:59:02.672343 5911 consensus_queue.cc:237] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } } peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: NON_VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: true } }
I20250626 13:59:02.702358 6130 raft_consensus.cc:2953] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index -1 to 10, NON_VOTER f7c67fd060a84947a1abd37e5b54d1c9 (127.2.107.66) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } } peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: NON_VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: true } } }
I20250626 13:59:02.714687 5755 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 23a90d8fe96e47eaa5971c96f177b60a with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 8)
I20250626 13:59:02.720825 5769 catalog_manager.cc:5582] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 reported cstate change: config changed from index -1 to 10, NON_VOTER f7c67fd060a84947a1abd37e5b54d1c9 (127.2.107.66) added. New cstate: current_term: 2 leader_uuid: "adfc75fd62a24908b775fe41af11b077" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: NON_VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250626 13:59:02.723636 5845 consensus_peers.cc:487] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 -> Peer f7c67fd060a84947a1abd37e5b54d1c9 (127.2.107.66:40681): Couldn't send request to peer f7c67fd060a84947a1abd37e5b54d1c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 23a90d8fe96e47eaa5971c96f177b60a. This is attempt 1: this message will repeat every 5th retry.
I20250626 13:59:02.749265 6138 ts_tablet_manager.cc:927] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P adfc75fd62a24908b775fe41af11b077: Initiating tablet copy from peer f7c67fd060a84947a1abd37e5b54d1c9 (127.2.107.66:40681)
I20250626 13:59:02.752702 6138 tablet_copy_client.cc:323] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P adfc75fd62a24908b775fe41af11b077: tablet copy: Beginning tablet copy session from remote peer at address 127.2.107.66:40681
I20250626 13:59:02.754879 6075 tablet_copy_service.cc:140] P f7c67fd060a84947a1abd37e5b54d1c9: Received BeginTabletCopySession request for tablet 0d6f39a2629e4584a8bbb8a5a40fe5ed from peer adfc75fd62a24908b775fe41af11b077 ({username='slave'} at 127.2.107.65:54095)
I20250626 13:59:02.755460 6075 tablet_copy_service.cc:161] P f7c67fd060a84947a1abd37e5b54d1c9: Beginning new tablet copy session on tablet 0d6f39a2629e4584a8bbb8a5a40fe5ed from peer adfc75fd62a24908b775fe41af11b077 at {username='slave'} at 127.2.107.65:54095: session id = adfc75fd62a24908b775fe41af11b077-0d6f39a2629e4584a8bbb8a5a40fe5ed
I20250626 13:59:02.762903 6075 tablet_copy_source_session.cc:215] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 13:59:02.767663 6138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d6f39a2629e4584a8bbb8a5a40fe5ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:59:02.785737 6138 tablet_copy_client.cc:806] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P adfc75fd62a24908b775fe41af11b077: tablet copy: Starting download of 0 data blocks...
I20250626 13:59:02.786573 6138 tablet_copy_client.cc:670] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P adfc75fd62a24908b775fe41af11b077: tablet copy: Starting download of 1 WAL segments...
I20250626 13:59:02.792146 6138 tablet_copy_client.cc:538] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P adfc75fd62a24908b775fe41af11b077: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 13:59:02.801596 6138 tablet_bootstrap.cc:492] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P adfc75fd62a24908b775fe41af11b077: Bootstrap starting.
I20250626 13:59:02.907419 6138 tablet_bootstrap.cc:492] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P adfc75fd62a24908b775fe41af11b077: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 13:59:02.908769 6138 tablet_bootstrap.cc:492] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P adfc75fd62a24908b775fe41af11b077: Bootstrap complete.
I20250626 13:59:02.909454 6138 ts_tablet_manager.cc:1397] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P adfc75fd62a24908b775fe41af11b077: Time spent bootstrapping tablet: real 0.108s user 0.099s sys 0.008s
I20250626 13:59:02.911788 6138 raft_consensus.cc:357] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P adfc75fd62a24908b775fe41af11b077 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } } peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: NON_VOTER last_known_addr { host: "127.2.107.65" port: 41115 } attrs { promote: true } }
I20250626 13:59:02.912338 6138 raft_consensus.cc:738] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P adfc75fd62a24908b775fe41af11b077 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: adfc75fd62a24908b775fe41af11b077, State: Initialized, Role: LEARNER
I20250626 13:59:02.912864 6138 consensus_queue.cc:260] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P adfc75fd62a24908b775fe41af11b077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } } peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: NON_VOTER last_known_addr { host: "127.2.107.65" port: 41115 } attrs { promote: true } }
I20250626 13:59:02.914698 6138 ts_tablet_manager.cc:1428] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P adfc75fd62a24908b775fe41af11b077: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250626 13:59:02.916690 6075 tablet_copy_service.cc:342] P f7c67fd060a84947a1abd37e5b54d1c9: Request end of tablet copy session adfc75fd62a24908b775fe41af11b077-0d6f39a2629e4584a8bbb8a5a40fe5ed received from {username='slave'} at 127.2.107.65:54095
I20250626 13:59:02.917217 6075 tablet_copy_service.cc:434] P f7c67fd060a84947a1abd37e5b54d1c9: ending tablet copy session adfc75fd62a24908b775fe41af11b077-0d6f39a2629e4584a8bbb8a5a40fe5ed on tablet 0d6f39a2629e4584a8bbb8a5a40fe5ed with peer adfc75fd62a24908b775fe41af11b077
I20250626 13:59:03.119825 6141 ts_tablet_manager.cc:927] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9: Initiating tablet copy from peer adfc75fd62a24908b775fe41af11b077 (127.2.107.65:41115)
I20250626 13:59:03.122380 6141 tablet_copy_client.cc:323] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9: tablet copy: Beginning tablet copy session from remote peer at address 127.2.107.65:41115
I20250626 13:59:03.124338 5931 tablet_copy_service.cc:140] P adfc75fd62a24908b775fe41af11b077: Received BeginTabletCopySession request for tablet 23a90d8fe96e47eaa5971c96f177b60a from peer f7c67fd060a84947a1abd37e5b54d1c9 ({username='slave'} at 127.2.107.66:42675)
I20250626 13:59:03.124792 5931 tablet_copy_service.cc:161] P adfc75fd62a24908b775fe41af11b077: Beginning new tablet copy session on tablet 23a90d8fe96e47eaa5971c96f177b60a from peer f7c67fd060a84947a1abd37e5b54d1c9 at {username='slave'} at 127.2.107.66:42675: session id = f7c67fd060a84947a1abd37e5b54d1c9-23a90d8fe96e47eaa5971c96f177b60a
I20250626 13:59:03.129390 5931 tablet_copy_source_session.cc:215] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 13:59:03.132177 6141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23a90d8fe96e47eaa5971c96f177b60a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:59:03.142638 6141 tablet_copy_client.cc:806] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9: tablet copy: Starting download of 0 data blocks...
I20250626 13:59:03.143146 6141 tablet_copy_client.cc:670] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9: tablet copy: Starting download of 1 WAL segments...
I20250626 13:59:03.147877 6141 tablet_copy_client.cc:538] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 13:59:03.153957 6141 tablet_bootstrap.cc:492] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9: Bootstrap starting.
I20250626 13:59:03.244020 5911 raft_consensus.cc:1215] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P adfc75fd62a24908b775fe41af11b077 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20250626 13:59:03.276983 6141 tablet_bootstrap.cc:492] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 13:59:03.277969 6141 tablet_bootstrap.cc:492] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9: Bootstrap complete.
I20250626 13:59:03.278695 6141 ts_tablet_manager.cc:1397] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9: Time spent bootstrapping tablet: real 0.125s user 0.093s sys 0.020s
I20250626 13:59:03.281600 6141 raft_consensus.cc:357] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } } peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: NON_VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: true } }
I20250626 13:59:03.282305 6141 raft_consensus.cc:738] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: f7c67fd060a84947a1abd37e5b54d1c9, State: Initialized, Role: LEARNER
I20250626 13:59:03.282892 6141 consensus_queue.cc:260] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } } peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: NON_VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: true } }
I20250626 13:59:03.284986 6141 ts_tablet_manager.cc:1428] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9: Time spent starting tablet: real 0.006s user 0.000s sys 0.004s
I20250626 13:59:03.286880 5931 tablet_copy_service.cc:342] P adfc75fd62a24908b775fe41af11b077: Request end of tablet copy session f7c67fd060a84947a1abd37e5b54d1c9-23a90d8fe96e47eaa5971c96f177b60a received from {username='slave'} at 127.2.107.66:42675
I20250626 13:59:03.287416 5931 tablet_copy_service.cc:434] P adfc75fd62a24908b775fe41af11b077: ending tablet copy session f7c67fd060a84947a1abd37e5b54d1c9-23a90d8fe96e47eaa5971c96f177b60a on tablet 23a90d8fe96e47eaa5971c96f177b60a with peer f7c67fd060a84947a1abd37e5b54d1c9
I20250626 13:59:03.499035 6055 raft_consensus.cc:1215] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20250626 13:59:03.652345 6137 raft_consensus.cc:1062] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9: attempting to promote NON_VOTER adfc75fd62a24908b775fe41af11b077 to VOTER
I20250626 13:59:03.654381 6137 consensus_queue.cc:237] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } } peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } attrs { promote: false } }
I20250626 13:59:03.659580 5911 raft_consensus.cc:1273] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P adfc75fd62a24908b775fe41af11b077 [term 2 LEARNER]: Refusing update from remote peer f7c67fd060a84947a1abd37e5b54d1c9: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250626 13:59:03.660826 6137 consensus_queue.cc:1035] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250626 13:59:03.668249 6144 raft_consensus.cc:2953] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, adfc75fd62a24908b775fe41af11b077 (127.2.107.65) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } } peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } attrs { promote: false } } }
I20250626 13:59:03.669706 5911 raft_consensus.cc:2953] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P adfc75fd62a24908b775fe41af11b077 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, adfc75fd62a24908b775fe41af11b077 (127.2.107.65) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } } peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } attrs { promote: false } } }
I20250626 13:59:03.680511 5769 catalog_manager.cc:5582] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 reported cstate change: config changed from index 9 to 10, adfc75fd62a24908b775fe41af11b077 (127.2.107.65) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250626 13:59:02.641567 6125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:59:02.641320 6122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:59:03.781915 6124 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 13:59:03.782120 6104 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 13:59:03.790128 6104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:59:03.792843 6104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:59:03.794332 6104 hybrid_clock.cc:648] HybridClock initialized: now 1750946343794260 us; error 82 us; skew 500 ppm
I20250626 13:59:03.795364 6104 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:59:03.807543 6104 webserver.cc:469] Webserver started at http://127.2.107.67:34703/ using document root <none> and password file <none>
I20250626 13:59:03.808796 6104 fs_manager.cc:362] Metadata directory not provided
I20250626 13:59:03.809120 6104 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:59:03.820153 6104 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.006s sys 0.000s
I20250626 13:59:03.826143 6158 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:59:03.827382 6104 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250626 13:59:03.827762 6104 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "8e2735d1eb124fceae9b6b3c5d5ab261"
format_stamp: "Formatted at 2025-06-26 13:58:50 on dist-test-slave-7p9x"
I20250626 13:59:03.830497 6104 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:59:03.876047 6146 raft_consensus.cc:1062] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077: attempting to promote NON_VOTER f7c67fd060a84947a1abd37e5b54d1c9 to VOTER
I20250626 13:59:03.877455 6146 consensus_queue.cc:237] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } } peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: false } }
I20250626 13:59:03.881812 6055 raft_consensus.cc:1273] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9 [term 2 LEARNER]: Refusing update from remote peer adfc75fd62a24908b775fe41af11b077: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250626 13:59:03.882990 6145 consensus_queue.cc:1035] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20250626 13:59:03.890537 6146 raft_consensus.cc:2953] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, f7c67fd060a84947a1abd37e5b54d1c9 (127.2.107.66) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } } peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: false } } }
I20250626 13:59:03.891904 6055 raft_consensus.cc:2953] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, f7c67fd060a84947a1abd37e5b54d1c9 (127.2.107.66) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } } peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: false } } }
I20250626 13:59:03.900559 5769 catalog_manager.cc:5582] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 reported cstate change: config changed from index 10 to 11, f7c67fd060a84947a1abd37e5b54d1c9 (127.2.107.66) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "adfc75fd62a24908b775fe41af11b077" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 13:59:03.902204 6104 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:59:03.903923 6104 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:59:03.904410 6104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:59:03.908205 6104 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 13:59:03.916762 6167 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250626 13:59:03.926716 6104 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250626 13:59:03.927006 6104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.012s user 0.002s sys 0.000s
I20250626 13:59:03.927305 6104 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250626 13:59:03.935190 6104 ts_tablet_manager.cc:610] Registered 1 tablets
I20250626 13:59:03.935477 6104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.008s user 0.007s sys 0.000s
I20250626 13:59:03.935930 6167 tablet_bootstrap.cc:492] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261: Bootstrap starting.
I20250626 13:59:04.003885 6167 log.cc:826] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261: Log is configured to *not* fsync() on all Append() calls
I20250626 13:59:04.120746 6104 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.67:32943
I20250626 13:59:04.120911 6274 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.67:32943 every 8 connection(s)
I20250626 13:59:04.122834 6167 tablet_bootstrap.cc:492] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 13:59:04.123412 6104 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250626 13:59:04.123581 6167 tablet_bootstrap.cc:492] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261: Bootstrap complete.
I20250626 13:59:04.124722 6167 ts_tablet_manager.cc:1397] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261: Time spent bootstrapping tablet: real 0.189s user 0.136s sys 0.046s
I20250626 13:59:04.128268 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 6104
I20250626 13:59:04.148562 6167 raft_consensus.cc:357] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } }
I20250626 13:59:04.151881 6167 raft_consensus.cc:738] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e2735d1eb124fceae9b6b3c5d5ab261, State: Initialized, Role: FOLLOWER
I20250626 13:59:04.152678 6167 consensus_queue.cc:260] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } }
I20250626 13:59:04.153316 6167 raft_consensus.cc:397] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 13:59:04.153652 6167 raft_consensus.cc:491] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 13:59:04.153991 6167 raft_consensus.cc:3058] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 1 FOLLOWER]: Advancing to term 2
I20250626 13:59:04.162274 6167 raft_consensus.cc:513] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } }
I20250626 13:59:04.163625 6275 heartbeater.cc:344] Connected to a master server at 127.2.107.126:39915
I20250626 13:59:04.163561 6167 leader_election.cc:304] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e2735d1eb124fceae9b6b3c5d5ab261; no voters:
I20250626 13:59:04.164170 6275 heartbeater.cc:461] Registering TS with master...
I20250626 13:59:04.165529 6275 heartbeater.cc:507] Master 127.2.107.126:39915 requested a full tablet report, sending...
I20250626 13:59:04.165776 6167 leader_election.cc:290] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250626 13:59:04.166188 6281 raft_consensus.cc:2802] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 13:59:04.168288 6281 raft_consensus.cc:695] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 2 LEADER]: Becoming Leader. State: Replica: 8e2735d1eb124fceae9b6b3c5d5ab261, State: Running, Role: LEADER
I20250626 13:59:04.169157 6281 consensus_queue.cc:237] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } }
I20250626 13:59:04.169593 6167 ts_tablet_manager.cc:1428] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261: Time spent starting tablet: real 0.045s user 0.047s sys 0.000s
I20250626 13:59:04.180790 5768 ts_manager.cc:194] Registered new tserver with Master: 8e2735d1eb124fceae9b6b3c5d5ab261 (127.2.107.67:32943)
I20250626 13:59:04.182000 5768 catalog_manager.cc:5582] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } health_report { overall_health: HEALTHY } } }
I20250626 13:59:04.188884 5768 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.107.67:57447
I20250626 13:59:04.192368 6275 heartbeater.cc:499] Master 127.2.107.126:39915 was elected leader, sending a full tablet report...
I20250626 13:59:04.196237 2477 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 13:59:04.200726 2477 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
W20250626 13:59:04.204015 2477 ts_itest-base.cc:209] found only 2 out of 3 replicas of tablet 23a90d8fe96e47eaa5971c96f177b60a: tablet_id: "23a90d8fe96e47eaa5971c96f177b60a" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER } interned_replicas { ts_info_idx: 1 role: FOLLOWER }
I20250626 13:59:04.241863 5911 consensus_queue.cc:237] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } } peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: NON_VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: true } }
I20250626 13:59:04.247305 6055 raft_consensus.cc:1273] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9 [term 2 FOLLOWER]: Refusing update from remote peer adfc75fd62a24908b775fe41af11b077: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250626 13:59:04.248839 6145 consensus_queue.cc:1035] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250626 13:59:04.255352 6146 raft_consensus.cc:2953] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 8e2735d1eb124fceae9b6b3c5d5ab261 (127.2.107.67) added. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } } peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: NON_VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: true } } }
I20250626 13:59:04.256430 6055 raft_consensus.cc:2953] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 8e2735d1eb124fceae9b6b3c5d5ab261 (127.2.107.67) added. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } } peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: NON_VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: true } } }
W20250626 13:59:04.262297 5843 consensus_peers.cc:487] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 -> Peer 8e2735d1eb124fceae9b6b3c5d5ab261 (127.2.107.67:32943): Couldn't send request to peer 8e2735d1eb124fceae9b6b3c5d5ab261. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 23a90d8fe96e47eaa5971c96f177b60a. This is attempt 1: this message will repeat every 5th retry.
I20250626 13:59:04.263904 5755 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 23a90d8fe96e47eaa5971c96f177b60a with cas_config_opid_index 11: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 5)
I20250626 13:59:04.268487 5768 catalog_manager.cc:5582] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 reported cstate change: config changed from index 11 to 12, NON_VOTER 8e2735d1eb124fceae9b6b3c5d5ab261 (127.2.107.67) added. New cstate: current_term: 2 leader_uuid: "adfc75fd62a24908b775fe41af11b077" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: NON_VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 13:59:04.301306 6055 consensus_queue.cc:237] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } } peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: NON_VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: true } }
I20250626 13:59:04.306352 5911 raft_consensus.cc:1273] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P adfc75fd62a24908b775fe41af11b077 [term 2 FOLLOWER]: Refusing update from remote peer f7c67fd060a84947a1abd37e5b54d1c9: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250626 13:59:04.307628 6144 consensus_queue.cc:1035] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250626 13:59:04.313333 6288 raft_consensus.cc:2953] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 8e2735d1eb124fceae9b6b3c5d5ab261 (127.2.107.67) added. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } } peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: NON_VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: true } } }
W20250626 13:59:04.314616 5987 consensus_peers.cc:487] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 -> Peer 8e2735d1eb124fceae9b6b3c5d5ab261 (127.2.107.67:32943): Couldn't send request to peer 8e2735d1eb124fceae9b6b3c5d5ab261. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0d6f39a2629e4584a8bbb8a5a40fe5ed. This is attempt 1: this message will repeat every 5th retry.
I20250626 13:59:04.314836 5911 raft_consensus.cc:2953] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P adfc75fd62a24908b775fe41af11b077 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 8e2735d1eb124fceae9b6b3c5d5ab261 (127.2.107.67) added. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } } peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: NON_VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: true } } }
I20250626 13:59:04.320909 5756 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0d6f39a2629e4584a8bbb8a5a40fe5ed with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 6)
I20250626 13:59:04.324666 5769 catalog_manager.cc:5582] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P adfc75fd62a24908b775fe41af11b077 reported cstate change: config changed from index 10 to 11, NON_VOTER 8e2735d1eb124fceae9b6b3c5d5ab261 (127.2.107.67) added. New cstate: current_term: 2 leader_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } } peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: NON_VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: true } } }
I20250626 13:59:04.380688 5755 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0d6f39a2629e4584a8bbb8a5a40fe5ed with cas_config_opid_index 9: aborting the task: latest config opid_index 11; task opid_index 9
I20250626 13:59:04.735311 6291 ts_tablet_manager.cc:927] T 23a90d8fe96e47eaa5971c96f177b60a P 8e2735d1eb124fceae9b6b3c5d5ab261: Initiating tablet copy from peer adfc75fd62a24908b775fe41af11b077 (127.2.107.65:41115)
I20250626 13:59:04.737061 6291 tablet_copy_client.cc:323] T 23a90d8fe96e47eaa5971c96f177b60a P 8e2735d1eb124fceae9b6b3c5d5ab261: tablet copy: Beginning tablet copy session from remote peer at address 127.2.107.65:41115
I20250626 13:59:04.748108 5931 tablet_copy_service.cc:140] P adfc75fd62a24908b775fe41af11b077: Received BeginTabletCopySession request for tablet 23a90d8fe96e47eaa5971c96f177b60a from peer 8e2735d1eb124fceae9b6b3c5d5ab261 ({username='slave'} at 127.2.107.67:43343)
I20250626 13:59:04.748533 5931 tablet_copy_service.cc:161] P adfc75fd62a24908b775fe41af11b077: Beginning new tablet copy session on tablet 23a90d8fe96e47eaa5971c96f177b60a from peer 8e2735d1eb124fceae9b6b3c5d5ab261 at {username='slave'} at 127.2.107.67:43343: session id = 8e2735d1eb124fceae9b6b3c5d5ab261-23a90d8fe96e47eaa5971c96f177b60a
I20250626 13:59:04.753295 5931 tablet_copy_source_session.cc:215] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 13:59:04.756465 6291 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23a90d8fe96e47eaa5971c96f177b60a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:59:04.771088 6291 tablet_copy_client.cc:806] T 23a90d8fe96e47eaa5971c96f177b60a P 8e2735d1eb124fceae9b6b3c5d5ab261: tablet copy: Starting download of 0 data blocks...
I20250626 13:59:04.771591 6291 tablet_copy_client.cc:670] T 23a90d8fe96e47eaa5971c96f177b60a P 8e2735d1eb124fceae9b6b3c5d5ab261: tablet copy: Starting download of 1 WAL segments...
I20250626 13:59:04.776453 6291 tablet_copy_client.cc:538] T 23a90d8fe96e47eaa5971c96f177b60a P 8e2735d1eb124fceae9b6b3c5d5ab261: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 13:59:04.784822 6295 ts_tablet_manager.cc:927] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P 8e2735d1eb124fceae9b6b3c5d5ab261: Initiating tablet copy from peer f7c67fd060a84947a1abd37e5b54d1c9 (127.2.107.66:40681)
I20250626 13:59:04.785007 6291 tablet_bootstrap.cc:492] T 23a90d8fe96e47eaa5971c96f177b60a P 8e2735d1eb124fceae9b6b3c5d5ab261: Bootstrap starting.
I20250626 13:59:04.787163 6295 tablet_copy_client.cc:323] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P 8e2735d1eb124fceae9b6b3c5d5ab261: tablet copy: Beginning tablet copy session from remote peer at address 127.2.107.66:40681
I20250626 13:59:04.797114 6075 tablet_copy_service.cc:140] P f7c67fd060a84947a1abd37e5b54d1c9: Received BeginTabletCopySession request for tablet 0d6f39a2629e4584a8bbb8a5a40fe5ed from peer 8e2735d1eb124fceae9b6b3c5d5ab261 ({username='slave'} at 127.2.107.67:38507)
I20250626 13:59:04.797564 6075 tablet_copy_service.cc:161] P f7c67fd060a84947a1abd37e5b54d1c9: Beginning new tablet copy session on tablet 0d6f39a2629e4584a8bbb8a5a40fe5ed from peer 8e2735d1eb124fceae9b6b3c5d5ab261 at {username='slave'} at 127.2.107.67:38507: session id = 8e2735d1eb124fceae9b6b3c5d5ab261-0d6f39a2629e4584a8bbb8a5a40fe5ed
I20250626 13:59:04.802428 6075 tablet_copy_source_session.cc:215] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 13:59:04.804798 6295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d6f39a2629e4584a8bbb8a5a40fe5ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:59:04.816272 6295 tablet_copy_client.cc:806] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P 8e2735d1eb124fceae9b6b3c5d5ab261: tablet copy: Starting download of 0 data blocks...
I20250626 13:59:04.816843 6295 tablet_copy_client.cc:670] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P 8e2735d1eb124fceae9b6b3c5d5ab261: tablet copy: Starting download of 1 WAL segments...
I20250626 13:59:04.820206 6295 tablet_copy_client.cc:538] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P 8e2735d1eb124fceae9b6b3c5d5ab261: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 13:59:04.825927 6295 tablet_bootstrap.cc:492] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P 8e2735d1eb124fceae9b6b3c5d5ab261: Bootstrap starting.
I20250626 13:59:04.897971 6291 tablet_bootstrap.cc:492] T 23a90d8fe96e47eaa5971c96f177b60a P 8e2735d1eb124fceae9b6b3c5d5ab261: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 13:59:04.898716 6291 tablet_bootstrap.cc:492] T 23a90d8fe96e47eaa5971c96f177b60a P 8e2735d1eb124fceae9b6b3c5d5ab261: Bootstrap complete.
I20250626 13:59:04.899159 6291 ts_tablet_manager.cc:1397] T 23a90d8fe96e47eaa5971c96f177b60a P 8e2735d1eb124fceae9b6b3c5d5ab261: Time spent bootstrapping tablet: real 0.114s user 0.092s sys 0.021s
I20250626 13:59:04.900969 6291 raft_consensus.cc:357] T 23a90d8fe96e47eaa5971c96f177b60a P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } } peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: NON_VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: true } }
I20250626 13:59:04.901671 6291 raft_consensus.cc:738] T 23a90d8fe96e47eaa5971c96f177b60a P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 8e2735d1eb124fceae9b6b3c5d5ab261, State: Initialized, Role: LEARNER
I20250626 13:59:04.902369 6291 consensus_queue.cc:260] T 23a90d8fe96e47eaa5971c96f177b60a P 8e2735d1eb124fceae9b6b3c5d5ab261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } } peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: NON_VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: true } }
I20250626 13:59:04.905380 6291 ts_tablet_manager.cc:1428] T 23a90d8fe96e47eaa5971c96f177b60a P 8e2735d1eb124fceae9b6b3c5d5ab261: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20250626 13:59:04.907544 5931 tablet_copy_service.cc:342] P adfc75fd62a24908b775fe41af11b077: Request end of tablet copy session 8e2735d1eb124fceae9b6b3c5d5ab261-23a90d8fe96e47eaa5971c96f177b60a received from {username='slave'} at 127.2.107.67:43343
I20250626 13:59:04.908084 5931 tablet_copy_service.cc:434] P adfc75fd62a24908b775fe41af11b077: ending tablet copy session 8e2735d1eb124fceae9b6b3c5d5ab261-23a90d8fe96e47eaa5971c96f177b60a on tablet 23a90d8fe96e47eaa5971c96f177b60a with peer 8e2735d1eb124fceae9b6b3c5d5ab261
I20250626 13:59:04.914124 6295 tablet_bootstrap.cc:492] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P 8e2735d1eb124fceae9b6b3c5d5ab261: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 13:59:04.914888 6295 tablet_bootstrap.cc:492] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P 8e2735d1eb124fceae9b6b3c5d5ab261: Bootstrap complete.
I20250626 13:59:04.915494 6295 ts_tablet_manager.cc:1397] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P 8e2735d1eb124fceae9b6b3c5d5ab261: Time spent bootstrapping tablet: real 0.090s user 0.072s sys 0.015s
I20250626 13:59:04.917688 6295 raft_consensus.cc:357] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } } peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: NON_VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: true } }
I20250626 13:59:04.918215 6295 raft_consensus.cc:738] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 8e2735d1eb124fceae9b6b3c5d5ab261, State: Initialized, Role: LEARNER
I20250626 13:59:04.918561 6295 consensus_queue.cc:260] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P 8e2735d1eb124fceae9b6b3c5d5ab261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } } peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: NON_VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: true } }
I20250626 13:59:04.920188 6295 ts_tablet_manager.cc:1428] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P 8e2735d1eb124fceae9b6b3c5d5ab261: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250626 13:59:04.921895 6075 tablet_copy_service.cc:342] P f7c67fd060a84947a1abd37e5b54d1c9: Request end of tablet copy session 8e2735d1eb124fceae9b6b3c5d5ab261-0d6f39a2629e4584a8bbb8a5a40fe5ed received from {username='slave'} at 127.2.107.67:38507
I20250626 13:59:04.922292 6075 tablet_copy_service.cc:434] P f7c67fd060a84947a1abd37e5b54d1c9: ending tablet copy session 8e2735d1eb124fceae9b6b3c5d5ab261-0d6f39a2629e4584a8bbb8a5a40fe5ed on tablet 0d6f39a2629e4584a8bbb8a5a40fe5ed with peer 8e2735d1eb124fceae9b6b3c5d5ab261
I20250626 13:59:04.957916 5753 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 23a90d8fe96e47eaa5971c96f177b60a with cas_config_opid_index 10: aborting the task: latest config opid_index 12; task opid_index 10
I20250626 13:59:05.208449 2477 ts_itest-base.cc:246] Waiting for 1 tablets on tserver adfc75fd62a24908b775fe41af11b077 to finish bootstrapping
I20250626 13:59:05.221419 2477 ts_itest-base.cc:246] Waiting for 1 tablets on tserver f7c67fd060a84947a1abd37e5b54d1c9 to finish bootstrapping
I20250626 13:59:05.234686 2477 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 8e2735d1eb124fceae9b6b3c5d5ab261 to finish bootstrapping
I20250626 13:59:05.309796 6230 raft_consensus.cc:1215] T 23a90d8fe96e47eaa5971c96f177b60a P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
I20250626 13:59:05.338985 6230 raft_consensus.cc:1215] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
I20250626 13:59:05.514796 6209 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250626 13:59:05.517232 5891 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250626 13:59:05.528957 6034 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250626 13:59:05.706878 6310 raft_consensus.cc:1062] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077: attempting to promote NON_VOTER 8e2735d1eb124fceae9b6b3c5d5ab261 to VOTER
I20250626 13:59:05.708806 6310 consensus_queue.cc:237] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } } peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: false } }
I20250626 13:59:05.714941 6230 raft_consensus.cc:1273] T 23a90d8fe96e47eaa5971c96f177b60a P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 2 LEARNER]: Refusing update from remote peer adfc75fd62a24908b775fe41af11b077: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250626 13:59:05.716048 6055 raft_consensus.cc:1273] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9 [term 2 FOLLOWER]: Refusing update from remote peer adfc75fd62a24908b775fe41af11b077: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250626 13:59:05.716472 6311 consensus_queue.cc:1035] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20250626 13:59:05.717559 6310 consensus_queue.cc:1035] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250626 13:59:05.724548 6311 raft_consensus.cc:2953] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 8e2735d1eb124fceae9b6b3c5d5ab261 (127.2.107.67) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } } peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: false } } }
I20250626 13:59:05.726603 6055 raft_consensus.cc:2953] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 8e2735d1eb124fceae9b6b3c5d5ab261 (127.2.107.67) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } } peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: false } } }
I20250626 13:59:05.726909 6230 raft_consensus.cc:2953] T 23a90d8fe96e47eaa5971c96f177b60a P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 8e2735d1eb124fceae9b6b3c5d5ab261 (127.2.107.67) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } } peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: false } } }
I20250626 13:59:05.739799 5769 catalog_manager.cc:5582] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9 reported cstate change: config changed from index 12 to 13, 8e2735d1eb124fceae9b6b3c5d5ab261 (127.2.107.67) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "adfc75fd62a24908b775fe41af11b077" committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } } peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: false } } }
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
dff2ce1fa5a3439a92429a6dd10c6097 | 127.2.107.126:39915 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.2.107.84:40679 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
8e2735d1eb124fceae9b6b3c5d5ab261 | 127.2.107.67:32943 | HEALTHY | <none> | 1 | 0
adfc75fd62a24908b775fe41af11b077 | 127.2.107.65:41115 | HEALTHY | <none> | 1 | 0
f7c67fd060a84947a1abd37e5b54d1c9 | 127.2.107.66:40681 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.2.107.65 | experimental | 127.2.107.65:41115
local_ip_for_outbound_sockets | 127.2.107.66 | experimental | 127.2.107.66:40681
local_ip_for_outbound_sockets | 127.2.107.67 | experimental | 127.2.107.67:32943
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.107.65:41115
server_dump_info_path | /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.107.66:40681
server_dump_info_path | /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.107.67:32943
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.2.107.84:40679 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable1 | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable2 | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 3
Tablets | 3
Replicas | 7
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250626 13:59:05.756390 6313 raft_consensus.cc:1062] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9: attempting to promote NON_VOTER 8e2735d1eb124fceae9b6b3c5d5ab261 to VOTER
I20250626 13:59:05.757807 6313 consensus_queue.cc:237] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } } peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: false } }
I20250626 13:59:05.762913 6230 raft_consensus.cc:1273] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 2 LEARNER]: Refusing update from remote peer f7c67fd060a84947a1abd37e5b54d1c9: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250626 13:59:05.763556 2477 log_verifier.cc:126] Checking tablet 0d6f39a2629e4584a8bbb8a5a40fe5ed
I20250626 13:59:05.764025 5911 raft_consensus.cc:1273] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P adfc75fd62a24908b775fe41af11b077 [term 2 FOLLOWER]: Refusing update from remote peer f7c67fd060a84947a1abd37e5b54d1c9: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250626 13:59:05.764839 6313 consensus_queue.cc:1035] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250626 13:59:05.765645 6312 consensus_queue.cc:1035] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20250626 13:59:05.772842 6312 raft_consensus.cc:2953] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 8e2735d1eb124fceae9b6b3c5d5ab261 (127.2.107.67) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } } peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: false } } }
I20250626 13:59:05.775688 6229 raft_consensus.cc:2953] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 8e2735d1eb124fceae9b6b3c5d5ab261 (127.2.107.67) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } } peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: false } } }
I20250626 13:59:05.774748 5911 raft_consensus.cc:2953] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P adfc75fd62a24908b775fe41af11b077 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 8e2735d1eb124fceae9b6b3c5d5ab261 (127.2.107.67) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } } peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: false } } }
I20250626 13:59:05.787249 5768 catalog_manager.cc:5582] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 reported cstate change: config changed from index 11 to 12, 8e2735d1eb124fceae9b6b3c5d5ab261 (127.2.107.67) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 13:59:05.879021 2477 log_verifier.cc:177] Verified matching terms for 12 ops in tablet 0d6f39a2629e4584a8bbb8a5a40fe5ed
I20250626 13:59:05.879323 2477 log_verifier.cc:126] Checking tablet 23a90d8fe96e47eaa5971c96f177b60a
I20250626 13:59:05.977039 2477 log_verifier.cc:177] Verified matching terms for 13 ops in tablet 23a90d8fe96e47eaa5971c96f177b60a
I20250626 13:59:05.977345 2477 log_verifier.cc:126] Checking tablet 2d6e309ecdf64c5aaffdc5b24aff9086
I20250626 13:59:06.005625 2477 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 2d6e309ecdf64c5aaffdc5b24aff9086
I20250626 13:59:06.006067 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 5736
I20250626 13:59:06.033601 2477 minidump.cc:252] Setting minidump size limit to 20M
I20250626 13:59:06.041595 2477 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:59:06.042637 2477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:59:06.053445 6344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:59:06.053438 6343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:59:06.188814 6346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:59:06.188822 2477 server_base.cc:1048] running on GCE node
I20250626 13:59:06.190160 2477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 13:59:06.190394 2477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 13:59:06.190552 2477 hybrid_clock.cc:648] HybridClock initialized: now 1750946346190538 us; error 0 us; skew 500 ppm
I20250626 13:59:06.191203 2477 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:59:06.194628 2477 webserver.cc:469] Webserver started at http://0.0.0.0:43959/ using document root <none> and password file <none>
I20250626 13:59:06.195431 2477 fs_manager.cc:362] Metadata directory not provided
I20250626 13:59:06.195626 2477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:59:06.200908 2477 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20250626 13:59:06.204406 6351 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:59:06.205369 2477 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250626 13:59:06.205678 2477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
uuid: "dff2ce1fa5a3439a92429a6dd10c6097"
format_stamp: "Formatted at 2025-06-26 13:58:45 on dist-test-slave-7p9x"
I20250626 13:59:06.207553 2477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:59:06.232546 2477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:59:06.234073 2477 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:59:06.234516 2477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:59:06.243489 2477 sys_catalog.cc:263] Verifying existing consensus state
W20250626 13:59:06.247756 2477 sys_catalog.cc:243] For a single master config, on-disk Raft master: 127.2.107.126:39915 exists but no master address supplied!
I20250626 13:59:06.249920 2477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097: Bootstrap starting.
I20250626 13:59:06.295800 2477 log.cc:826] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097: Log is configured to *not* fsync() on all Append() calls
I20250626 13:59:06.362967 2477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=13 ignored=0} mutations{seen=21 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 13:59:06.363740 2477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097: Bootstrap complete.
I20250626 13:59:06.377748 2477 raft_consensus.cc:357] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff2ce1fa5a3439a92429a6dd10c6097" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 39915 } }
I20250626 13:59:06.378418 2477 raft_consensus.cc:738] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff2ce1fa5a3439a92429a6dd10c6097, State: Initialized, Role: FOLLOWER
I20250626 13:59:06.379331 2477 consensus_queue.cc:260] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff2ce1fa5a3439a92429a6dd10c6097" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 39915 } }
I20250626 13:59:06.379841 2477 raft_consensus.cc:397] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [term 3 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 13:59:06.380105 2477 raft_consensus.cc:491] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [term 3 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 13:59:06.380415 2477 raft_consensus.cc:3058] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [term 3 FOLLOWER]: Advancing to term 4
I20250626 13:59:06.385862 2477 raft_consensus.cc:513] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff2ce1fa5a3439a92429a6dd10c6097" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 39915 } }
I20250626 13:59:06.386790 2477 leader_election.cc:304] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dff2ce1fa5a3439a92429a6dd10c6097; no voters:
I20250626 13:59:06.388373 2477 leader_election.cc:290] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [CANDIDATE]: Term 4 election: Requested vote from peers
I20250626 13:59:06.388685 6360 raft_consensus.cc:2802] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 13:59:06.390184 6360 raft_consensus.cc:695] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [term 4 LEADER]: Becoming Leader. State: Replica: dff2ce1fa5a3439a92429a6dd10c6097, State: Running, Role: LEADER
I20250626 13:59:06.391462 6360 consensus_queue.cc:237] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff2ce1fa5a3439a92429a6dd10c6097" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 39915 } }
I20250626 13:59:06.400632 6362 sys_catalog.cc:455] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dff2ce1fa5a3439a92429a6dd10c6097. Latest consensus state: current_term: 4 leader_uuid: "dff2ce1fa5a3439a92429a6dd10c6097" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff2ce1fa5a3439a92429a6dd10c6097" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 39915 } } }
I20250626 13:59:06.400471 6361 sys_catalog.cc:455] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 leader_uuid: "dff2ce1fa5a3439a92429a6dd10c6097" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff2ce1fa5a3439a92429a6dd10c6097" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 39915 } } }
I20250626 13:59:06.401158 6362 sys_catalog.cc:458] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [sys.catalog]: This master's current role is: LEADER
I20250626 13:59:06.401173 6361 sys_catalog.cc:458] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [sys.catalog]: This master's current role is: LEADER
I20250626 13:59:06.427891 2477 tablet_replica.cc:331] stopping tablet replica
I20250626 13:59:06.428627 2477 raft_consensus.cc:2241] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [term 4 LEADER]: Raft consensus shutting down.
I20250626 13:59:06.429477 2477 raft_consensus.cc:2270] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250626 13:59:06.432871 2477 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250626 13:59:06.433596 2477 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250626 13:59:06.478307 2477 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
W20250626 13:59:06.818375 6275 heartbeater.cc:646] Failed to heartbeat to 127.2.107.126:39915 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.107.126:39915: connect: Connection refused (error 111)
W20250626 13:59:06.820983 6100 heartbeater.cc:646] Failed to heartbeat to 127.2.107.126:39915 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.107.126:39915: connect: Connection refused (error 111)
W20250626 13:59:06.822528 5956 heartbeater.cc:646] Failed to heartbeat to 127.2.107.126:39915 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.107.126:39915: connect: Connection refused (error 111)
I20250626 13:59:11.609408 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 5806
I20250626 13:59:11.631167 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 5957
I20250626 13:59:11.655195 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 6104
I20250626 13:59:11.680430 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.107.126:39915
--webserver_interface=127.2.107.126
--webserver_port=45001
--builtin_ntp_servers=127.2.107.84:40679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.107.126:39915 with env {}
W20250626 13:59:11.989266 6436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:59:11.989866 6436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:59:11.990351 6436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:59:12.022940 6436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 13:59:12.023269 6436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:59:12.023772 6436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 13:59:12.024194 6436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 13:59:12.060741 6436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:40679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.107.126:39915
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.107.126:39915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.107.126
--webserver_port=45001
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:59:12.062111 6436 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:59:12.063803 6436 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:59:12.078673 6442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:59:12.078682 6445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:59:12.078675 6443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:59:12.080600 6436 server_base.cc:1048] running on GCE node
I20250626 13:59:13.208582 6436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:59:13.211189 6436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:59:13.212591 6436 hybrid_clock.cc:648] HybridClock initialized: now 1750946353212531 us; error 73 us; skew 500 ppm
I20250626 13:59:13.213826 6436 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:59:13.235455 6436 webserver.cc:469] Webserver started at http://127.2.107.126:45001/ using document root <none> and password file <none>
I20250626 13:59:13.236430 6436 fs_manager.cc:362] Metadata directory not provided
I20250626 13:59:13.236639 6436 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:59:13.244925 6436 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20250626 13:59:13.249570 6452 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:59:13.250849 6436 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250626 13:59:13.251170 6436 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
uuid: "dff2ce1fa5a3439a92429a6dd10c6097"
format_stamp: "Formatted at 2025-06-26 13:58:45 on dist-test-slave-7p9x"
I20250626 13:59:13.253176 6436 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:59:13.310537 6436 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:59:13.312188 6436 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:59:13.312695 6436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:59:13.387389 6436 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.126:39915
I20250626 13:59:13.387493 6503 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.126:39915 every 8 connection(s)
I20250626 13:59:13.390265 6436 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250626 13:59:13.395702 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 6436
I20250626 13:59:13.397080 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.107.65:41115
--local_ip_for_outbound_sockets=127.2.107.65
--tserver_master_addrs=127.2.107.126:39915
--webserver_port=42549
--webserver_interface=127.2.107.65
--builtin_ntp_servers=127.2.107.84:40679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 13:59:13.400657 6504 sys_catalog.cc:263] Verifying existing consensus state
I20250626 13:59:13.407011 6504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097: Bootstrap starting.
I20250626 13:59:13.417580 6504 log.cc:826] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097: Log is configured to *not* fsync() on all Append() calls
I20250626 13:59:13.497787 6504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=15 ignored=0} mutations{seen=23 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 13:59:13.498585 6504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097: Bootstrap complete.
I20250626 13:59:13.519541 6504 raft_consensus.cc:357] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff2ce1fa5a3439a92429a6dd10c6097" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 39915 } }
I20250626 13:59:13.521541 6504 raft_consensus.cc:738] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff2ce1fa5a3439a92429a6dd10c6097, State: Initialized, Role: FOLLOWER
I20250626 13:59:13.522333 6504 consensus_queue.cc:260] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff2ce1fa5a3439a92429a6dd10c6097" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 39915 } }
I20250626 13:59:13.522821 6504 raft_consensus.cc:397] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [term 5 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 13:59:13.523051 6504 raft_consensus.cc:491] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [term 5 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 13:59:13.523314 6504 raft_consensus.cc:3058] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [term 5 FOLLOWER]: Advancing to term 6
I20250626 13:59:13.528651 6504 raft_consensus.cc:513] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff2ce1fa5a3439a92429a6dd10c6097" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 39915 } }
I20250626 13:59:13.529357 6504 leader_election.cc:304] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dff2ce1fa5a3439a92429a6dd10c6097; no voters:
I20250626 13:59:13.531291 6504 leader_election.cc:290] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [CANDIDATE]: Term 6 election: Requested vote from peers
I20250626 13:59:13.531773 6508 raft_consensus.cc:2802] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [term 6 FOLLOWER]: Leader election won for term 6
I20250626 13:59:13.534726 6508 raft_consensus.cc:695] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [term 6 LEADER]: Becoming Leader. State: Replica: dff2ce1fa5a3439a92429a6dd10c6097, State: Running, Role: LEADER
I20250626 13:59:13.535638 6508 consensus_queue.cc:237] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 6, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff2ce1fa5a3439a92429a6dd10c6097" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 39915 } }
I20250626 13:59:13.536049 6504 sys_catalog.cc:564] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 13:59:13.545958 6509 sys_catalog.cc:455] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 leader_uuid: "dff2ce1fa5a3439a92429a6dd10c6097" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff2ce1fa5a3439a92429a6dd10c6097" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 39915 } } }
I20250626 13:59:13.546667 6509 sys_catalog.cc:458] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [sys.catalog]: This master's current role is: LEADER
I20250626 13:59:13.548787 6510 sys_catalog.cc:455] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dff2ce1fa5a3439a92429a6dd10c6097. Latest consensus state: current_term: 6 leader_uuid: "dff2ce1fa5a3439a92429a6dd10c6097" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff2ce1fa5a3439a92429a6dd10c6097" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 39915 } } }
I20250626 13:59:13.550185 6510 sys_catalog.cc:458] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097 [sys.catalog]: This master's current role is: LEADER
I20250626 13:59:13.557302 6514 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 13:59:13.572450 6514 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=216035caa9b64b13a7bc7dc18a6eed4d]
I20250626 13:59:13.574258 6514 catalog_manager.cc:671] Loaded metadata for table TestTable [id=9fe232cee3e44f598ef072c65193731a]
I20250626 13:59:13.576064 6514 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=a310ba57cf9c410394fdd12e608ba840]
I20250626 13:59:13.584648 6514 tablet_loader.cc:96] loaded metadata for tablet 0d6f39a2629e4584a8bbb8a5a40fe5ed (table TestTable1 [id=a310ba57cf9c410394fdd12e608ba840])
I20250626 13:59:13.586849 6514 tablet_loader.cc:96] loaded metadata for tablet 23a90d8fe96e47eaa5971c96f177b60a (table TestTable [id=9fe232cee3e44f598ef072c65193731a])
I20250626 13:59:13.588224 6514 tablet_loader.cc:96] loaded metadata for tablet 2d6e309ecdf64c5aaffdc5b24aff9086 (table TestTable2 [id=216035caa9b64b13a7bc7dc18a6eed4d])
I20250626 13:59:13.589758 6514 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 13:59:13.595603 6514 catalog_manager.cc:1261] Loaded cluster ID: fdd2a3c7727c496ca7c541014bb33fe4
I20250626 13:59:13.596673 6514 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 13:59:13.605590 6514 catalog_manager.cc:1506] Loading token signing keys...
I20250626 13:59:13.611568 6514 catalog_manager.cc:5966] T 00000000000000000000000000000000 P dff2ce1fa5a3439a92429a6dd10c6097: Loaded TSK: 0
I20250626 13:59:13.613466 6514 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250626 13:59:13.746214 6506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:59:13.746716 6506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:59:13.747231 6506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:59:13.780424 6506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:59:13.781353 6506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.107.65
I20250626 13:59:13.817754 6506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:40679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.107.65:41115
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.107.65
--webserver_port=42549
--tserver_master_addrs=127.2.107.126:39915
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.107.65
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:59:13.819100 6506 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:59:13.820813 6506 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:59:13.839283 6533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:59:13.840634 6535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:59:13.839340 6532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:59:14.986630 6534 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1142 milliseconds
I20250626 13:59:14.986990 6506 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 13:59:14.988729 6506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:59:14.992018 6506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:59:14.993491 6506 hybrid_clock.cc:648] HybridClock initialized: now 1750946354993425 us; error 84 us; skew 500 ppm
I20250626 13:59:14.994377 6506 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:59:15.001233 6506 webserver.cc:469] Webserver started at http://127.2.107.65:42549/ using document root <none> and password file <none>
I20250626 13:59:15.002172 6506 fs_manager.cc:362] Metadata directory not provided
I20250626 13:59:15.002399 6506 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:59:15.011312 6506 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20250626 13:59:15.016602 6542 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:59:15.018121 6506 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250626 13:59:15.018466 6506 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "adfc75fd62a24908b775fe41af11b077"
format_stamp: "Formatted at 2025-06-26 13:58:47 on dist-test-slave-7p9x"
I20250626 13:59:15.020409 6506 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:59:15.075774 6506 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:59:15.077314 6506 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:59:15.077749 6506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:59:15.080456 6506 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 13:59:15.087334 6549 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250626 13:59:15.106537 6506 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250626 13:59:15.106832 6506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.021s user 0.002s sys 0.000s
I20250626 13:59:15.107115 6506 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250626 13:59:15.113608 6549 tablet_bootstrap.cc:492] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077: Bootstrap starting.
I20250626 13:59:15.116290 6506 ts_tablet_manager.cc:610] Registered 2 tablets
I20250626 13:59:15.116516 6506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.009s user 0.007s sys 0.000s
I20250626 13:59:15.189613 6549 log.cc:826] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077: Log is configured to *not* fsync() on all Append() calls
I20250626 13:59:15.315456 6506 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.65:41115
I20250626 13:59:15.315654 6656 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.65:41115 every 8 connection(s)
I20250626 13:59:15.319353 6506 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250626 13:59:15.321861 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 6506
I20250626 13:59:15.324514 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.107.66:40681
--local_ip_for_outbound_sockets=127.2.107.66
--tserver_master_addrs=127.2.107.126:39915
--webserver_port=42749
--webserver_interface=127.2.107.66
--builtin_ntp_servers=127.2.107.84:40679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 13:59:15.340224 6549 tablet_bootstrap.cc:492] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 13:59:15.341703 6549 tablet_bootstrap.cc:492] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077: Bootstrap complete.
I20250626 13:59:15.344107 6549 ts_tablet_manager.cc:1397] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077: Time spent bootstrapping tablet: real 0.231s user 0.195s sys 0.028s
I20250626 13:59:15.364578 6549 raft_consensus.cc:357] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } } peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: false } }
I20250626 13:59:15.368017 6549 raft_consensus.cc:738] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: adfc75fd62a24908b775fe41af11b077, State: Initialized, Role: FOLLOWER
I20250626 13:59:15.369448 6549 consensus_queue.cc:260] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } } peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: false } }
I20250626 13:59:15.378810 6657 heartbeater.cc:344] Connected to a master server at 127.2.107.126:39915
I20250626 13:59:15.379441 6657 heartbeater.cc:461] Registering TS with master...
I20250626 13:59:15.379781 6549 ts_tablet_manager.cc:1428] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077: Time spent starting tablet: real 0.035s user 0.024s sys 0.004s
I20250626 13:59:15.380549 6549 tablet_bootstrap.cc:492] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P adfc75fd62a24908b775fe41af11b077: Bootstrap starting.
I20250626 13:59:15.380746 6657 heartbeater.cc:507] Master 127.2.107.126:39915 requested a full tablet report, sending...
I20250626 13:59:15.386847 6469 ts_manager.cc:194] Registered new tserver with Master: adfc75fd62a24908b775fe41af11b077 (127.2.107.65:41115)
I20250626 13:59:15.391665 6469 catalog_manager.cc:5582] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 reported cstate change: config changed from index -1 to 13, term changed from 0 to 2, VOTER 8e2735d1eb124fceae9b6b3c5d5ab261 (127.2.107.67) added, VOTER adfc75fd62a24908b775fe41af11b077 (127.2.107.65) added, VOTER f7c67fd060a84947a1abd37e5b54d1c9 (127.2.107.66) added. New cstate: current_term: 2 committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } } peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: false } } }
I20250626 13:59:15.470808 6469 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.107.65:45347
I20250626 13:59:15.475715 6657 heartbeater.cc:499] Master 127.2.107.126:39915 was elected leader, sending a full tablet report...
I20250626 13:59:15.580803 6549 tablet_bootstrap.cc:492] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P adfc75fd62a24908b775fe41af11b077: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 13:59:15.581743 6549 tablet_bootstrap.cc:492] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P adfc75fd62a24908b775fe41af11b077: Bootstrap complete.
I20250626 13:59:15.583178 6549 ts_tablet_manager.cc:1397] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P adfc75fd62a24908b775fe41af11b077: Time spent bootstrapping tablet: real 0.203s user 0.147s sys 0.012s
I20250626 13:59:15.585525 6549 raft_consensus.cc:357] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P adfc75fd62a24908b775fe41af11b077 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } } peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: false } }
I20250626 13:59:15.586182 6549 raft_consensus.cc:738] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P adfc75fd62a24908b775fe41af11b077 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: adfc75fd62a24908b775fe41af11b077, State: Initialized, Role: FOLLOWER
I20250626 13:59:15.586808 6549 consensus_queue.cc:260] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P adfc75fd62a24908b775fe41af11b077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } } peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: false } }
I20250626 13:59:15.588845 6549 ts_tablet_manager.cc:1428] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P adfc75fd62a24908b775fe41af11b077: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20250626 13:59:15.708889 6661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:59:15.709458 6661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:59:15.709975 6661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:59:15.743587 6661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:59:15.744478 6661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.107.66
I20250626 13:59:15.781539 6661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:40679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.107.66:40681
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.107.66
--webserver_port=42749
--tserver_master_addrs=127.2.107.126:39915
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.107.66
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:59:15.782950 6661 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:59:15.784643 6661 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:59:15.803182 6674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:59:15.803216 6676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:59:15.806294 6673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:59:17.001688 6675 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1190 milliseconds
I20250626 13:59:17.001808 6661 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 13:59:17.003715 6661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:59:17.005941 6661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:59:17.007324 6661 hybrid_clock.cc:648] HybridClock initialized: now 1750946357007275 us; error 62 us; skew 500 ppm
I20250626 13:59:17.008181 6661 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:59:17.015556 6661 webserver.cc:469] Webserver started at http://127.2.107.66:42749/ using document root <none> and password file <none>
I20250626 13:59:17.016456 6661 fs_manager.cc:362] Metadata directory not provided
I20250626 13:59:17.016661 6661 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:59:17.024504 6661 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20250626 13:59:17.030093 6683 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:59:17.031733 6661 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20250626 13:59:17.032038 6661 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "f7c67fd060a84947a1abd37e5b54d1c9"
format_stamp: "Formatted at 2025-06-26 13:58:49 on dist-test-slave-7p9x"
I20250626 13:59:17.034008 6661 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:59:17.095516 6661 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:59:17.097096 6661 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:59:17.097561 6661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:59:17.100099 6661 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 13:59:17.106397 6690 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250626 13:59:17.119395 6661 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250626 13:59:17.119652 6661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.002s sys 0.000s
I20250626 13:59:17.119946 6661 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250626 13:59:17.122041 6691 raft_consensus.cc:491] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 13:59:17.122488 6691 raft_consensus.cc:513] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } } peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: false } }
I20250626 13:59:17.124897 6691 leader_election.cc:290] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f7c67fd060a84947a1abd37e5b54d1c9 (127.2.107.66:40681), 8e2735d1eb124fceae9b6b3c5d5ab261 (127.2.107.67:32943)
I20250626 13:59:17.126833 6690 tablet_bootstrap.cc:492] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9: Bootstrap starting.
W20250626 13:59:17.129628 6544 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.107.67:32943: connect: Connection refused (error 111)
I20250626 13:59:17.132850 6661 ts_tablet_manager.cc:610] Registered 2 tablets
I20250626 13:59:17.133245 6661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.013s user 0.012s sys 0.000s
W20250626 13:59:17.134140 6544 leader_election.cc:336] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8e2735d1eb124fceae9b6b3c5d5ab261 (127.2.107.67:32943): Network error: Client connection negotiation failed: client connection to 127.2.107.67:32943: connect: Connection refused (error 111)
W20250626 13:59:17.134876 6546 leader_election.cc:336] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f7c67fd060a84947a1abd37e5b54d1c9 (127.2.107.66:40681): Network error: Client connection negotiation failed: client connection to 127.2.107.66:40681: connect: Connection refused (error 111)
I20250626 13:59:17.135313 6546 leader_election.cc:304] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: adfc75fd62a24908b775fe41af11b077; no voters: 8e2735d1eb124fceae9b6b3c5d5ab261, f7c67fd060a84947a1abd37e5b54d1c9
I20250626 13:59:17.136119 6691 raft_consensus.cc:2747] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 13:59:17.186228 6690 log.cc:826] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9: Log is configured to *not* fsync() on all Append() calls
I20250626 13:59:17.268435 6691 raft_consensus.cc:491] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P adfc75fd62a24908b775fe41af11b077 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 13:59:17.268831 6691 raft_consensus.cc:513] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P adfc75fd62a24908b775fe41af11b077 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } } peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: false } }
I20250626 13:59:17.270859 6691 leader_election.cc:290] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P adfc75fd62a24908b775fe41af11b077 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f7c67fd060a84947a1abd37e5b54d1c9 (127.2.107.66:40681), 8e2735d1eb124fceae9b6b3c5d5ab261 (127.2.107.67:32943)
W20250626 13:59:17.275640 6546 leader_election.cc:336] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P adfc75fd62a24908b775fe41af11b077 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f7c67fd060a84947a1abd37e5b54d1c9 (127.2.107.66:40681): Network error: Client connection negotiation failed: client connection to 127.2.107.66:40681: connect: Connection refused (error 111)
W20250626 13:59:17.276087 6544 leader_election.cc:336] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P adfc75fd62a24908b775fe41af11b077 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8e2735d1eb124fceae9b6b3c5d5ab261 (127.2.107.67:32943): Network error: Client connection negotiation failed: client connection to 127.2.107.67:32943: connect: Connection refused (error 111)
I20250626 13:59:17.276460 6544 leader_election.cc:304] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P adfc75fd62a24908b775fe41af11b077 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: adfc75fd62a24908b775fe41af11b077; no voters: 8e2735d1eb124fceae9b6b3c5d5ab261, f7c67fd060a84947a1abd37e5b54d1c9
I20250626 13:59:17.277202 6691 raft_consensus.cc:2747] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P adfc75fd62a24908b775fe41af11b077 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 13:59:17.305627 6661 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.66:40681
I20250626 13:59:17.305745 6801 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.66:40681 every 8 connection(s)
I20250626 13:59:17.309396 6661 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250626 13:59:17.312011 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 6661
I20250626 13:59:17.313989 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.107.67:32943
--local_ip_for_outbound_sockets=127.2.107.67
--tserver_master_addrs=127.2.107.126:39915
--webserver_port=34703
--webserver_interface=127.2.107.67
--builtin_ntp_servers=127.2.107.84:40679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 13:59:17.334761 6690 tablet_bootstrap.cc:492] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 13:59:17.336108 6690 tablet_bootstrap.cc:492] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9: Bootstrap complete.
I20250626 13:59:17.338126 6690 ts_tablet_manager.cc:1397] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9: Time spent bootstrapping tablet: real 0.212s user 0.169s sys 0.039s
I20250626 13:59:17.359196 6690 raft_consensus.cc:357] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } } peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: false } }
I20250626 13:59:17.361634 6690 raft_consensus.cc:738] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c67fd060a84947a1abd37e5b54d1c9, State: Initialized, Role: FOLLOWER
I20250626 13:59:17.362591 6690 consensus_queue.cc:260] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } } peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: false } }
I20250626 13:59:17.365733 6690 ts_tablet_manager.cc:1428] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9: Time spent starting tablet: real 0.027s user 0.021s sys 0.001s
I20250626 13:59:17.366400 6690 tablet_bootstrap.cc:492] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9: Bootstrap starting.
I20250626 13:59:17.371615 6802 heartbeater.cc:344] Connected to a master server at 127.2.107.126:39915
I20250626 13:59:17.372151 6802 heartbeater.cc:461] Registering TS with master...
I20250626 13:59:17.373432 6802 heartbeater.cc:507] Master 127.2.107.126:39915 requested a full tablet report, sending...
I20250626 13:59:17.378401 6468 ts_manager.cc:194] Registered new tserver with Master: f7c67fd060a84947a1abd37e5b54d1c9 (127.2.107.66:40681)
I20250626 13:59:17.382766 6468 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.107.66:39375
I20250626 13:59:17.387951 6802 heartbeater.cc:499] Master 127.2.107.126:39915 was elected leader, sending a full tablet report...
I20250626 13:59:17.498082 6690 tablet_bootstrap.cc:492] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 13:59:17.498766 6690 tablet_bootstrap.cc:492] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9: Bootstrap complete.
I20250626 13:59:17.499747 6690 ts_tablet_manager.cc:1397] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9: Time spent bootstrapping tablet: real 0.133s user 0.112s sys 0.018s
I20250626 13:59:17.501348 6690 raft_consensus.cc:357] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } } peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: false } }
I20250626 13:59:17.501837 6690 raft_consensus.cc:738] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c67fd060a84947a1abd37e5b54d1c9, State: Initialized, Role: FOLLOWER
I20250626 13:59:17.502475 6690 consensus_queue.cc:260] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } } peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: false } }
I20250626 13:59:17.504030 6690 ts_tablet_manager.cc:1428] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20250626 13:59:17.715564 6806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:59:17.716091 6806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:59:17.716585 6806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:59:17.749809 6806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:59:17.750679 6806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.107.67
I20250626 13:59:17.788794 6806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:40679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.107.67:32943
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.107.67
--webserver_port=34703
--tserver_master_addrs=127.2.107.126:39915
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.107.67
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:59:17.790186 6806 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:59:17.791795 6806 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:59:17.808296 6815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:59:18.670341 6820 raft_consensus.cc:491] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 13:59:18.670764 6820 raft_consensus.cc:513] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } } peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: false } }
I20250626 13:59:18.672194 6820 leader_election.cc:290] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f7c67fd060a84947a1abd37e5b54d1c9 (127.2.107.66:40681), 8e2735d1eb124fceae9b6b3c5d5ab261 (127.2.107.67:32943)
W20250626 13:59:18.688156 6544 leader_election.cc:336] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8e2735d1eb124fceae9b6b3c5d5ab261 (127.2.107.67:32943): Network error: Client connection negotiation failed: client connection to 127.2.107.67:32943: connect: Connection refused (error 111)
I20250626 13:59:18.697438 6757 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "23a90d8fe96e47eaa5971c96f177b60a" candidate_uuid: "adfc75fd62a24908b775fe41af11b077" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" is_pre_election: true
I20250626 13:59:18.698526 6757 raft_consensus.cc:2466] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adfc75fd62a24908b775fe41af11b077 in term 2.
I20250626 13:59:18.700452 6546 leader_election.cc:304] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: adfc75fd62a24908b775fe41af11b077, f7c67fd060a84947a1abd37e5b54d1c9; no voters: 8e2735d1eb124fceae9b6b3c5d5ab261
I20250626 13:59:18.701439 6820 raft_consensus.cc:2802] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 13:59:18.701773 6820 raft_consensus.cc:491] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 13:59:18.702030 6820 raft_consensus.cc:3058] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [term 2 FOLLOWER]: Advancing to term 3
I20250626 13:59:18.708863 6820 raft_consensus.cc:513] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } } peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: false } }
I20250626 13:59:18.710418 6820 leader_election.cc:290] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [CANDIDATE]: Term 3 election: Requested vote from peers f7c67fd060a84947a1abd37e5b54d1c9 (127.2.107.66:40681), 8e2735d1eb124fceae9b6b3c5d5ab261 (127.2.107.67:32943)
I20250626 13:59:18.711640 6757 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "23a90d8fe96e47eaa5971c96f177b60a" candidate_uuid: "adfc75fd62a24908b775fe41af11b077" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "f7c67fd060a84947a1abd37e5b54d1c9"
I20250626 13:59:18.712339 6757 raft_consensus.cc:3058] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9 [term 2 FOLLOWER]: Advancing to term 3
W20250626 13:59:18.713873 6544 leader_election.cc:336] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 8e2735d1eb124fceae9b6b3c5d5ab261 (127.2.107.67:32943): Network error: Client connection negotiation failed: client connection to 127.2.107.67:32943: connect: Connection refused (error 111)
I20250626 13:59:18.722651 6757 raft_consensus.cc:2466] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adfc75fd62a24908b775fe41af11b077 in term 3.
I20250626 13:59:18.723862 6546 leader_election.cc:304] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: adfc75fd62a24908b775fe41af11b077, f7c67fd060a84947a1abd37e5b54d1c9; no voters: 8e2735d1eb124fceae9b6b3c5d5ab261
I20250626 13:59:18.724674 6820 raft_consensus.cc:2802] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 13:59:18.726343 6820 raft_consensus.cc:695] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [term 3 LEADER]: Becoming Leader. State: Replica: adfc75fd62a24908b775fe41af11b077, State: Running, Role: LEADER
I20250626 13:59:18.727246 6820 consensus_queue.cc:237] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } } peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: false } }
I20250626 13:59:18.739182 6468 catalog_manager.cc:5582] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 reported cstate change: term changed from 2 to 3, leader changed from <none> to adfc75fd62a24908b775fe41af11b077 (127.2.107.65). New cstate: current_term: 3 leader_uuid: "adfc75fd62a24908b775fe41af11b077" committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 13:59:18.795110 6831 raft_consensus.cc:491] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 13:59:18.795571 6831 raft_consensus.cc:513] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } } peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: false } }
I20250626 13:59:18.798008 6831 leader_election.cc:290] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers adfc75fd62a24908b775fe41af11b077 (127.2.107.65:41115), 8e2735d1eb124fceae9b6b3c5d5ab261 (127.2.107.67:32943)
W20250626 13:59:18.803406 6685 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.107.67:32943: connect: Connection refused (error 111)
W20250626 13:59:18.808542 6685 leader_election.cc:336] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8e2735d1eb124fceae9b6b3c5d5ab261 (127.2.107.67:32943): Network error: Client connection negotiation failed: client connection to 127.2.107.67:32943: connect: Connection refused (error 111)
I20250626 13:59:18.814913 6612 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d6f39a2629e4584a8bbb8a5a40fe5ed" candidate_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "adfc75fd62a24908b775fe41af11b077" is_pre_election: true
I20250626 13:59:18.815582 6612 raft_consensus.cc:2466] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P adfc75fd62a24908b775fe41af11b077 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c67fd060a84947a1abd37e5b54d1c9 in term 2.
I20250626 13:59:18.817214 6686 leader_election.cc:304] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: adfc75fd62a24908b775fe41af11b077, f7c67fd060a84947a1abd37e5b54d1c9; no voters: 8e2735d1eb124fceae9b6b3c5d5ab261
I20250626 13:59:18.817962 6831 raft_consensus.cc:2802] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 13:59:18.818308 6831 raft_consensus.cc:491] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 13:59:18.818619 6831 raft_consensus.cc:3058] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [term 2 FOLLOWER]: Advancing to term 3
I20250626 13:59:18.825392 6831 raft_consensus.cc:513] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } } peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: false } }
I20250626 13:59:18.827308 6831 leader_election.cc:290] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [CANDIDATE]: Term 3 election: Requested vote from peers adfc75fd62a24908b775fe41af11b077 (127.2.107.65:41115), 8e2735d1eb124fceae9b6b3c5d5ab261 (127.2.107.67:32943)
I20250626 13:59:18.828241 6612 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d6f39a2629e4584a8bbb8a5a40fe5ed" candidate_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "adfc75fd62a24908b775fe41af11b077"
I20250626 13:59:18.828696 6612 raft_consensus.cc:3058] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P adfc75fd62a24908b775fe41af11b077 [term 2 FOLLOWER]: Advancing to term 3
W20250626 13:59:18.831187 6685 leader_election.cc:336] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 8e2735d1eb124fceae9b6b3c5d5ab261 (127.2.107.67:32943): Network error: Client connection negotiation failed: client connection to 127.2.107.67:32943: connect: Connection refused (error 111)
I20250626 13:59:18.833797 6612 raft_consensus.cc:2466] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P adfc75fd62a24908b775fe41af11b077 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c67fd060a84947a1abd37e5b54d1c9 in term 3.
I20250626 13:59:18.834846 6686 leader_election.cc:304] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: adfc75fd62a24908b775fe41af11b077, f7c67fd060a84947a1abd37e5b54d1c9; no voters: 8e2735d1eb124fceae9b6b3c5d5ab261
I20250626 13:59:18.835511 6831 raft_consensus.cc:2802] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 13:59:18.837587 6831 raft_consensus.cc:695] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [term 3 LEADER]: Becoming Leader. State: Replica: f7c67fd060a84947a1abd37e5b54d1c9, State: Running, Role: LEADER
I20250626 13:59:18.838532 6831 consensus_queue.cc:237] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } } peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: false } }
I20250626 13:59:18.851297 6468 catalog_manager.cc:5582] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250626 13:59:17.808349 6814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:59:17.809672 6817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:59:18.947429 6816 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 13:59:18.947429 6806 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 13:59:18.954476 6806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:59:18.957067 6806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:59:18.958506 6806 hybrid_clock.cc:648] HybridClock initialized: now 1750946358958473 us; error 44 us; skew 500 ppm
I20250626 13:59:18.959385 6806 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:59:18.969050 6806 webserver.cc:469] Webserver started at http://127.2.107.67:34703/ using document root <none> and password file <none>
I20250626 13:59:18.970003 6806 fs_manager.cc:362] Metadata directory not provided
I20250626 13:59:18.970207 6806 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:59:18.978168 6806 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.001s
I20250626 13:59:18.983299 6842 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:59:18.984579 6806 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20250626 13:59:18.984912 6806 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "8e2735d1eb124fceae9b6b3c5d5ab261"
format_stamp: "Formatted at 2025-06-26 13:58:50 on dist-test-slave-7p9x"
I20250626 13:59:18.986945 6806 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:59:19.039256 6806 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:59:19.040700 6806 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:59:19.041126 6806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:59:19.043779 6806 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 13:59:19.049413 6849 ts_tablet_manager.cc:542] Loading tablet metadata (0/3 complete)
I20250626 13:59:19.067385 6806 ts_tablet_manager.cc:579] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20250626 13:59:19.067642 6806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.020s user 0.002s sys 0.001s
I20250626 13:59:19.068004 6806 ts_tablet_manager.cc:594] Registering tablets (0/3 complete)
I20250626 13:59:19.074151 6849 tablet_bootstrap.cc:492] T 23a90d8fe96e47eaa5971c96f177b60a P 8e2735d1eb124fceae9b6b3c5d5ab261: Bootstrap starting.
I20250626 13:59:19.080104 6806 ts_tablet_manager.cc:610] Registered 3 tablets
I20250626 13:59:19.080318 6806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.012s user 0.010s sys 0.000s
I20250626 13:59:19.132738 6849 log.cc:826] T 23a90d8fe96e47eaa5971c96f177b60a P 8e2735d1eb124fceae9b6b3c5d5ab261: Log is configured to *not* fsync() on all Append() calls
W20250626 13:59:19.245602 6544 consensus_peers.cc:487] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 -> Peer 8e2735d1eb124fceae9b6b3c5d5ab261 (127.2.107.67:32943): Couldn't send request to peer 8e2735d1eb124fceae9b6b3c5d5ab261. Status: Network error: Client connection negotiation failed: client connection to 127.2.107.67:32943: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 13:59:19.250524 6806 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.67:32943
I20250626 13:59:19.250622 6959 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.67:32943 every 8 connection(s)
I20250626 13:59:19.254103 6806 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250626 13:59:19.256906 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 6806
I20250626 13:59:19.261446 6757 raft_consensus.cc:1273] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9 [term 3 FOLLOWER]: Refusing update from remote peer adfc75fd62a24908b775fe41af11b077: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20250626 13:59:19.263561 6948 consensus_queue.cc:1035] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250626 13:59:19.265115 6612 raft_consensus.cc:1273] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P adfc75fd62a24908b775fe41af11b077 [term 3 FOLLOWER]: Refusing update from remote peer f7c67fd060a84947a1abd37e5b54d1c9: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20250626 13:59:19.267052 6836 consensus_queue.cc:1035] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20250626 13:59:19.278609 6849 tablet_bootstrap.cc:492] T 23a90d8fe96e47eaa5971c96f177b60a P 8e2735d1eb124fceae9b6b3c5d5ab261: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 13:59:19.279748 6849 tablet_bootstrap.cc:492] T 23a90d8fe96e47eaa5971c96f177b60a P 8e2735d1eb124fceae9b6b3c5d5ab261: Bootstrap complete.
I20250626 13:59:19.281538 6849 ts_tablet_manager.cc:1397] T 23a90d8fe96e47eaa5971c96f177b60a P 8e2735d1eb124fceae9b6b3c5d5ab261: Time spent bootstrapping tablet: real 0.208s user 0.159s sys 0.040s
I20250626 13:59:19.315343 6849 raft_consensus.cc:357] T 23a90d8fe96e47eaa5971c96f177b60a P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } } peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: false } }
I20250626 13:59:19.318876 6849 raft_consensus.cc:738] T 23a90d8fe96e47eaa5971c96f177b60a P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e2735d1eb124fceae9b6b3c5d5ab261, State: Initialized, Role: FOLLOWER
I20250626 13:59:19.319928 6849 consensus_queue.cc:260] T 23a90d8fe96e47eaa5971c96f177b60a P 8e2735d1eb124fceae9b6b3c5d5ab261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } } peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: false } }
I20250626 13:59:19.351044 6960 heartbeater.cc:344] Connected to a master server at 127.2.107.126:39915
I20250626 13:59:19.351612 6960 heartbeater.cc:461] Registering TS with master...
I20250626 13:59:19.352945 6960 heartbeater.cc:507] Master 127.2.107.126:39915 requested a full tablet report, sending...
I20250626 13:59:19.358407 6468 ts_manager.cc:194] Registered new tserver with Master: 8e2735d1eb124fceae9b6b3c5d5ab261 (127.2.107.67:32943)
W20250626 13:59:19.363741 6685 consensus_peers.cc:487] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P f7c67fd060a84947a1abd37e5b54d1c9 -> Peer 8e2735d1eb124fceae9b6b3c5d5ab261 (127.2.107.67:32943): Couldn't send request to peer 8e2735d1eb124fceae9b6b3c5d5ab261. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250626 13:59:19.364408 6468 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.107.67:47349
I20250626 13:59:19.370285 2477 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 13:59:19.380730 2477 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 13:59:19.386260 2477 ts_itest-base.cc:246] Waiting for 1 tablets on tserver adfc75fd62a24908b775fe41af11b077 to finish bootstrapping
I20250626 13:59:19.402365 6960 heartbeater.cc:499] Master 127.2.107.126:39915 was elected leader, sending a full tablet report...
I20250626 13:59:19.403230 6849 ts_tablet_manager.cc:1428] T 23a90d8fe96e47eaa5971c96f177b60a P 8e2735d1eb124fceae9b6b3c5d5ab261: Time spent starting tablet: real 0.121s user 0.029s sys 0.052s
I20250626 13:59:19.403962 6849 tablet_bootstrap.cc:492] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261: Bootstrap starting.
I20250626 13:59:19.438365 6612 consensus_queue.cc:237] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 13, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } } peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: false } }
I20250626 13:59:19.443082 6757 raft_consensus.cc:1273] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9 [term 3 FOLLOWER]: Refusing update from remote peer adfc75fd62a24908b775fe41af11b077: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20250626 13:59:19.444667 6969 consensus_queue.cc:1035] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20250626 13:59:19.450773 2477 ts_itest-base.cc:246] Waiting for 1 tablets on tserver f7c67fd060a84947a1abd37e5b54d1c9 to finish bootstrapping
I20250626 13:59:19.451403 6969 raft_consensus.cc:2953] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [term 3 LEADER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER 8e2735d1eb124fceae9b6b3c5d5ab261 (127.2.107.67) evicted. New config: { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } } peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: false } } }
I20250626 13:59:19.454382 6756 raft_consensus.cc:2953] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9 [term 3 FOLLOWER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER 8e2735d1eb124fceae9b6b3c5d5ab261 (127.2.107.67) evicted. New config: { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } } peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: false } } }
I20250626 13:59:19.476234 6469 catalog_manager.cc:5582] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9 reported cstate change: config changed from index 13 to 15, VOTER 8e2735d1eb124fceae9b6b3c5d5ab261 (127.2.107.67) evicted. New cstate: current_term: 3 leader_uuid: "adfc75fd62a24908b775fe41af11b077" committed_config { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } } peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } attrs { promote: false } } }
I20250626 13:59:19.483397 6455 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 23a90d8fe96e47eaa5971c96f177b60a with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 13:59:19.491004 2477 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 8e2735d1eb124fceae9b6b3c5d5ab261 to finish bootstrapping
I20250626 13:59:19.504030 6612 consensus_queue.cc:237] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 3.15, Last appended by leader: 13, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } }
I20250626 13:59:19.506691 6969 raft_consensus.cc:2953] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 [term 3 LEADER]: Committing config change with OpId 3.16: config changed from index 15 to 16, VOTER f7c67fd060a84947a1abd37e5b54d1c9 (127.2.107.66) evicted. New config: { opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } } }
I20250626 13:59:19.517158 6455 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 23a90d8fe96e47eaa5971c96f177b60a with cas_config_opid_index 15: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 13:59:19.519273 6468 catalog_manager.cc:5582] T 23a90d8fe96e47eaa5971c96f177b60a P adfc75fd62a24908b775fe41af11b077 reported cstate change: config changed from index 15 to 16, VOTER f7c67fd060a84947a1abd37e5b54d1c9 (127.2.107.66) evicted. New cstate: current_term: 3 leader_uuid: "adfc75fd62a24908b775fe41af11b077" committed_config { opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } health_report { overall_health: HEALTHY } } }
I20250626 13:59:19.522841 6892 tablet_service.cc:1515] Processing DeleteTablet for tablet 23a90d8fe96e47eaa5971c96f177b60a with delete_type TABLET_DATA_TOMBSTONED (TS 8e2735d1eb124fceae9b6b3c5d5ab261 not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:41408
I20250626 13:59:19.525990 6984 tablet_replica.cc:331] stopping tablet replica
I20250626 13:59:19.527029 6984 raft_consensus.cc:2241] T 23a90d8fe96e47eaa5971c96f177b60a P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 13:59:19.527565 6984 raft_consensus.cc:2270] T 23a90d8fe96e47eaa5971c96f177b60a P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 13:59:19.531909 6984 ts_tablet_manager.cc:1905] T 23a90d8fe96e47eaa5971c96f177b60a P 8e2735d1eb124fceae9b6b3c5d5ab261: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 13:59:19.539367 6737 tablet_service.cc:1515] Processing DeleteTablet for tablet 23a90d8fe96e47eaa5971c96f177b60a with delete_type TABLET_DATA_TOMBSTONED (TS f7c67fd060a84947a1abd37e5b54d1c9 not found in new config with opid_index 16) from {username='slave'} at 127.0.0.1:39456
I20250626 13:59:19.543799 6985 tablet_replica.cc:331] stopping tablet replica
I20250626 13:59:19.544447 6985 raft_consensus.cc:2241] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250626 13:59:19.545040 6985 raft_consensus.cc:2270] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 13:59:19.549439 6985 ts_tablet_manager.cc:1905] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 13:59:19.550890 6984 ts_tablet_manager.cc:1918] T 23a90d8fe96e47eaa5971c96f177b60a P 8e2735d1eb124fceae9b6b3c5d5ab261: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.13
I20250626 13:59:19.551298 6984 log.cc:1199] T 23a90d8fe96e47eaa5971c96f177b60a P 8e2735d1eb124fceae9b6b3c5d5ab261: Deleting WAL directory at /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal/wals/23a90d8fe96e47eaa5971c96f177b60a
I20250626 13:59:19.553049 6454 catalog_manager.cc:4928] TS 8e2735d1eb124fceae9b6b3c5d5ab261 (127.2.107.67:32943): tablet 23a90d8fe96e47eaa5971c96f177b60a (table TestTable [id=9fe232cee3e44f598ef072c65193731a]) successfully deleted
I20250626 13:59:19.560730 6985 ts_tablet_manager.cc:1918] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.15
I20250626 13:59:19.561194 6985 log.cc:1199] T 23a90d8fe96e47eaa5971c96f177b60a P f7c67fd060a84947a1abd37e5b54d1c9: Deleting WAL directory at /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal/wals/23a90d8fe96e47eaa5971c96f177b60a
I20250626 13:59:19.562914 6456 catalog_manager.cc:4928] TS f7c67fd060a84947a1abd37e5b54d1c9 (127.2.107.66:40681): tablet 23a90d8fe96e47eaa5971c96f177b60a (table TestTable [id=9fe232cee3e44f598ef072c65193731a]) successfully deleted
I20250626 13:59:19.582980 6849 tablet_bootstrap.cc:492] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 13:59:19.583644 6849 tablet_bootstrap.cc:492] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261: Bootstrap complete.
I20250626 13:59:19.584599 6849 ts_tablet_manager.cc:1397] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261: Time spent bootstrapping tablet: real 0.181s user 0.135s sys 0.015s
I20250626 13:59:19.586807 6849 raft_consensus.cc:357] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } }
I20250626 13:59:19.587241 6849 raft_consensus.cc:738] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e2735d1eb124fceae9b6b3c5d5ab261, State: Initialized, Role: FOLLOWER
I20250626 13:59:19.587714 6849 consensus_queue.cc:260] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } }
I20250626 13:59:19.588120 6849 raft_consensus.cc:397] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 13:59:19.588423 6849 raft_consensus.cc:491] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 13:59:19.588855 6849 raft_consensus.cc:3058] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 2 FOLLOWER]: Advancing to term 3
I20250626 13:59:19.592519 6849 raft_consensus.cc:513] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } }
I20250626 13:59:19.593215 6849 leader_election.cc:304] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e2735d1eb124fceae9b6b3c5d5ab261; no voters:
I20250626 13:59:19.593721 6849 leader_election.cc:290] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250626 13:59:19.594111 6970 raft_consensus.cc:2802] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 13:59:19.595929 6849 ts_tablet_manager.cc:1428] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261: Time spent starting tablet: real 0.011s user 0.014s sys 0.000s
I20250626 13:59:19.596131 6970 raft_consensus.cc:695] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 3 LEADER]: Becoming Leader. State: Replica: 8e2735d1eb124fceae9b6b3c5d5ab261, State: Running, Role: LEADER
I20250626 13:59:19.596619 6849 tablet_bootstrap.cc:492] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P 8e2735d1eb124fceae9b6b3c5d5ab261: Bootstrap starting.
I20250626 13:59:19.596843 6970 consensus_queue.cc:237] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } }
I20250626 13:59:19.608654 6468 catalog_manager.cc:5582] T 2d6e309ecdf64c5aaffdc5b24aff9086 P 8e2735d1eb124fceae9b6b3c5d5ab261 reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } health_report { overall_health: HEALTHY } } }
I20250626 13:59:19.704109 6849 tablet_bootstrap.cc:492] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P 8e2735d1eb124fceae9b6b3c5d5ab261: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 13:59:19.704806 6849 tablet_bootstrap.cc:492] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P 8e2735d1eb124fceae9b6b3c5d5ab261: Bootstrap complete.
I20250626 13:59:19.705825 6849 ts_tablet_manager.cc:1397] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P 8e2735d1eb124fceae9b6b3c5d5ab261: Time spent bootstrapping tablet: real 0.109s user 0.089s sys 0.019s
I20250626 13:59:19.707298 6849 raft_consensus.cc:357] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } } peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: false } }
I20250626 13:59:19.707746 6849 raft_consensus.cc:738] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e2735d1eb124fceae9b6b3c5d5ab261, State: Initialized, Role: FOLLOWER
I20250626 13:59:19.708197 6849 consensus_queue.cc:260] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P 8e2735d1eb124fceae9b6b3c5d5ab261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f7c67fd060a84947a1abd37e5b54d1c9" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 40681 } } peers { permanent_uuid: "adfc75fd62a24908b775fe41af11b077" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41115 } attrs { promote: false } } peers { permanent_uuid: "8e2735d1eb124fceae9b6b3c5d5ab261" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 32943 } attrs { promote: false } }
I20250626 13:59:19.709743 6849 ts_tablet_manager.cc:1428] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P 8e2735d1eb124fceae9b6b3c5d5ab261: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250626 13:59:20.252167 6912 raft_consensus.cc:3058] T 0d6f39a2629e4584a8bbb8a5a40fe5ed P 8e2735d1eb124fceae9b6b3c5d5ab261 [term 2 FOLLOWER]: Advancing to term 3
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/kudu-admin-test.cc:3914: Failure
Failed
Bad status: Timed out: status_and_schema {
tablet_status {
tablet_id: "0d6f39a2629e4584a8bbb8a5a40fe5ed"
table_name: "TestTable1"
state: RUNNING
last_status: "Bootstrap complete."
estimated_on_disk_size: 8415795
tablet_data_state: TABLET_DATA_READY
partition {
partition_key_start: ""
partition_key_end: ""
}
data_dirs: "/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/data"
}
schema {
columns {
id: 10
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
id: 11
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
id: 12
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
partition_schema {
range_schema {
columns {
id: 10
}
}
}
schema_version: 0
role: FOLLOWER
}
status_and_schema {
tablet_status {
tablet_id: "2d6e309ecdf64c5aaffdc5b24aff9086"
table_name: "TestTable2"
state: RUNNING
last_status: "Bootstrap complete."
estimated_on_disk_size: 8415713
tablet_data_state: TABLET_DATA_READY
partition {
partition_key_start: ""
partition_key_end: ""
}
data_dirs: "/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/data"
}
schema {
columns {
id: 10
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
id: 11
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
id: 12
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
partition_schema {
range_schema {
columns {
id: 10
}
}
}
schema_version: 0
role: LEADER
}
status_and_schema {
tablet_status {
tablet_id: "23a90d8fe96e47eaa5971c96f177b60a"
table_name: "TestTable"
state: STOPPED
last_status: "Deleted tablet blocks from disk"
estimated_on_disk_size: 8401339
tablet_data_state: TABLET_DATA_TOMBSTONED
partition {
partition_key_start: ""
partition_key_end: ""
}
}
schema {
columns {
id: 10
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
id: 11
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
id: 12
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
partition_schema {
range_schema {
columns {
id: 10
}
}
}
schema_version: 0
role: UNKNOWN_ROLE
}
I20250626 13:59:39.517711 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 6506
I20250626 13:59:39.538671 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 6661
I20250626 13:59:39.562124 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 6806
I20250626 13:59:39.586750 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 6436
2025-06-26T13:59:39Z chronyd exiting
I20250626 13:59:39.630407 2477 test_util.cc:183] -----------------------------------------------
I20250626 13:59:39.630601 2477 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750946263504804-2477-0
[ FAILED ] AdminCliTest.TestRebuildTables (55467 ms)
[----------] 5 tests from AdminCliTest (116059 ms total)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest
[ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4
I20250626 13:59:39.634063 2477 test_util.cc:276] Using random seed: -117795466
I20250626 13:59:39.638149 2477 ts_itest-base.cc:115] Starting cluster with:
I20250626 13:59:39.638300 2477 ts_itest-base.cc:116] --------------
I20250626 13:59:39.638438 2477 ts_itest-base.cc:117] 5 tablet servers
I20250626 13:59:39.638561 2477 ts_itest-base.cc:118] 3 replicas per TS
I20250626 13:59:39.638688 2477 ts_itest-base.cc:119] --------------
2025-06-26T13:59:39Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-26T13:59:39Z Disabled control of system clock
I20250626 13:59:39.679615 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.107.126:45791
--webserver_interface=127.2.107.126
--webserver_port=0
--builtin_ntp_servers=127.2.107.84:37923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.107.126:45791
--raft_prepare_replacement_before_eviction=true with env {}
W20250626 13:59:39.980315 7019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:59:39.980940 7019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:59:39.981422 7019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:59:40.012276 7019 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250626 13:59:40.012717 7019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 13:59:40.012972 7019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:59:40.013228 7019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 13:59:40.013453 7019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 13:59:40.048770 7019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:37923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.107.126:45791
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.107.126:45791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.107.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:59:40.050110 7019 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:59:40.051760 7019 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:59:40.067231 7025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:59:40.067250 7028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:59:40.067258 7026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:59:40.068627 7019 server_base.cc:1048] running on GCE node
I20250626 13:59:41.182332 7019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:59:41.185441 7019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:59:41.186903 7019 hybrid_clock.cc:648] HybridClock initialized: now 1750946381186852 us; error 65 us; skew 500 ppm
I20250626 13:59:41.187752 7019 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:59:41.199234 7019 webserver.cc:469] Webserver started at http://127.2.107.126:40445/ using document root <none> and password file <none>
I20250626 13:59:41.200222 7019 fs_manager.cc:362] Metadata directory not provided
I20250626 13:59:41.200431 7019 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:59:41.200924 7019 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 13:59:41.205466 7019 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d15e95349e6146a5b068d949f6fab514"
format_stamp: "Formatted at 2025-06-26 13:59:41 on dist-test-slave-7p9x"
I20250626 13:59:41.206652 7019 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d15e95349e6146a5b068d949f6fab514"
format_stamp: "Formatted at 2025-06-26 13:59:41 on dist-test-slave-7p9x"
I20250626 13:59:41.214246 7019 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20250626 13:59:41.220091 7035 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:59:41.221347 7019 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20250626 13:59:41.221724 7019 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d15e95349e6146a5b068d949f6fab514"
format_stamp: "Formatted at 2025-06-26 13:59:41 on dist-test-slave-7p9x"
I20250626 13:59:41.222159 7019 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:59:41.278894 7019 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:59:41.280413 7019 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:59:41.280969 7019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:59:41.351459 7019 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.126:45791
I20250626 13:59:41.351528 7086 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.126:45791 every 8 connection(s)
I20250626 13:59:41.354115 7019 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250626 13:59:41.359773 7087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:59:41.361898 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 7019
I20250626 13:59:41.362315 2477 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250626 13:59:41.379276 7087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d15e95349e6146a5b068d949f6fab514: Bootstrap starting.
I20250626 13:59:41.385051 7087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d15e95349e6146a5b068d949f6fab514: Neither blocks nor log segments found. Creating new log.
I20250626 13:59:41.386866 7087 log.cc:826] T 00000000000000000000000000000000 P d15e95349e6146a5b068d949f6fab514: Log is configured to *not* fsync() on all Append() calls
I20250626 13:59:41.391762 7087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d15e95349e6146a5b068d949f6fab514: No bootstrap required, opened a new log
I20250626 13:59:41.410252 7087 raft_consensus.cc:357] T 00000000000000000000000000000000 P d15e95349e6146a5b068d949f6fab514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d15e95349e6146a5b068d949f6fab514" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 45791 } }
I20250626 13:59:41.411108 7087 raft_consensus.cc:383] T 00000000000000000000000000000000 P d15e95349e6146a5b068d949f6fab514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 13:59:41.411316 7087 raft_consensus.cc:738] T 00000000000000000000000000000000 P d15e95349e6146a5b068d949f6fab514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d15e95349e6146a5b068d949f6fab514, State: Initialized, Role: FOLLOWER
I20250626 13:59:41.411901 7087 consensus_queue.cc:260] T 00000000000000000000000000000000 P d15e95349e6146a5b068d949f6fab514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d15e95349e6146a5b068d949f6fab514" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 45791 } }
I20250626 13:59:41.412381 7087 raft_consensus.cc:397] T 00000000000000000000000000000000 P d15e95349e6146a5b068d949f6fab514 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 13:59:41.412603 7087 raft_consensus.cc:491] T 00000000000000000000000000000000 P d15e95349e6146a5b068d949f6fab514 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 13:59:41.412854 7087 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d15e95349e6146a5b068d949f6fab514 [term 0 FOLLOWER]: Advancing to term 1
I20250626 13:59:41.417097 7087 raft_consensus.cc:513] T 00000000000000000000000000000000 P d15e95349e6146a5b068d949f6fab514 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d15e95349e6146a5b068d949f6fab514" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 45791 } }
I20250626 13:59:41.418056 7087 leader_election.cc:304] T 00000000000000000000000000000000 P d15e95349e6146a5b068d949f6fab514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d15e95349e6146a5b068d949f6fab514; no voters:
I20250626 13:59:41.419950 7087 leader_election.cc:290] T 00000000000000000000000000000000 P d15e95349e6146a5b068d949f6fab514 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 13:59:41.420588 7092 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d15e95349e6146a5b068d949f6fab514 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 13:59:41.422531 7092 raft_consensus.cc:695] T 00000000000000000000000000000000 P d15e95349e6146a5b068d949f6fab514 [term 1 LEADER]: Becoming Leader. State: Replica: d15e95349e6146a5b068d949f6fab514, State: Running, Role: LEADER
I20250626 13:59:41.423291 7092 consensus_queue.cc:237] T 00000000000000000000000000000000 P d15e95349e6146a5b068d949f6fab514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d15e95349e6146a5b068d949f6fab514" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 45791 } }
I20250626 13:59:41.423791 7087 sys_catalog.cc:564] T 00000000000000000000000000000000 P d15e95349e6146a5b068d949f6fab514 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 13:59:41.433908 7094 sys_catalog.cc:455] T 00000000000000000000000000000000 P d15e95349e6146a5b068d949f6fab514 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d15e95349e6146a5b068d949f6fab514. Latest consensus state: current_term: 1 leader_uuid: "d15e95349e6146a5b068d949f6fab514" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d15e95349e6146a5b068d949f6fab514" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 45791 } } }
I20250626 13:59:41.434746 7094 sys_catalog.cc:458] T 00000000000000000000000000000000 P d15e95349e6146a5b068d949f6fab514 [sys.catalog]: This master's current role is: LEADER
I20250626 13:59:41.435204 7093 sys_catalog.cc:455] T 00000000000000000000000000000000 P d15e95349e6146a5b068d949f6fab514 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d15e95349e6146a5b068d949f6fab514" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d15e95349e6146a5b068d949f6fab514" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 45791 } } }
I20250626 13:59:41.435891 7093 sys_catalog.cc:458] T 00000000000000000000000000000000 P d15e95349e6146a5b068d949f6fab514 [sys.catalog]: This master's current role is: LEADER
I20250626 13:59:41.438632 7102 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 13:59:41.449458 7102 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 13:59:41.468828 7102 catalog_manager.cc:1349] Generated new cluster ID: d83226660f0b4d93b6818369ba4177c4
I20250626 13:59:41.469213 7102 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 13:59:41.486737 7102 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 13:59:41.488276 7102 catalog_manager.cc:1506] Loading token signing keys...
I20250626 13:59:41.503005 7102 catalog_manager.cc:5955] T 00000000000000000000000000000000 P d15e95349e6146a5b068d949f6fab514: Generated new TSK 0
I20250626 13:59:41.504174 7102 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 13:59:41.520017 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.107.65:0
--local_ip_for_outbound_sockets=127.2.107.65
--webserver_interface=127.2.107.65
--webserver_port=0
--tserver_master_addrs=127.2.107.126:45791
--builtin_ntp_servers=127.2.107.84:37923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250626 13:59:41.834461 7111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:59:41.835127 7111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:59:41.835638 7111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:59:41.868167 7111 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250626 13:59:41.868561 7111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:59:41.869393 7111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.107.65
I20250626 13:59:41.905865 7111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:37923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.107.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.107.65
--webserver_port=0
--tserver_master_addrs=127.2.107.126:45791
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.107.65
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:59:41.907198 7111 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:59:41.909021 7111 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:59:41.927512 7118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:59:41.929479 7120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:59:43.027575 7119 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 13:59:43.027678 7111 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250626 13:59:41.927539 7117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:59:43.037390 7111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:59:43.040220 7111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:59:43.041697 7111 hybrid_clock.cc:648] HybridClock initialized: now 1750946383041633 us; error 77 us; skew 500 ppm
I20250626 13:59:43.042623 7111 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:59:43.049381 7111 webserver.cc:469] Webserver started at http://127.2.107.65:40959/ using document root <none> and password file <none>
I20250626 13:59:43.050316 7111 fs_manager.cc:362] Metadata directory not provided
I20250626 13:59:43.050527 7111 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:59:43.051265 7111 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 13:59:43.056075 7111 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "65bdee5c7aac49308de217fdb95386ca"
format_stamp: "Formatted at 2025-06-26 13:59:43 on dist-test-slave-7p9x"
I20250626 13:59:43.057237 7111 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "65bdee5c7aac49308de217fdb95386ca"
format_stamp: "Formatted at 2025-06-26 13:59:43 on dist-test-slave-7p9x"
I20250626 13:59:43.065021 7111 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.002s
I20250626 13:59:43.070991 7127 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:59:43.072345 7111 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250626 13:59:43.072718 7111 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "65bdee5c7aac49308de217fdb95386ca"
format_stamp: "Formatted at 2025-06-26 13:59:43 on dist-test-slave-7p9x"
I20250626 13:59:43.073067 7111 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:59:43.124817 7111 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:59:43.126399 7111 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:59:43.126850 7111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:59:43.129828 7111 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 13:59:43.134083 7111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 13:59:43.134351 7111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 13:59:43.134624 7111 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 13:59:43.134780 7111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 13:59:43.290374 7111 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.65:40703
I20250626 13:59:43.290479 7239 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.65:40703 every 8 connection(s)
I20250626 13:59:43.292894 7111 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250626 13:59:43.298875 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 7111
I20250626 13:59:43.299387 2477 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250626 13:59:43.306145 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.107.66:0
--local_ip_for_outbound_sockets=127.2.107.66
--webserver_interface=127.2.107.66
--webserver_port=0
--tserver_master_addrs=127.2.107.126:45791
--builtin_ntp_servers=127.2.107.84:37923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250626 13:59:43.315375 7240 heartbeater.cc:344] Connected to a master server at 127.2.107.126:45791
I20250626 13:59:43.315841 7240 heartbeater.cc:461] Registering TS with master...
I20250626 13:59:43.316879 7240 heartbeater.cc:507] Master 127.2.107.126:45791 requested a full tablet report, sending...
I20250626 13:59:43.319593 7052 ts_manager.cc:194] Registered new tserver with Master: 65bdee5c7aac49308de217fdb95386ca (127.2.107.65:40703)
I20250626 13:59:43.321602 7052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.107.65:57493
W20250626 13:59:43.617117 7244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:59:43.617776 7244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:59:43.618345 7244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:59:43.653566 7244 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250626 13:59:43.653983 7244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:59:43.654996 7244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.107.66
I20250626 13:59:43.691032 7244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:37923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.107.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.107.66
--webserver_port=0
--tserver_master_addrs=127.2.107.126:45791
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.107.66
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:59:43.692306 7244 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:59:43.694056 7244 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:59:43.711442 7250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:59:44.325194 7240 heartbeater.cc:499] Master 127.2.107.126:45791 was elected leader, sending a full tablet report...
W20250626 13:59:43.711550 7253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:59:43.713582 7244 server_base.cc:1048] running on GCE node
W20250626 13:59:43.711694 7251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:59:44.803689 7244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:59:44.807859 7244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:59:44.809319 7244 hybrid_clock.cc:648] HybridClock initialized: now 1750946384809269 us; error 64 us; skew 500 ppm
I20250626 13:59:44.810154 7244 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:59:44.816627 7244 webserver.cc:469] Webserver started at http://127.2.107.66:35747/ using document root <none> and password file <none>
I20250626 13:59:44.817628 7244 fs_manager.cc:362] Metadata directory not provided
I20250626 13:59:44.817852 7244 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:59:44.818300 7244 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 13:59:44.822955 7244 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "261c132e9df948198ba27516e3e8b7e6"
format_stamp: "Formatted at 2025-06-26 13:59:44 on dist-test-slave-7p9x"
I20250626 13:59:44.824046 7244 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "261c132e9df948198ba27516e3e8b7e6"
format_stamp: "Formatted at 2025-06-26 13:59:44 on dist-test-slave-7p9x"
I20250626 13:59:44.832140 7244 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20250626 13:59:44.837929 7261 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:59:44.839093 7244 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250626 13:59:44.839438 7244 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "261c132e9df948198ba27516e3e8b7e6"
format_stamp: "Formatted at 2025-06-26 13:59:44 on dist-test-slave-7p9x"
I20250626 13:59:44.839771 7244 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:59:44.896136 7244 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:59:44.897668 7244 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:59:44.898114 7244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:59:44.900621 7244 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 13:59:44.904842 7244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 13:59:44.905177 7244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250626 13:59:44.905484 7244 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 13:59:44.905710 7244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 13:59:45.041343 7244 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.66:38741
I20250626 13:59:45.041478 7373 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.66:38741 every 8 connection(s)
I20250626 13:59:45.043865 7244 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250626 13:59:45.047674 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 7244
I20250626 13:59:45.048403 2477 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250626 13:59:45.057071 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.107.67:0
--local_ip_for_outbound_sockets=127.2.107.67
--webserver_interface=127.2.107.67
--webserver_port=0
--tserver_master_addrs=127.2.107.126:45791
--builtin_ntp_servers=127.2.107.84:37923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250626 13:59:45.065549 7374 heartbeater.cc:344] Connected to a master server at 127.2.107.126:45791
I20250626 13:59:45.065979 7374 heartbeater.cc:461] Registering TS with master...
I20250626 13:59:45.066985 7374 heartbeater.cc:507] Master 127.2.107.126:45791 requested a full tablet report, sending...
I20250626 13:59:45.069013 7052 ts_manager.cc:194] Registered new tserver with Master: 261c132e9df948198ba27516e3e8b7e6 (127.2.107.66:38741)
I20250626 13:59:45.070254 7052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.107.66:48529
W20250626 13:59:45.363948 7378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:59:45.364475 7378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:59:45.365005 7378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:59:45.396989 7378 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250626 13:59:45.397439 7378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:59:45.398241 7378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.107.67
I20250626 13:59:45.433725 7378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:37923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.107.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.107.67
--webserver_port=0
--tserver_master_addrs=127.2.107.126:45791
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.107.67
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:59:45.435094 7378 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:59:45.437574 7378 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:59:45.455235 7385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:59:46.073523 7374 heartbeater.cc:499] Master 127.2.107.126:45791 was elected leader, sending a full tablet report...
W20250626 13:59:45.455298 7384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:59:45.455721 7387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:59:46.601516 7386 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 13:59:46.601857 7378 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 13:59:46.607499 7378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:59:46.610237 7378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:59:46.611809 7378 hybrid_clock.cc:648] HybridClock initialized: now 1750946386611707 us; error 99 us; skew 500 ppm
I20250626 13:59:46.613018 7378 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:59:46.626953 7378 webserver.cc:469] Webserver started at http://127.2.107.67:35555/ using document root <none> and password file <none>
I20250626 13:59:46.627970 7378 fs_manager.cc:362] Metadata directory not provided
I20250626 13:59:46.628217 7378 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:59:46.628710 7378 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 13:59:46.633704 7378 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "7baf7aac9c3648b682928de716ccdbb4"
format_stamp: "Formatted at 2025-06-26 13:59:46 on dist-test-slave-7p9x"
I20250626 13:59:46.634976 7378 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "7baf7aac9c3648b682928de716ccdbb4"
format_stamp: "Formatted at 2025-06-26 13:59:46 on dist-test-slave-7p9x"
I20250626 13:59:46.642309 7378 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250626 13:59:46.650307 7394 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:59:46.651502 7378 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20250626 13:59:46.651854 7378 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "7baf7aac9c3648b682928de716ccdbb4"
format_stamp: "Formatted at 2025-06-26 13:59:46 on dist-test-slave-7p9x"
I20250626 13:59:46.652185 7378 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:59:46.738514 7378 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:59:46.740028 7378 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:59:46.740468 7378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:59:46.743297 7378 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 13:59:46.747407 7378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 13:59:46.747630 7378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 13:59:46.747895 7378 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 13:59:46.748039 7378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 13:59:46.881659 7378 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.67:33967
I20250626 13:59:46.881762 7506 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.67:33967 every 8 connection(s)
I20250626 13:59:46.884122 7378 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250626 13:59:46.889454 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 7378
I20250626 13:59:46.890069 2477 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250626 13:59:46.897260 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.107.68:0
--local_ip_for_outbound_sockets=127.2.107.68
--webserver_interface=127.2.107.68
--webserver_port=0
--tserver_master_addrs=127.2.107.126:45791
--builtin_ntp_servers=127.2.107.84:37923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250626 13:59:46.910460 7507 heartbeater.cc:344] Connected to a master server at 127.2.107.126:45791
I20250626 13:59:46.911011 7507 heartbeater.cc:461] Registering TS with master...
I20250626 13:59:46.912560 7507 heartbeater.cc:507] Master 127.2.107.126:45791 requested a full tablet report, sending...
I20250626 13:59:46.915359 7052 ts_manager.cc:194] Registered new tserver with Master: 7baf7aac9c3648b682928de716ccdbb4 (127.2.107.67:33967)
I20250626 13:59:46.916589 7052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.107.67:56771
W20250626 13:59:47.207576 7511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:59:47.208094 7511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:59:47.208659 7511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:59:47.241425 7511 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250626 13:59:47.241848 7511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:59:47.242694 7511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.107.68
I20250626 13:59:47.278609 7511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:37923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.107.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.2.107.68
--webserver_port=0
--tserver_master_addrs=127.2.107.126:45791
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.107.68
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:59:47.279894 7511 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:59:47.281567 7511 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:59:47.299373 7518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:59:47.919911 7507 heartbeater.cc:499] Master 127.2.107.126:45791 was elected leader, sending a full tablet report...
W20250626 13:59:47.300133 7517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:59:47.300335 7520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:59:47.302560 7511 server_base.cc:1048] running on GCE node
I20250626 13:59:48.423449 7511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:59:48.426026 7511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:59:48.427477 7511 hybrid_clock.cc:648] HybridClock initialized: now 1750946388427452 us; error 45 us; skew 500 ppm
I20250626 13:59:48.428306 7511 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:59:48.435056 7511 webserver.cc:469] Webserver started at http://127.2.107.68:34585/ using document root <none> and password file <none>
I20250626 13:59:48.435989 7511 fs_manager.cc:362] Metadata directory not provided
I20250626 13:59:48.436198 7511 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:59:48.436645 7511 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 13:59:48.441123 7511 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "a0b1e34f82ab42d69456ddf4d526d415"
format_stamp: "Formatted at 2025-06-26 13:59:48 on dist-test-slave-7p9x"
I20250626 13:59:48.442247 7511 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "a0b1e34f82ab42d69456ddf4d526d415"
format_stamp: "Formatted at 2025-06-26 13:59:48 on dist-test-slave-7p9x"
I20250626 13:59:48.449375 7511 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20250626 13:59:48.455510 7527 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:59:48.456749 7511 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250626 13:59:48.457053 7511 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "a0b1e34f82ab42d69456ddf4d526d415"
format_stamp: "Formatted at 2025-06-26 13:59:48 on dist-test-slave-7p9x"
I20250626 13:59:48.457428 7511 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:59:48.519841 7511 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:59:48.521461 7511 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:59:48.521875 7511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:59:48.524457 7511 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 13:59:48.529347 7511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 13:59:48.529562 7511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 13:59:48.529778 7511 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 13:59:48.529918 7511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 13:59:48.664916 7511 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.68:34469
I20250626 13:59:48.665006 7639 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.68:34469 every 8 connection(s)
I20250626 13:59:48.667369 7511 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250626 13:59:48.671957 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 7511
I20250626 13:59:48.672480 2477 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250626 13:59:48.678637 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.107.69:0
--local_ip_for_outbound_sockets=127.2.107.69
--webserver_interface=127.2.107.69
--webserver_port=0
--tserver_master_addrs=127.2.107.126:45791
--builtin_ntp_servers=127.2.107.84:37923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250626 13:59:48.688184 7640 heartbeater.cc:344] Connected to a master server at 127.2.107.126:45791
I20250626 13:59:48.688597 7640 heartbeater.cc:461] Registering TS with master...
I20250626 13:59:48.689678 7640 heartbeater.cc:507] Master 127.2.107.126:45791 requested a full tablet report, sending...
I20250626 13:59:48.691722 7052 ts_manager.cc:194] Registered new tserver with Master: a0b1e34f82ab42d69456ddf4d526d415 (127.2.107.68:34469)
I20250626 13:59:48.692945 7052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.107.68:51445
W20250626 13:59:48.992329 7644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:59:48.992852 7644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:59:48.993395 7644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:59:49.025458 7644 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250626 13:59:49.025899 7644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:59:49.026736 7644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.107.69
I20250626 13:59:49.063189 7644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:37923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.107.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.2.107.69
--webserver_port=0
--tserver_master_addrs=127.2.107.126:45791
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.107.69
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:59:49.064853 7644 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:59:49.066874 7644 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:59:49.083743 7650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:59:49.695952 7640 heartbeater.cc:499] Master 127.2.107.126:45791 was elected leader, sending a full tablet report...
W20250626 13:59:49.084103 7653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:59:49.086506 7644 server_base.cc:1048] running on GCE node
W20250626 13:59:49.084219 7651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:59:50.182300 7644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:59:50.184537 7644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:59:50.185935 7644 hybrid_clock.cc:648] HybridClock initialized: now 1750946390185890 us; error 52 us; skew 500 ppm
I20250626 13:59:50.186946 7644 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:59:50.193352 7644 webserver.cc:469] Webserver started at http://127.2.107.69:34145/ using document root <none> and password file <none>
I20250626 13:59:50.194339 7644 fs_manager.cc:362] Metadata directory not provided
I20250626 13:59:50.194566 7644 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:59:50.195184 7644 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 13:59:50.199846 7644 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "a26a33ad83dd448397ae3c7831ae1197"
format_stamp: "Formatted at 2025-06-26 13:59:50 on dist-test-slave-7p9x"
I20250626 13:59:50.200945 7644 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "a26a33ad83dd448397ae3c7831ae1197"
format_stamp: "Formatted at 2025-06-26 13:59:50 on dist-test-slave-7p9x"
I20250626 13:59:50.208461 7644 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20250626 13:59:50.214303 7661 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:59:50.215494 7644 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250626 13:59:50.215819 7644 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "a26a33ad83dd448397ae3c7831ae1197"
format_stamp: "Formatted at 2025-06-26 13:59:50 on dist-test-slave-7p9x"
I20250626 13:59:50.216152 7644 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:59:50.265455 7644 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:59:50.266924 7644 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:59:50.267369 7644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:59:50.269971 7644 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 13:59:50.274709 7644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 13:59:50.274940 7644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 13:59:50.275202 7644 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 13:59:50.275360 7644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 13:59:50.438077 7644 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.69:40081
I20250626 13:59:50.438194 7773 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.69:40081 every 8 connection(s)
I20250626 13:59:50.440606 7644 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250626 13:59:50.447968 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 7644
I20250626 13:59:50.448402 2477 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250626 13:59:50.464692 7774 heartbeater.cc:344] Connected to a master server at 127.2.107.126:45791
I20250626 13:59:50.465157 7774 heartbeater.cc:461] Registering TS with master...
I20250626 13:59:50.466207 7774 heartbeater.cc:507] Master 127.2.107.126:45791 requested a full tablet report, sending...
I20250626 13:59:50.469029 7052 ts_manager.cc:194] Registered new tserver with Master: a26a33ad83dd448397ae3c7831ae1197 (127.2.107.69:40081)
I20250626 13:59:50.471395 7052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.107.69:36771
I20250626 13:59:50.472065 2477 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250626 13:59:50.513628 7052 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54154:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250626 13:59:50.591485 7575 tablet_service.cc:1468] Processing CreateTablet for tablet 3c6eeb55f086409aa94c91812c07dbbd (DEFAULT_TABLE table=TestTable [id=cb9f1e74eacf4daa8a1e6bc7c3babac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 13:59:50.594507 7575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c6eeb55f086409aa94c91812c07dbbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:59:50.594902 7442 tablet_service.cc:1468] Processing CreateTablet for tablet 3c6eeb55f086409aa94c91812c07dbbd (DEFAULT_TABLE table=TestTable [id=cb9f1e74eacf4daa8a1e6bc7c3babac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 13:59:50.597239 7442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c6eeb55f086409aa94c91812c07dbbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:59:50.599521 7175 tablet_service.cc:1468] Processing CreateTablet for tablet 3c6eeb55f086409aa94c91812c07dbbd (DEFAULT_TABLE table=TestTable [id=cb9f1e74eacf4daa8a1e6bc7c3babac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 13:59:50.602008 7175 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c6eeb55f086409aa94c91812c07dbbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:59:50.624696 7793 tablet_bootstrap.cc:492] T 3c6eeb55f086409aa94c91812c07dbbd P a0b1e34f82ab42d69456ddf4d526d415: Bootstrap starting.
I20250626 13:59:50.630004 7794 tablet_bootstrap.cc:492] T 3c6eeb55f086409aa94c91812c07dbbd P 65bdee5c7aac49308de217fdb95386ca: Bootstrap starting.
I20250626 13:59:50.635159 7793 tablet_bootstrap.cc:654] T 3c6eeb55f086409aa94c91812c07dbbd P a0b1e34f82ab42d69456ddf4d526d415: Neither blocks nor log segments found. Creating new log.
I20250626 13:59:50.637918 7794 tablet_bootstrap.cc:654] T 3c6eeb55f086409aa94c91812c07dbbd P 65bdee5c7aac49308de217fdb95386ca: Neither blocks nor log segments found. Creating new log.
I20250626 13:59:50.637991 7793 log.cc:826] T 3c6eeb55f086409aa94c91812c07dbbd P a0b1e34f82ab42d69456ddf4d526d415: Log is configured to *not* fsync() on all Append() calls
I20250626 13:59:50.640522 7794 log.cc:826] T 3c6eeb55f086409aa94c91812c07dbbd P 65bdee5c7aac49308de217fdb95386ca: Log is configured to *not* fsync() on all Append() calls
I20250626 13:59:50.642246 7795 tablet_bootstrap.cc:492] T 3c6eeb55f086409aa94c91812c07dbbd P 7baf7aac9c3648b682928de716ccdbb4: Bootstrap starting.
I20250626 13:59:50.645979 7793 tablet_bootstrap.cc:492] T 3c6eeb55f086409aa94c91812c07dbbd P a0b1e34f82ab42d69456ddf4d526d415: No bootstrap required, opened a new log
I20250626 13:59:50.646602 7793 ts_tablet_manager.cc:1397] T 3c6eeb55f086409aa94c91812c07dbbd P a0b1e34f82ab42d69456ddf4d526d415: Time spent bootstrapping tablet: real 0.023s user 0.014s sys 0.005s
I20250626 13:59:50.648198 7794 tablet_bootstrap.cc:492] T 3c6eeb55f086409aa94c91812c07dbbd P 65bdee5c7aac49308de217fdb95386ca: No bootstrap required, opened a new log
I20250626 13:59:50.648752 7794 ts_tablet_manager.cc:1397] T 3c6eeb55f086409aa94c91812c07dbbd P 65bdee5c7aac49308de217fdb95386ca: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.004s
I20250626 13:59:50.650693 7795 tablet_bootstrap.cc:654] T 3c6eeb55f086409aa94c91812c07dbbd P 7baf7aac9c3648b682928de716ccdbb4: Neither blocks nor log segments found. Creating new log.
I20250626 13:59:50.652995 7795 log.cc:826] T 3c6eeb55f086409aa94c91812c07dbbd P 7baf7aac9c3648b682928de716ccdbb4: Log is configured to *not* fsync() on all Append() calls
I20250626 13:59:50.659118 7795 tablet_bootstrap.cc:492] T 3c6eeb55f086409aa94c91812c07dbbd P 7baf7aac9c3648b682928de716ccdbb4: No bootstrap required, opened a new log
I20250626 13:59:50.660200 7795 ts_tablet_manager.cc:1397] T 3c6eeb55f086409aa94c91812c07dbbd P 7baf7aac9c3648b682928de716ccdbb4: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.004s
I20250626 13:59:50.671351 7794 raft_consensus.cc:357] T 3c6eeb55f086409aa94c91812c07dbbd P 65bdee5c7aac49308de217fdb95386ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b1e34f82ab42d69456ddf4d526d415" member_type: VOTER last_known_addr { host: "127.2.107.68" port: 34469 } } peers { permanent_uuid: "7baf7aac9c3648b682928de716ccdbb4" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 33967 } } peers { permanent_uuid: "65bdee5c7aac49308de217fdb95386ca" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 40703 } }
I20250626 13:59:50.672420 7794 raft_consensus.cc:383] T 3c6eeb55f086409aa94c91812c07dbbd P 65bdee5c7aac49308de217fdb95386ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 13:59:50.672871 7794 raft_consensus.cc:738] T 3c6eeb55f086409aa94c91812c07dbbd P 65bdee5c7aac49308de217fdb95386ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65bdee5c7aac49308de217fdb95386ca, State: Initialized, Role: FOLLOWER
I20250626 13:59:50.674544 7793 raft_consensus.cc:357] T 3c6eeb55f086409aa94c91812c07dbbd P a0b1e34f82ab42d69456ddf4d526d415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b1e34f82ab42d69456ddf4d526d415" member_type: VOTER last_known_addr { host: "127.2.107.68" port: 34469 } } peers { permanent_uuid: "7baf7aac9c3648b682928de716ccdbb4" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 33967 } } peers { permanent_uuid: "65bdee5c7aac49308de217fdb95386ca" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 40703 } }
I20250626 13:59:50.675957 7793 raft_consensus.cc:383] T 3c6eeb55f086409aa94c91812c07dbbd P a0b1e34f82ab42d69456ddf4d526d415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 13:59:50.674975 7794 consensus_queue.cc:260] T 3c6eeb55f086409aa94c91812c07dbbd P 65bdee5c7aac49308de217fdb95386ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b1e34f82ab42d69456ddf4d526d415" member_type: VOTER last_known_addr { host: "127.2.107.68" port: 34469 } } peers { permanent_uuid: "7baf7aac9c3648b682928de716ccdbb4" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 33967 } } peers { permanent_uuid: "65bdee5c7aac49308de217fdb95386ca" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 40703 } }
I20250626 13:59:50.676272 7793 raft_consensus.cc:738] T 3c6eeb55f086409aa94c91812c07dbbd P a0b1e34f82ab42d69456ddf4d526d415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0b1e34f82ab42d69456ddf4d526d415, State: Initialized, Role: FOLLOWER
I20250626 13:59:50.677115 7793 consensus_queue.cc:260] T 3c6eeb55f086409aa94c91812c07dbbd P a0b1e34f82ab42d69456ddf4d526d415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b1e34f82ab42d69456ddf4d526d415" member_type: VOTER last_known_addr { host: "127.2.107.68" port: 34469 } } peers { permanent_uuid: "7baf7aac9c3648b682928de716ccdbb4" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 33967 } } peers { permanent_uuid: "65bdee5c7aac49308de217fdb95386ca" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 40703 } }
I20250626 13:59:50.686601 7794 ts_tablet_manager.cc:1428] T 3c6eeb55f086409aa94c91812c07dbbd P 65bdee5c7aac49308de217fdb95386ca: Time spent starting tablet: real 0.038s user 0.027s sys 0.007s
I20250626 13:59:50.688907 7795 raft_consensus.cc:357] T 3c6eeb55f086409aa94c91812c07dbbd P 7baf7aac9c3648b682928de716ccdbb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b1e34f82ab42d69456ddf4d526d415" member_type: VOTER last_known_addr { host: "127.2.107.68" port: 34469 } } peers { permanent_uuid: "7baf7aac9c3648b682928de716ccdbb4" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 33967 } } peers { permanent_uuid: "65bdee5c7aac49308de217fdb95386ca" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 40703 } }
I20250626 13:59:50.689956 7795 raft_consensus.cc:383] T 3c6eeb55f086409aa94c91812c07dbbd P 7baf7aac9c3648b682928de716ccdbb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 13:59:50.690948 7795 raft_consensus.cc:738] T 3c6eeb55f086409aa94c91812c07dbbd P 7baf7aac9c3648b682928de716ccdbb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7baf7aac9c3648b682928de716ccdbb4, State: Initialized, Role: FOLLOWER
I20250626 13:59:50.692538 7793 ts_tablet_manager.cc:1428] T 3c6eeb55f086409aa94c91812c07dbbd P a0b1e34f82ab42d69456ddf4d526d415: Time spent starting tablet: real 0.046s user 0.038s sys 0.007s
I20250626 13:59:50.692066 7795 consensus_queue.cc:260] T 3c6eeb55f086409aa94c91812c07dbbd P 7baf7aac9c3648b682928de716ccdbb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b1e34f82ab42d69456ddf4d526d415" member_type: VOTER last_known_addr { host: "127.2.107.68" port: 34469 } } peers { permanent_uuid: "7baf7aac9c3648b682928de716ccdbb4" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 33967 } } peers { permanent_uuid: "65bdee5c7aac49308de217fdb95386ca" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 40703 } }
I20250626 13:59:50.699990 7795 ts_tablet_manager.cc:1428] T 3c6eeb55f086409aa94c91812c07dbbd P 7baf7aac9c3648b682928de716ccdbb4: Time spent starting tablet: real 0.039s user 0.034s sys 0.000s
W20250626 13:59:50.805706 7241 tablet.cc:2378] T 3c6eeb55f086409aa94c91812c07dbbd P 65bdee5c7aac49308de217fdb95386ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 13:59:50.863288 7799 raft_consensus.cc:491] T 3c6eeb55f086409aa94c91812c07dbbd P 65bdee5c7aac49308de217fdb95386ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 13:59:50.863806 7799 raft_consensus.cc:513] T 3c6eeb55f086409aa94c91812c07dbbd P 65bdee5c7aac49308de217fdb95386ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b1e34f82ab42d69456ddf4d526d415" member_type: VOTER last_known_addr { host: "127.2.107.68" port: 34469 } } peers { permanent_uuid: "7baf7aac9c3648b682928de716ccdbb4" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 33967 } } peers { permanent_uuid: "65bdee5c7aac49308de217fdb95386ca" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 40703 } }
I20250626 13:59:50.866443 7799 leader_election.cc:290] T 3c6eeb55f086409aa94c91812c07dbbd P 65bdee5c7aac49308de217fdb95386ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0b1e34f82ab42d69456ddf4d526d415 (127.2.107.68:34469), 7baf7aac9c3648b682928de716ccdbb4 (127.2.107.67:33967)
I20250626 13:59:50.873831 7800 raft_consensus.cc:491] T 3c6eeb55f086409aa94c91812c07dbbd P a0b1e34f82ab42d69456ddf4d526d415 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 13:59:50.874892 7800 raft_consensus.cc:513] T 3c6eeb55f086409aa94c91812c07dbbd P a0b1e34f82ab42d69456ddf4d526d415 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b1e34f82ab42d69456ddf4d526d415" member_type: VOTER last_known_addr { host: "127.2.107.68" port: 34469 } } peers { permanent_uuid: "7baf7aac9c3648b682928de716ccdbb4" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 33967 } } peers { permanent_uuid: "65bdee5c7aac49308de217fdb95386ca" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 40703 } }
I20250626 13:59:50.878157 7800 leader_election.cc:290] T 3c6eeb55f086409aa94c91812c07dbbd P a0b1e34f82ab42d69456ddf4d526d415 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7baf7aac9c3648b682928de716ccdbb4 (127.2.107.67:33967), 65bdee5c7aac49308de217fdb95386ca (127.2.107.65:40703)
I20250626 13:59:50.881479 7462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c6eeb55f086409aa94c91812c07dbbd" candidate_uuid: "65bdee5c7aac49308de217fdb95386ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7baf7aac9c3648b682928de716ccdbb4" is_pre_election: true
I20250626 13:59:50.882377 7462 raft_consensus.cc:2466] T 3c6eeb55f086409aa94c91812c07dbbd P 7baf7aac9c3648b682928de716ccdbb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65bdee5c7aac49308de217fdb95386ca in term 0.
I20250626 13:59:50.883975 7130 leader_election.cc:304] T 3c6eeb55f086409aa94c91812c07dbbd P 65bdee5c7aac49308de217fdb95386ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65bdee5c7aac49308de217fdb95386ca, 7baf7aac9c3648b682928de716ccdbb4; no voters:
I20250626 13:59:50.884953 7799 raft_consensus.cc:2802] T 3c6eeb55f086409aa94c91812c07dbbd P 65bdee5c7aac49308de217fdb95386ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 13:59:50.885399 7799 raft_consensus.cc:491] T 3c6eeb55f086409aa94c91812c07dbbd P 65bdee5c7aac49308de217fdb95386ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 13:59:50.885708 7799 raft_consensus.cc:3058] T 3c6eeb55f086409aa94c91812c07dbbd P 65bdee5c7aac49308de217fdb95386ca [term 0 FOLLOWER]: Advancing to term 1
W20250626 13:59:50.893112 7508 tablet.cc:2378] T 3c6eeb55f086409aa94c91812c07dbbd P 7baf7aac9c3648b682928de716ccdbb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 13:59:50.893343 7799 raft_consensus.cc:513] T 3c6eeb55f086409aa94c91812c07dbbd P 65bdee5c7aac49308de217fdb95386ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b1e34f82ab42d69456ddf4d526d415" member_type: VOTER last_known_addr { host: "127.2.107.68" port: 34469 } } peers { permanent_uuid: "7baf7aac9c3648b682928de716ccdbb4" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 33967 } } peers { permanent_uuid: "65bdee5c7aac49308de217fdb95386ca" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 40703 } }
I20250626 13:59:50.900736 7462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c6eeb55f086409aa94c91812c07dbbd" candidate_uuid: "65bdee5c7aac49308de217fdb95386ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7baf7aac9c3648b682928de716ccdbb4"
I20250626 13:59:50.901341 7462 raft_consensus.cc:3058] T 3c6eeb55f086409aa94c91812c07dbbd P 7baf7aac9c3648b682928de716ccdbb4 [term 0 FOLLOWER]: Advancing to term 1
I20250626 13:59:50.905514 7799 leader_election.cc:290] T 3c6eeb55f086409aa94c91812c07dbbd P 65bdee5c7aac49308de217fdb95386ca [CANDIDATE]: Term 1 election: Requested vote from peers a0b1e34f82ab42d69456ddf4d526d415 (127.2.107.68:34469), 7baf7aac9c3648b682928de716ccdbb4 (127.2.107.67:33967)
I20250626 13:59:50.908946 7595 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c6eeb55f086409aa94c91812c07dbbd" candidate_uuid: "65bdee5c7aac49308de217fdb95386ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0b1e34f82ab42d69456ddf4d526d415" is_pre_election: true
I20250626 13:59:50.909832 7595 raft_consensus.cc:2466] T 3c6eeb55f086409aa94c91812c07dbbd P a0b1e34f82ab42d69456ddf4d526d415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65bdee5c7aac49308de217fdb95386ca in term 0.
I20250626 13:59:50.911254 7594 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c6eeb55f086409aa94c91812c07dbbd" candidate_uuid: "65bdee5c7aac49308de217fdb95386ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0b1e34f82ab42d69456ddf4d526d415"
I20250626 13:59:50.911873 7594 raft_consensus.cc:3058] T 3c6eeb55f086409aa94c91812c07dbbd P a0b1e34f82ab42d69456ddf4d526d415 [term 0 FOLLOWER]: Advancing to term 1
I20250626 13:59:50.913026 7461 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c6eeb55f086409aa94c91812c07dbbd" candidate_uuid: "a0b1e34f82ab42d69456ddf4d526d415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7baf7aac9c3648b682928de716ccdbb4" is_pre_election: true
I20250626 13:59:50.913733 7195 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c6eeb55f086409aa94c91812c07dbbd" candidate_uuid: "a0b1e34f82ab42d69456ddf4d526d415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65bdee5c7aac49308de217fdb95386ca" is_pre_election: true
I20250626 13:59:50.914712 7195 raft_consensus.cc:2391] T 3c6eeb55f086409aa94c91812c07dbbd P 65bdee5c7aac49308de217fdb95386ca [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a0b1e34f82ab42d69456ddf4d526d415 in current term 1: Already voted for candidate 65bdee5c7aac49308de217fdb95386ca in this term.
I20250626 13:59:50.916016 7530 leader_election.cc:304] T 3c6eeb55f086409aa94c91812c07dbbd P a0b1e34f82ab42d69456ddf4d526d415 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0b1e34f82ab42d69456ddf4d526d415; no voters: 65bdee5c7aac49308de217fdb95386ca, 7baf7aac9c3648b682928de716ccdbb4
I20250626 13:59:50.916446 7462 raft_consensus.cc:2466] T 3c6eeb55f086409aa94c91812c07dbbd P 7baf7aac9c3648b682928de716ccdbb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65bdee5c7aac49308de217fdb95386ca in term 1.
I20250626 13:59:50.917454 7130 leader_election.cc:304] T 3c6eeb55f086409aa94c91812c07dbbd P 65bdee5c7aac49308de217fdb95386ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65bdee5c7aac49308de217fdb95386ca, 7baf7aac9c3648b682928de716ccdbb4; no voters:
I20250626 13:59:50.918182 7799 raft_consensus.cc:2802] T 3c6eeb55f086409aa94c91812c07dbbd P 65bdee5c7aac49308de217fdb95386ca [term 1 FOLLOWER]: Leader election won for term 1
I20250626 13:59:50.918565 7594 raft_consensus.cc:2466] T 3c6eeb55f086409aa94c91812c07dbbd P a0b1e34f82ab42d69456ddf4d526d415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65bdee5c7aac49308de217fdb95386ca in term 1.
I20250626 13:59:50.918897 7800 raft_consensus.cc:2747] T 3c6eeb55f086409aa94c91812c07dbbd P a0b1e34f82ab42d69456ddf4d526d415 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 13:59:50.920542 7799 raft_consensus.cc:695] T 3c6eeb55f086409aa94c91812c07dbbd P 65bdee5c7aac49308de217fdb95386ca [term 1 LEADER]: Becoming Leader. State: Replica: 65bdee5c7aac49308de217fdb95386ca, State: Running, Role: LEADER
I20250626 13:59:50.921481 7799 consensus_queue.cc:237] T 3c6eeb55f086409aa94c91812c07dbbd P 65bdee5c7aac49308de217fdb95386ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b1e34f82ab42d69456ddf4d526d415" member_type: VOTER last_known_addr { host: "127.2.107.68" port: 34469 } } peers { permanent_uuid: "7baf7aac9c3648b682928de716ccdbb4" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 33967 } } peers { permanent_uuid: "65bdee5c7aac49308de217fdb95386ca" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 40703 } }
W20250626 13:59:50.926220 7641 tablet.cc:2378] T 3c6eeb55f086409aa94c91812c07dbbd P a0b1e34f82ab42d69456ddf4d526d415: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 13:59:50.935318 7052 catalog_manager.cc:5582] T 3c6eeb55f086409aa94c91812c07dbbd P 65bdee5c7aac49308de217fdb95386ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 65bdee5c7aac49308de217fdb95386ca (127.2.107.65). New cstate: current_term: 1 leader_uuid: "65bdee5c7aac49308de217fdb95386ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b1e34f82ab42d69456ddf4d526d415" member_type: VOTER last_known_addr { host: "127.2.107.68" port: 34469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7baf7aac9c3648b682928de716ccdbb4" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 33967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65bdee5c7aac49308de217fdb95386ca" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 40703 } health_report { overall_health: HEALTHY } } }
I20250626 13:59:50.979607 2477 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250626 13:59:50.983600 2477 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 65bdee5c7aac49308de217fdb95386ca to finish bootstrapping
I20250626 13:59:50.997382 2477 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 7baf7aac9c3648b682928de716ccdbb4 to finish bootstrapping
I20250626 13:59:51.009557 2477 ts_itest-base.cc:246] Waiting for 1 tablets on tserver a0b1e34f82ab42d69456ddf4d526d415 to finish bootstrapping
I20250626 13:59:51.024621 2477 test_util.cc:276] Using random seed: -106404910
I20250626 13:59:51.056501 2477 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20250626 13:59:51.057520 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 7511
I20250626 13:59:51.090955 7462 raft_consensus.cc:1273] T 3c6eeb55f086409aa94c91812c07dbbd P 7baf7aac9c3648b682928de716ccdbb4 [term 1 FOLLOWER]: Refusing update from remote peer 65bdee5c7aac49308de217fdb95386ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250626 13:59:51.092056 7129 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.107.68:34469: connect: Connection refused (error 111)
I20250626 13:59:51.092447 7799 consensus_queue.cc:1035] T 3c6eeb55f086409aa94c91812c07dbbd P 65bdee5c7aac49308de217fdb95386ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "7baf7aac9c3648b682928de716ccdbb4" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 33967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 13:59:51.097007 7129 consensus_peers.cc:487] T 3c6eeb55f086409aa94c91812c07dbbd P 65bdee5c7aac49308de217fdb95386ca -> Peer a0b1e34f82ab42d69456ddf4d526d415 (127.2.107.68:34469): Couldn't send request to peer a0b1e34f82ab42d69456ddf4d526d415. Status: Network error: Client connection negotiation failed: client connection to 127.2.107.68:34469: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 13:59:51.120008 7820 mvcc.cc:204] Tried to move back new op lower bound from 7171876417893740544 to 7171876417225981952. Current Snapshot: MvccSnapshot[applied={T|T < 7171876417893740544}]
I20250626 13:59:51.475751 7774 heartbeater.cc:499] Master 127.2.107.126:45791 was elected leader, sending a full tablet report...
W20250626 13:59:53.630277 7129 consensus_peers.cc:487] T 3c6eeb55f086409aa94c91812c07dbbd P 65bdee5c7aac49308de217fdb95386ca -> Peer a0b1e34f82ab42d69456ddf4d526d415 (127.2.107.68:34469): Couldn't send request to peer a0b1e34f82ab42d69456ddf4d526d415. Status: Network error: Client connection negotiation failed: client connection to 127.2.107.68:34469: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250626 13:59:53.730480 7309 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 13:59:53.738258 7175 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250626 13:59:53.770701 7709 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 13:59:53.773430 7442 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 13:59:55.636533 7309 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 13:59:55.639336 7709 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 13:59:55.656769 7175 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250626 13:59:55.672466 7442 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250626 13:59:56.374706 7129 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.107.68:34469: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250626 13:59:56.378808 7129 consensus_peers.cc:487] T 3c6eeb55f086409aa94c91812c07dbbd P 65bdee5c7aac49308de217fdb95386ca -> Peer a0b1e34f82ab42d69456ddf4d526d415 (127.2.107.68:34469): Couldn't send request to peer a0b1e34f82ab42d69456ddf4d526d415. Status: Network error: Client connection negotiation failed: client connection to 127.2.107.68:34469: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250626 13:59:58.107841 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 7111
I20250626 13:59:58.143144 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 7244
I20250626 13:59:58.164453 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 7378
I20250626 13:59:58.192891 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 7644
I20250626 13:59:58.216351 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 7019
2025-06-26T13:59:58Z chronyd exiting
[ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4 (18643 ms)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest (18644 ms total)
[----------] 1 test from ListTableCliSimpleParamTest
[ RUN ] ListTableCliSimpleParamTest.TestListTables/2
I20250626 13:59:58.278811 2477 test_util.cc:276] Using random seed: -99150749
I20250626 13:59:58.284693 2477 ts_itest-base.cc:115] Starting cluster with:
I20250626 13:59:58.284886 2477 ts_itest-base.cc:116] --------------
I20250626 13:59:58.285046 2477 ts_itest-base.cc:117] 1 tablet servers
I20250626 13:59:58.285387 2477 ts_itest-base.cc:118] 1 replicas per TS
I20250626 13:59:58.285703 2477 ts_itest-base.cc:119] --------------
2025-06-26T13:59:58Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-26T13:59:58Z Disabled control of system clock
I20250626 13:59:58.334115 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.107.126:37663
--webserver_interface=127.2.107.126
--webserver_port=0
--builtin_ntp_servers=127.2.107.84:34253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.107.126:37663 with env {}
W20250626 13:59:58.657471 7925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:59:58.658038 7925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:59:58.658567 7925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:59:58.690855 7925 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 13:59:58.691171 7925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:59:58.691385 7925 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 13:59:58.691569 7925 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 13:59:58.730090 7925 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:34253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.107.126:37663
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.107.126:37663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.107.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:59:58.731454 7925 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:59:58.733326 7925 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:59:58.749980 7931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:59:58.750380 7932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:59:58.750708 7925 server_base.cc:1048] running on GCE node
W20250626 13:59:58.751165 7934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:59:59.886441 7925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:59:59.889505 7925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:59:59.891021 7925 hybrid_clock.cc:648] HybridClock initialized: now 1750946399890966 us; error 73 us; skew 500 ppm
I20250626 13:59:59.891937 7925 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:59:59.903576 7925 webserver.cc:469] Webserver started at http://127.2.107.126:45523/ using document root <none> and password file <none>
I20250626 13:59:59.904631 7925 fs_manager.cc:362] Metadata directory not provided
I20250626 13:59:59.904879 7925 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:59:59.905383 7925 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 13:59:59.910073 7925 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "ff18e45cdd8149c1941109eca2390346"
format_stamp: "Formatted at 2025-06-26 13:59:59 on dist-test-slave-7p9x"
I20250626 13:59:59.911388 7925 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "ff18e45cdd8149c1941109eca2390346"
format_stamp: "Formatted at 2025-06-26 13:59:59 on dist-test-slave-7p9x"
I20250626 13:59:59.919148 7925 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20250626 13:59:59.925151 7941 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:59:59.926401 7925 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250626 13:59:59.926735 7925 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ff18e45cdd8149c1941109eca2390346"
format_stamp: "Formatted at 2025-06-26 13:59:59 on dist-test-slave-7p9x"
I20250626 13:59:59.927090 7925 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:59:59.991375 7925 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:59:59.992863 7925 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:59:59.993391 7925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 14:00:00.062882 7925 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.126:37663
I20250626 14:00:00.062965 7992 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.126:37663 every 8 connection(s)
I20250626 14:00:00.065600 7925 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250626 14:00:00.070645 7993 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:00:00.075520 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 7925
I20250626 14:00:00.075932 2477 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750946263504804-2477-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250626 14:00:00.094226 7993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff18e45cdd8149c1941109eca2390346: Bootstrap starting.
I20250626 14:00:00.101024 7993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff18e45cdd8149c1941109eca2390346: Neither blocks nor log segments found. Creating new log.
I20250626 14:00:00.102824 7993 log.cc:826] T 00000000000000000000000000000000 P ff18e45cdd8149c1941109eca2390346: Log is configured to *not* fsync() on all Append() calls
I20250626 14:00:00.107707 7993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff18e45cdd8149c1941109eca2390346: No bootstrap required, opened a new log
I20250626 14:00:00.126005 7993 raft_consensus.cc:357] T 00000000000000000000000000000000 P ff18e45cdd8149c1941109eca2390346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff18e45cdd8149c1941109eca2390346" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 37663 } }
I20250626 14:00:00.126813 7993 raft_consensus.cc:383] T 00000000000000000000000000000000 P ff18e45cdd8149c1941109eca2390346 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:00:00.127043 7993 raft_consensus.cc:738] T 00000000000000000000000000000000 P ff18e45cdd8149c1941109eca2390346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff18e45cdd8149c1941109eca2390346, State: Initialized, Role: FOLLOWER
I20250626 14:00:00.127725 7993 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff18e45cdd8149c1941109eca2390346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff18e45cdd8149c1941109eca2390346" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 37663 } }
I20250626 14:00:00.128214 7993 raft_consensus.cc:397] T 00000000000000000000000000000000 P ff18e45cdd8149c1941109eca2390346 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 14:00:00.128490 7993 raft_consensus.cc:491] T 00000000000000000000000000000000 P ff18e45cdd8149c1941109eca2390346 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 14:00:00.128867 7993 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ff18e45cdd8149c1941109eca2390346 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:00:00.133405 7993 raft_consensus.cc:513] T 00000000000000000000000000000000 P ff18e45cdd8149c1941109eca2390346 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff18e45cdd8149c1941109eca2390346" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 37663 } }
I20250626 14:00:00.134152 7993 leader_election.cc:304] T 00000000000000000000000000000000 P ff18e45cdd8149c1941109eca2390346 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff18e45cdd8149c1941109eca2390346; no voters:
I20250626 14:00:00.135854 7993 leader_election.cc:290] T 00000000000000000000000000000000 P ff18e45cdd8149c1941109eca2390346 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 14:00:00.136816 7998 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ff18e45cdd8149c1941109eca2390346 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:00:00.138891 7998 raft_consensus.cc:695] T 00000000000000000000000000000000 P ff18e45cdd8149c1941109eca2390346 [term 1 LEADER]: Becoming Leader. State: Replica: ff18e45cdd8149c1941109eca2390346, State: Running, Role: LEADER
I20250626 14:00:00.139765 7993 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff18e45cdd8149c1941109eca2390346 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 14:00:00.139611 7998 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff18e45cdd8149c1941109eca2390346 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff18e45cdd8149c1941109eca2390346" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 37663 } }
I20250626 14:00:00.150432 7999 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff18e45cdd8149c1941109eca2390346 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff18e45cdd8149c1941109eca2390346" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff18e45cdd8149c1941109eca2390346" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 37663 } } }
I20250626 14:00:00.151387 7999 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff18e45cdd8149c1941109eca2390346 [sys.catalog]: This master's current role is: LEADER
I20250626 14:00:00.152884 8000 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff18e45cdd8149c1941109eca2390346 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff18e45cdd8149c1941109eca2390346. Latest consensus state: current_term: 1 leader_uuid: "ff18e45cdd8149c1941109eca2390346" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff18e45cdd8149c1941109eca2390346" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 37663 } } }
I20250626 14:00:00.153522 8000 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff18e45cdd8149c1941109eca2390346 [sys.catalog]: This master's current role is: LEADER
I20250626 14:00:00.156539 8008 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 14:00:00.169528 8008 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 14:00:00.186595 8008 catalog_manager.cc:1349] Generated new cluster ID: 81802330a5534fdfb98f74faac97d83e
I20250626 14:00:00.187140 8008 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 14:00:00.214895 8008 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 14:00:00.216933 8008 catalog_manager.cc:1506] Loading token signing keys...
I20250626 14:00:00.245630 8008 catalog_manager.cc:5955] T 00000000000000000000000000000000 P ff18e45cdd8149c1941109eca2390346: Generated new TSK 0
I20250626 14:00:00.246606 8008 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 14:00:00.264673 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.107.65:0
--local_ip_for_outbound_sockets=127.2.107.65
--webserver_interface=127.2.107.65
--webserver_port=0
--tserver_master_addrs=127.2.107.126:37663
--builtin_ntp_servers=127.2.107.84:34253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250626 14:00:00.573576 8017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 14:00:00.574134 8017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 14:00:00.574630 8017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 14:00:00.607991 8017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 14:00:00.608867 8017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.107.65
I20250626 14:00:00.644590 8017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:34253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.107.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.107.65
--webserver_port=0
--tserver_master_addrs=127.2.107.126:37663
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.107.65
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 14:00:00.645959 8017 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 14:00:00.647593 8017 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 14:00:00.665405 8024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 14:00:00.667611 8017 server_base.cc:1048] running on GCE node
W20250626 14:00:00.666900 8023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 14:00:00.666456 8026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 14:00:01.810420 8017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 14:00:01.813045 8017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 14:00:01.814504 8017 hybrid_clock.cc:648] HybridClock initialized: now 1750946401814447 us; error 75 us; skew 500 ppm
I20250626 14:00:01.815356 8017 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 14:00:01.822326 8017 webserver.cc:469] Webserver started at http://127.2.107.65:45187/ using document root <none> and password file <none>
I20250626 14:00:01.823280 8017 fs_manager.cc:362] Metadata directory not provided
I20250626 14:00:01.823499 8017 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 14:00:01.824003 8017 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 14:00:01.828529 8017 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "fb4ed636ffea4f41bc55005426dda200"
format_stamp: "Formatted at 2025-06-26 14:00:01 on dist-test-slave-7p9x"
I20250626 14:00:01.829694 8017 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "fb4ed636ffea4f41bc55005426dda200"
format_stamp: "Formatted at 2025-06-26 14:00:01 on dist-test-slave-7p9x"
I20250626 14:00:01.837116 8017 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20250626 14:00:01.843232 8033 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 14:00:01.844362 8017 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20250626 14:00:01.844681 8017 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "fb4ed636ffea4f41bc55005426dda200"
format_stamp: "Formatted at 2025-06-26 14:00:01 on dist-test-slave-7p9x"
I20250626 14:00:01.845021 8017 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 14:00:01.892493 8017 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 14:00:01.894073 8017 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 14:00:01.894510 8017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 14:00:01.897516 8017 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 14:00:01.902112 8017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 14:00:01.902329 8017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 14:00:01.902823 8017 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 14:00:01.903021 8017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 14:00:02.062887 8017 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.65:41749
I20250626 14:00:02.063052 8145 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.65:41749 every 8 connection(s)
I20250626 14:00:02.065689 8017 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250626 14:00:02.074950 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 8017
I20250626 14:00:02.075482 2477 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750946263504804-2477-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250626 14:00:02.090359 8146 heartbeater.cc:344] Connected to a master server at 127.2.107.126:37663
I20250626 14:00:02.090813 8146 heartbeater.cc:461] Registering TS with master...
I20250626 14:00:02.091854 8146 heartbeater.cc:507] Master 127.2.107.126:37663 requested a full tablet report, sending...
I20250626 14:00:02.095042 7957 ts_manager.cc:194] Registered new tserver with Master: fb4ed636ffea4f41bc55005426dda200 (127.2.107.65:41749)
I20250626 14:00:02.097200 2477 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250626 14:00:02.098188 7957 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.107.65:37389
I20250626 14:00:02.131119 7957 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43872:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250626 14:00:02.190223 8081 tablet_service.cc:1468] Processing CreateTablet for tablet ba69a20cb49b43f5afd9db92090485a8 (DEFAULT_TABLE table=TestTable [id=d07bd7ac14e743ca8d9d5b65a2aa0389]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:00:02.191660 8081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba69a20cb49b43f5afd9db92090485a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:00:02.211350 8161 tablet_bootstrap.cc:492] T ba69a20cb49b43f5afd9db92090485a8 P fb4ed636ffea4f41bc55005426dda200: Bootstrap starting.
I20250626 14:00:02.216627 8161 tablet_bootstrap.cc:654] T ba69a20cb49b43f5afd9db92090485a8 P fb4ed636ffea4f41bc55005426dda200: Neither blocks nor log segments found. Creating new log.
I20250626 14:00:02.218328 8161 log.cc:826] T ba69a20cb49b43f5afd9db92090485a8 P fb4ed636ffea4f41bc55005426dda200: Log is configured to *not* fsync() on all Append() calls
I20250626 14:00:02.222738 8161 tablet_bootstrap.cc:492] T ba69a20cb49b43f5afd9db92090485a8 P fb4ed636ffea4f41bc55005426dda200: No bootstrap required, opened a new log
I20250626 14:00:02.223131 8161 ts_tablet_manager.cc:1397] T ba69a20cb49b43f5afd9db92090485a8 P fb4ed636ffea4f41bc55005426dda200: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20250626 14:00:02.240145 8161 raft_consensus.cc:357] T ba69a20cb49b43f5afd9db92090485a8 P fb4ed636ffea4f41bc55005426dda200 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4ed636ffea4f41bc55005426dda200" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41749 } }
I20250626 14:00:02.240694 8161 raft_consensus.cc:383] T ba69a20cb49b43f5afd9db92090485a8 P fb4ed636ffea4f41bc55005426dda200 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:00:02.240903 8161 raft_consensus.cc:738] T ba69a20cb49b43f5afd9db92090485a8 P fb4ed636ffea4f41bc55005426dda200 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb4ed636ffea4f41bc55005426dda200, State: Initialized, Role: FOLLOWER
I20250626 14:00:02.241611 8161 consensus_queue.cc:260] T ba69a20cb49b43f5afd9db92090485a8 P fb4ed636ffea4f41bc55005426dda200 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4ed636ffea4f41bc55005426dda200" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41749 } }
I20250626 14:00:02.242097 8161 raft_consensus.cc:397] T ba69a20cb49b43f5afd9db92090485a8 P fb4ed636ffea4f41bc55005426dda200 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 14:00:02.242378 8161 raft_consensus.cc:491] T ba69a20cb49b43f5afd9db92090485a8 P fb4ed636ffea4f41bc55005426dda200 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 14:00:02.242632 8161 raft_consensus.cc:3058] T ba69a20cb49b43f5afd9db92090485a8 P fb4ed636ffea4f41bc55005426dda200 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:00:02.247179 8161 raft_consensus.cc:513] T ba69a20cb49b43f5afd9db92090485a8 P fb4ed636ffea4f41bc55005426dda200 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4ed636ffea4f41bc55005426dda200" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41749 } }
I20250626 14:00:02.247834 8161 leader_election.cc:304] T ba69a20cb49b43f5afd9db92090485a8 P fb4ed636ffea4f41bc55005426dda200 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb4ed636ffea4f41bc55005426dda200; no voters:
I20250626 14:00:02.249495 8161 leader_election.cc:290] T ba69a20cb49b43f5afd9db92090485a8 P fb4ed636ffea4f41bc55005426dda200 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 14:00:02.249840 8163 raft_consensus.cc:2802] T ba69a20cb49b43f5afd9db92090485a8 P fb4ed636ffea4f41bc55005426dda200 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:00:02.252334 8163 raft_consensus.cc:695] T ba69a20cb49b43f5afd9db92090485a8 P fb4ed636ffea4f41bc55005426dda200 [term 1 LEADER]: Becoming Leader. State: Replica: fb4ed636ffea4f41bc55005426dda200, State: Running, Role: LEADER
I20250626 14:00:02.252955 8146 heartbeater.cc:499] Master 127.2.107.126:37663 was elected leader, sending a full tablet report...
I20250626 14:00:02.253604 8161 ts_tablet_manager.cc:1428] T ba69a20cb49b43f5afd9db92090485a8 P fb4ed636ffea4f41bc55005426dda200: Time spent starting tablet: real 0.030s user 0.025s sys 0.007s
I20250626 14:00:02.253310 8163 consensus_queue.cc:237] T ba69a20cb49b43f5afd9db92090485a8 P fb4ed636ffea4f41bc55005426dda200 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4ed636ffea4f41bc55005426dda200" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41749 } }
I20250626 14:00:02.265507 7958 catalog_manager.cc:5582] T ba69a20cb49b43f5afd9db92090485a8 P fb4ed636ffea4f41bc55005426dda200 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb4ed636ffea4f41bc55005426dda200 (127.2.107.65). New cstate: current_term: 1 leader_uuid: "fb4ed636ffea4f41bc55005426dda200" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb4ed636ffea4f41bc55005426dda200" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 41749 } health_report { overall_health: HEALTHY } } }
I20250626 14:00:02.291651 2477 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250626 14:00:02.294648 2477 ts_itest-base.cc:246] Waiting for 1 tablets on tserver fb4ed636ffea4f41bc55005426dda200 to finish bootstrapping
I20250626 14:00:04.883647 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 8017
I20250626 14:00:04.905511 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 7925
2025-06-26T14:00:04Z chronyd exiting
[ OK ] ListTableCliSimpleParamTest.TestListTables/2 (6685 ms)
[----------] 1 test from ListTableCliSimpleParamTest (6686 ms total)
[----------] 1 test from ListTableCliParamTest
[ RUN ] ListTableCliParamTest.ListTabletWithPartitionInfo/4
I20250626 14:00:04.965261 2477 test_util.cc:276] Using random seed: -92464272
[ OK ] ListTableCliParamTest.ListTabletWithPartitionInfo/4 (13 ms)
[----------] 1 test from ListTableCliParamTest (13 ms total)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest
[ RUN ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0
2025-06-26T14:00:04Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-26T14:00:04Z Disabled control of system clock
I20250626 14:00:05.018498 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.107.126:40413
--webserver_interface=127.2.107.126
--webserver_port=0
--builtin_ntp_servers=127.2.107.84:46535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.107.126:40413 with env {}
W20250626 14:00:05.314482 8190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 14:00:05.315104 8190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 14:00:05.315568 8190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 14:00:05.347080 8190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 14:00:05.347396 8190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 14:00:05.347667 8190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 14:00:05.347913 8190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 14:00:05.384176 8190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:46535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.107.126:40413
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.107.126:40413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.107.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 14:00:05.385540 8190 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 14:00:05.387372 8190 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 14:00:05.402691 8199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 14:00:05.402711 8197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 14:00:05.403075 8196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 14:00:05.404374 8190 server_base.cc:1048] running on GCE node
I20250626 14:00:06.524086 8190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 14:00:06.527221 8190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 14:00:06.528679 8190 hybrid_clock.cc:648] HybridClock initialized: now 1750946406528636 us; error 58 us; skew 500 ppm
I20250626 14:00:06.529692 8190 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 14:00:06.541891 8190 webserver.cc:469] Webserver started at http://127.2.107.126:42067/ using document root <none> and password file <none>
I20250626 14:00:06.542999 8190 fs_manager.cc:362] Metadata directory not provided
I20250626 14:00:06.543234 8190 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 14:00:06.543819 8190 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 14:00:06.548441 8190 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/master-0/data/instance:
uuid: "1ee7d1504bda4fc398e0c558e3baaafa"
format_stamp: "Formatted at 2025-06-26 14:00:06 on dist-test-slave-7p9x"
I20250626 14:00:06.549829 8190 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/master-0/wal/instance:
uuid: "1ee7d1504bda4fc398e0c558e3baaafa"
format_stamp: "Formatted at 2025-06-26 14:00:06 on dist-test-slave-7p9x"
I20250626 14:00:06.557864 8190 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250626 14:00:06.564152 8206 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 14:00:06.565323 8190 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.006s sys 0.001s
I20250626 14:00:06.565654 8190 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/master-0/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/master-0/wal
uuid: "1ee7d1504bda4fc398e0c558e3baaafa"
format_stamp: "Formatted at 2025-06-26 14:00:06 on dist-test-slave-7p9x"
I20250626 14:00:06.565985 8190 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 14:00:06.634482 8190 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 14:00:06.636085 8190 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 14:00:06.636569 8190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 14:00:06.709957 8190 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.126:40413
I20250626 14:00:06.710065 8257 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.126:40413 every 8 connection(s)
I20250626 14:00:06.712630 8190 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/master-0/data/info.pb
I20250626 14:00:06.715353 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 8190
I20250626 14:00:06.715878 2477 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/master-0/wal/instance
I20250626 14:00:06.718237 8258 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:00:06.745630 8258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ee7d1504bda4fc398e0c558e3baaafa: Bootstrap starting.
I20250626 14:00:06.752444 8258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ee7d1504bda4fc398e0c558e3baaafa: Neither blocks nor log segments found. Creating new log.
I20250626 14:00:06.754648 8258 log.cc:826] T 00000000000000000000000000000000 P 1ee7d1504bda4fc398e0c558e3baaafa: Log is configured to *not* fsync() on all Append() calls
I20250626 14:00:06.759099 8258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ee7d1504bda4fc398e0c558e3baaafa: No bootstrap required, opened a new log
I20250626 14:00:06.778322 8258 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1ee7d1504bda4fc398e0c558e3baaafa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee7d1504bda4fc398e0c558e3baaafa" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 40413 } }
I20250626 14:00:06.779008 8258 raft_consensus.cc:383] T 00000000000000000000000000000000 P 1ee7d1504bda4fc398e0c558e3baaafa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:00:06.779217 8258 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1ee7d1504bda4fc398e0c558e3baaafa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ee7d1504bda4fc398e0c558e3baaafa, State: Initialized, Role: FOLLOWER
I20250626 14:00:06.779834 8258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ee7d1504bda4fc398e0c558e3baaafa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee7d1504bda4fc398e0c558e3baaafa" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 40413 } }
I20250626 14:00:06.780306 8258 raft_consensus.cc:397] T 00000000000000000000000000000000 P 1ee7d1504bda4fc398e0c558e3baaafa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 14:00:06.780553 8258 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1ee7d1504bda4fc398e0c558e3baaafa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 14:00:06.780802 8258 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1ee7d1504bda4fc398e0c558e3baaafa [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:00:06.784870 8258 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1ee7d1504bda4fc398e0c558e3baaafa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee7d1504bda4fc398e0c558e3baaafa" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 40413 } }
I20250626 14:00:06.785581 8258 leader_election.cc:304] T 00000000000000000000000000000000 P 1ee7d1504bda4fc398e0c558e3baaafa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ee7d1504bda4fc398e0c558e3baaafa; no voters:
I20250626 14:00:06.787146 8258 leader_election.cc:290] T 00000000000000000000000000000000 P 1ee7d1504bda4fc398e0c558e3baaafa [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 14:00:06.787950 8263 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1ee7d1504bda4fc398e0c558e3baaafa [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:00:06.790225 8263 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1ee7d1504bda4fc398e0c558e3baaafa [term 1 LEADER]: Becoming Leader. State: Replica: 1ee7d1504bda4fc398e0c558e3baaafa, State: Running, Role: LEADER
I20250626 14:00:06.791179 8263 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ee7d1504bda4fc398e0c558e3baaafa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee7d1504bda4fc398e0c558e3baaafa" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 40413 } }
I20250626 14:00:06.791584 8258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ee7d1504bda4fc398e0c558e3baaafa [sys.catalog]: configured and running, proceeding with master startup.
I20250626 14:00:06.801529 8264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ee7d1504bda4fc398e0c558e3baaafa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ee7d1504bda4fc398e0c558e3baaafa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee7d1504bda4fc398e0c558e3baaafa" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 40413 } } }
I20250626 14:00:06.802433 8264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ee7d1504bda4fc398e0c558e3baaafa [sys.catalog]: This master's current role is: LEADER
I20250626 14:00:06.802174 8265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ee7d1504bda4fc398e0c558e3baaafa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ee7d1504bda4fc398e0c558e3baaafa. Latest consensus state: current_term: 1 leader_uuid: "1ee7d1504bda4fc398e0c558e3baaafa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee7d1504bda4fc398e0c558e3baaafa" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 40413 } } }
I20250626 14:00:06.802819 8265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ee7d1504bda4fc398e0c558e3baaafa [sys.catalog]: This master's current role is: LEADER
I20250626 14:00:06.808404 8273 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 14:00:06.819262 8273 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 14:00:06.836143 8273 catalog_manager.cc:1349] Generated new cluster ID: 15a5ebb6ed26493cac4f61d1f6b37acd
I20250626 14:00:06.836411 8273 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 14:00:06.857337 8273 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 14:00:06.859136 8273 catalog_manager.cc:1506] Loading token signing keys...
I20250626 14:00:06.876893 8273 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 1ee7d1504bda4fc398e0c558e3baaafa: Generated new TSK 0
I20250626 14:00:06.878086 8273 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 14:00:06.897843 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.107.65:0
--local_ip_for_outbound_sockets=127.2.107.65
--webserver_interface=127.2.107.65
--webserver_port=0
--tserver_master_addrs=127.2.107.126:40413
--builtin_ntp_servers=127.2.107.84:46535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20250626 14:00:07.209324 8283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 14:00:07.209828 8283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 14:00:07.210318 8283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 14:00:07.242358 8283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 14:00:07.243208 8283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.107.65
I20250626 14:00:07.277722 8283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:46535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.107.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.107.65
--webserver_port=0
--tserver_master_addrs=127.2.107.126:40413
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.107.65
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 14:00:07.279054 8283 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 14:00:07.280680 8283 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 14:00:07.300935 8290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 14:00:07.301081 8292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 14:00:07.302881 8283 server_base.cc:1048] running on GCE node
W20250626 14:00:07.301612 8289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 14:00:08.412792 8283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 14:00:08.415750 8283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 14:00:08.417266 8283 hybrid_clock.cc:648] HybridClock initialized: now 1750946408417207 us; error 58 us; skew 500 ppm
I20250626 14:00:08.418373 8283 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 14:00:08.426123 8283 webserver.cc:469] Webserver started at http://127.2.107.65:34617/ using document root <none> and password file <none>
I20250626 14:00:08.427433 8283 fs_manager.cc:362] Metadata directory not provided
I20250626 14:00:08.427695 8283 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 14:00:08.428292 8283 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 14:00:08.434770 8283 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-0/data/instance:
uuid: "28dad11a06a14cc38086f0218694ac16"
format_stamp: "Formatted at 2025-06-26 14:00:08 on dist-test-slave-7p9x"
I20250626 14:00:08.436241 8283 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-0/wal/instance:
uuid: "28dad11a06a14cc38086f0218694ac16"
format_stamp: "Formatted at 2025-06-26 14:00:08 on dist-test-slave-7p9x"
I20250626 14:00:08.445329 8283 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s
I20250626 14:00:08.452534 8299 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 14:00:08.453894 8283 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250626 14:00:08.454281 8283 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-0/wal
uuid: "28dad11a06a14cc38086f0218694ac16"
format_stamp: "Formatted at 2025-06-26 14:00:08 on dist-test-slave-7p9x"
I20250626 14:00:08.454702 8283 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 14:00:08.516599 8283 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 14:00:08.518558 8283 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 14:00:08.519109 8283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 14:00:08.522452 8283 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 14:00:08.527786 8283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 14:00:08.528018 8283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 14:00:08.528250 8283 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 14:00:08.528412 8283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 14:00:08.680655 8283 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.65:39641
I20250626 14:00:08.680773 8411 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.65:39641 every 8 connection(s)
I20250626 14:00:08.683928 8283 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-0/data/info.pb
I20250626 14:00:08.691936 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 8283
I20250626 14:00:08.692342 2477 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-0/wal/instance
I20250626 14:00:08.698611 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.107.66:0
--local_ip_for_outbound_sockets=127.2.107.66
--webserver_interface=127.2.107.66
--webserver_port=0
--tserver_master_addrs=127.2.107.126:40413
--builtin_ntp_servers=127.2.107.84:46535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250626 14:00:08.709590 8412 heartbeater.cc:344] Connected to a master server at 127.2.107.126:40413
I20250626 14:00:08.710125 8412 heartbeater.cc:461] Registering TS with master...
I20250626 14:00:08.711462 8412 heartbeater.cc:507] Master 127.2.107.126:40413 requested a full tablet report, sending...
I20250626 14:00:08.714269 8223 ts_manager.cc:194] Registered new tserver with Master: 28dad11a06a14cc38086f0218694ac16 (127.2.107.65:39641)
I20250626 14:00:08.716332 8223 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.107.65:45139
W20250626 14:00:09.012444 8416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 14:00:09.012964 8416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 14:00:09.013722 8416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 14:00:09.046304 8416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 14:00:09.047209 8416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.107.66
I20250626 14:00:09.082291 8416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:46535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.107.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.107.66
--webserver_port=0
--tserver_master_addrs=127.2.107.126:40413
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.107.66
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 14:00:09.083607 8416 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 14:00:09.085314 8416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 14:00:09.102705 8422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 14:00:09.719792 8412 heartbeater.cc:499] Master 127.2.107.126:40413 was elected leader, sending a full tablet report...
W20250626 14:00:09.102769 8423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 14:00:09.105072 8416 server_base.cc:1048] running on GCE node
W20250626 14:00:09.102821 8425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 14:00:10.205960 8416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 14:00:10.208590 8416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 14:00:10.210153 8416 hybrid_clock.cc:648] HybridClock initialized: now 1750946410210075 us; error 108 us; skew 500 ppm
I20250626 14:00:10.211100 8416 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 14:00:10.217550 8416 webserver.cc:469] Webserver started at http://127.2.107.66:42395/ using document root <none> and password file <none>
I20250626 14:00:10.218483 8416 fs_manager.cc:362] Metadata directory not provided
I20250626 14:00:10.218675 8416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 14:00:10.219197 8416 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 14:00:10.223758 8416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-1/data/instance:
uuid: "4f08b723abda4c519a9a04ac98418a8f"
format_stamp: "Formatted at 2025-06-26 14:00:10 on dist-test-slave-7p9x"
I20250626 14:00:10.224826 8416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-1/wal/instance:
uuid: "4f08b723abda4c519a9a04ac98418a8f"
format_stamp: "Formatted at 2025-06-26 14:00:10 on dist-test-slave-7p9x"
I20250626 14:00:10.232056 8416 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20250626 14:00:10.237949 8432 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 14:00:10.239027 8416 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250626 14:00:10.239343 8416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-1/wal
uuid: "4f08b723abda4c519a9a04ac98418a8f"
format_stamp: "Formatted at 2025-06-26 14:00:10 on dist-test-slave-7p9x"
I20250626 14:00:10.239679 8416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 14:00:10.294075 8416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 14:00:10.295552 8416 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 14:00:10.295997 8416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 14:00:10.298547 8416 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 14:00:10.303231 8416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 14:00:10.303439 8416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 14:00:10.303692 8416 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 14:00:10.303861 8416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 14:00:10.439447 8416 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.66:36747
I20250626 14:00:10.439673 8544 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.66:36747 every 8 connection(s)
I20250626 14:00:10.442229 8416 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-1/data/info.pb
I20250626 14:00:10.451095 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 8416
I20250626 14:00:10.451498 2477 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-1/wal/instance
I20250626 14:00:10.457692 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.107.67:0
--local_ip_for_outbound_sockets=127.2.107.67
--webserver_interface=127.2.107.67
--webserver_port=0
--tserver_master_addrs=127.2.107.126:40413
--builtin_ntp_servers=127.2.107.84:46535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250626 14:00:10.464048 8545 heartbeater.cc:344] Connected to a master server at 127.2.107.126:40413
I20250626 14:00:10.464504 8545 heartbeater.cc:461] Registering TS with master...
I20250626 14:00:10.465564 8545 heartbeater.cc:507] Master 127.2.107.126:40413 requested a full tablet report, sending...
I20250626 14:00:10.467833 8223 ts_manager.cc:194] Registered new tserver with Master: 4f08b723abda4c519a9a04ac98418a8f (127.2.107.66:36747)
I20250626 14:00:10.469086 8223 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.107.66:53521
W20250626 14:00:10.770502 8549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 14:00:10.771030 8549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 14:00:10.771528 8549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 14:00:10.803939 8549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 14:00:10.804953 8549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.107.67
I20250626 14:00:10.840870 8549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:46535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.107.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.107.67
--webserver_port=0
--tserver_master_addrs=127.2.107.126:40413
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.107.67
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 14:00:10.842346 8549 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 14:00:10.844056 8549 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 14:00:10.860903 8556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 14:00:11.472673 8545 heartbeater.cc:499] Master 127.2.107.126:40413 was elected leader, sending a full tablet report...
W20250626 14:00:10.861114 8555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 14:00:10.861122 8558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 14:00:10.862407 8549 server_base.cc:1048] running on GCE node
I20250626 14:00:11.956951 8549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 14:00:11.959538 8549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 14:00:11.960983 8549 hybrid_clock.cc:648] HybridClock initialized: now 1750946411960935 us; error 64 us; skew 500 ppm
I20250626 14:00:11.961820 8549 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 14:00:11.972970 8549 webserver.cc:469] Webserver started at http://127.2.107.67:37217/ using document root <none> and password file <none>
I20250626 14:00:11.973958 8549 fs_manager.cc:362] Metadata directory not provided
I20250626 14:00:11.974152 8549 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 14:00:11.974612 8549 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 14:00:11.978984 8549 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-2/data/instance:
uuid: "d062bd8ea8b54c90a06a01953c0a21e8"
format_stamp: "Formatted at 2025-06-26 14:00:11 on dist-test-slave-7p9x"
I20250626 14:00:11.980046 8549 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-2/wal/instance:
uuid: "d062bd8ea8b54c90a06a01953c0a21e8"
format_stamp: "Formatted at 2025-06-26 14:00:11 on dist-test-slave-7p9x"
I20250626 14:00:11.987594 8549 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20250626 14:00:11.993412 8565 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 14:00:11.994958 8549 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250626 14:00:11.995306 8549 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-2/wal
uuid: "d062bd8ea8b54c90a06a01953c0a21e8"
format_stamp: "Formatted at 2025-06-26 14:00:11 on dist-test-slave-7p9x"
I20250626 14:00:11.995652 8549 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 14:00:12.057638 8549 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 14:00:12.059129 8549 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 14:00:12.059588 8549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 14:00:12.062110 8549 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 14:00:12.066229 8549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 14:00:12.066437 8549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 14:00:12.066799 8549 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 14:00:12.066982 8549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 14:00:12.198697 8549 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.67:42909
I20250626 14:00:12.198808 8677 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.67:42909 every 8 connection(s)
I20250626 14:00:12.201295 8549 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-2/data/info.pb
I20250626 14:00:12.207448 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 8549
I20250626 14:00:12.207933 2477 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-2/wal/instance
I20250626 14:00:12.221480 8678 heartbeater.cc:344] Connected to a master server at 127.2.107.126:40413
I20250626 14:00:12.221926 8678 heartbeater.cc:461] Registering TS with master...
I20250626 14:00:12.222908 8678 heartbeater.cc:507] Master 127.2.107.126:40413 requested a full tablet report, sending...
I20250626 14:00:12.225117 8223 ts_manager.cc:194] Registered new tserver with Master: d062bd8ea8b54c90a06a01953c0a21e8 (127.2.107.67:42909)
I20250626 14:00:12.226830 8223 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.107.67:54029
I20250626 14:00:12.228803 2477 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 14:00:12.255055 2477 test_util.cc:276] Using random seed: -85174473
I20250626 14:00:12.297194 8222 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48260:
name: "pre_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 14:00:12.299714 8222 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table pre_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 14:00:12.355355 8480 tablet_service.cc:1468] Processing CreateTablet for tablet 7f46a98c934042f98865dfbb088ddeea (DEFAULT_TABLE table=pre_rebuild [id=4408deb01b0442d6bf93fe81849b3b73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:00:12.356215 8613 tablet_service.cc:1468] Processing CreateTablet for tablet 7f46a98c934042f98865dfbb088ddeea (DEFAULT_TABLE table=pre_rebuild [id=4408deb01b0442d6bf93fe81849b3b73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:00:12.357437 8480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f46a98c934042f98865dfbb088ddeea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:00:12.357843 8613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f46a98c934042f98865dfbb088ddeea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:00:12.357285 8347 tablet_service.cc:1468] Processing CreateTablet for tablet 7f46a98c934042f98865dfbb088ddeea (DEFAULT_TABLE table=pre_rebuild [id=4408deb01b0442d6bf93fe81849b3b73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:00:12.358999 8347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f46a98c934042f98865dfbb088ddeea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:00:12.380281 8702 tablet_bootstrap.cc:492] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8: Bootstrap starting.
I20250626 14:00:12.386067 8703 tablet_bootstrap.cc:492] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f: Bootstrap starting.
I20250626 14:00:12.388792 8704 tablet_bootstrap.cc:492] T 7f46a98c934042f98865dfbb088ddeea P 28dad11a06a14cc38086f0218694ac16: Bootstrap starting.
I20250626 14:00:12.390372 8702 tablet_bootstrap.cc:654] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8: Neither blocks nor log segments found. Creating new log.
I20250626 14:00:12.392664 8702 log.cc:826] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8: Log is configured to *not* fsync() on all Append() calls
I20250626 14:00:12.393393 8703 tablet_bootstrap.cc:654] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f: Neither blocks nor log segments found. Creating new log.
I20250626 14:00:12.394152 8704 tablet_bootstrap.cc:654] T 7f46a98c934042f98865dfbb088ddeea P 28dad11a06a14cc38086f0218694ac16: Neither blocks nor log segments found. Creating new log.
I20250626 14:00:12.395678 8703 log.cc:826] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f: Log is configured to *not* fsync() on all Append() calls
I20250626 14:00:12.396350 8704 log.cc:826] T 7f46a98c934042f98865dfbb088ddeea P 28dad11a06a14cc38086f0218694ac16: Log is configured to *not* fsync() on all Append() calls
I20250626 14:00:12.399986 8702 tablet_bootstrap.cc:492] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8: No bootstrap required, opened a new log
I20250626 14:00:12.400558 8702 ts_tablet_manager.cc:1397] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.008s
I20250626 14:00:12.401270 8703 tablet_bootstrap.cc:492] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f: No bootstrap required, opened a new log
I20250626 14:00:12.401739 8703 ts_tablet_manager.cc:1397] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.012s
I20250626 14:00:12.402865 8704 tablet_bootstrap.cc:492] T 7f46a98c934042f98865dfbb088ddeea P 28dad11a06a14cc38086f0218694ac16: No bootstrap required, opened a new log
I20250626 14:00:12.403327 8704 ts_tablet_manager.cc:1397] T 7f46a98c934042f98865dfbb088ddeea P 28dad11a06a14cc38086f0218694ac16: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20250626 14:00:12.418907 8702 raft_consensus.cc:357] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 42909 } } peers { permanent_uuid: "4f08b723abda4c519a9a04ac98418a8f" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 36747 } } peers { permanent_uuid: "28dad11a06a14cc38086f0218694ac16" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 39641 } }
I20250626 14:00:12.419617 8702 raft_consensus.cc:383] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:00:12.419895 8702 raft_consensus.cc:738] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d062bd8ea8b54c90a06a01953c0a21e8, State: Initialized, Role: FOLLOWER
I20250626 14:00:12.420698 8702 consensus_queue.cc:260] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 42909 } } peers { permanent_uuid: "4f08b723abda4c519a9a04ac98418a8f" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 36747 } } peers { permanent_uuid: "28dad11a06a14cc38086f0218694ac16" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 39641 } }
I20250626 14:00:12.423761 8678 heartbeater.cc:499] Master 127.2.107.126:40413 was elected leader, sending a full tablet report...
I20250626 14:00:12.425114 8702 ts_tablet_manager.cc:1428] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8: Time spent starting tablet: real 0.024s user 0.022s sys 0.002s
I20250626 14:00:12.428192 8703 raft_consensus.cc:357] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 42909 } } peers { permanent_uuid: "4f08b723abda4c519a9a04ac98418a8f" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 36747 } } peers { permanent_uuid: "28dad11a06a14cc38086f0218694ac16" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 39641 } }
I20250626 14:00:12.429067 8703 raft_consensus.cc:383] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:00:12.430198 8703 raft_consensus.cc:738] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f08b723abda4c519a9a04ac98418a8f, State: Initialized, Role: FOLLOWER
I20250626 14:00:12.429451 8704 raft_consensus.cc:357] T 7f46a98c934042f98865dfbb088ddeea P 28dad11a06a14cc38086f0218694ac16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 42909 } } peers { permanent_uuid: "4f08b723abda4c519a9a04ac98418a8f" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 36747 } } peers { permanent_uuid: "28dad11a06a14cc38086f0218694ac16" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 39641 } }
I20250626 14:00:12.430693 8704 raft_consensus.cc:383] T 7f46a98c934042f98865dfbb088ddeea P 28dad11a06a14cc38086f0218694ac16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:00:12.431134 8704 raft_consensus.cc:738] T 7f46a98c934042f98865dfbb088ddeea P 28dad11a06a14cc38086f0218694ac16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28dad11a06a14cc38086f0218694ac16, State: Initialized, Role: FOLLOWER
I20250626 14:00:12.431293 8703 consensus_queue.cc:260] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 42909 } } peers { permanent_uuid: "4f08b723abda4c519a9a04ac98418a8f" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 36747 } } peers { permanent_uuid: "28dad11a06a14cc38086f0218694ac16" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 39641 } }
I20250626 14:00:12.432303 8704 consensus_queue.cc:260] T 7f46a98c934042f98865dfbb088ddeea P 28dad11a06a14cc38086f0218694ac16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 42909 } } peers { permanent_uuid: "4f08b723abda4c519a9a04ac98418a8f" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 36747 } } peers { permanent_uuid: "28dad11a06a14cc38086f0218694ac16" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 39641 } }
I20250626 14:00:12.435467 8703 ts_tablet_manager.cc:1428] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f: Time spent starting tablet: real 0.033s user 0.022s sys 0.010s
I20250626 14:00:12.438047 8704 ts_tablet_manager.cc:1428] T 7f46a98c934042f98865dfbb088ddeea P 28dad11a06a14cc38086f0218694ac16: Time spent starting tablet: real 0.034s user 0.031s sys 0.004s
W20250626 14:00:12.445755 8413 tablet.cc:2378] T 7f46a98c934042f98865dfbb088ddeea P 28dad11a06a14cc38086f0218694ac16: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250626 14:00:12.448756 8546 tablet.cc:2378] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 14:00:12.453289 8708 raft_consensus.cc:491] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:00:12.453863 8708 raft_consensus.cc:513] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 42909 } } peers { permanent_uuid: "4f08b723abda4c519a9a04ac98418a8f" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 36747 } } peers { permanent_uuid: "28dad11a06a14cc38086f0218694ac16" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 39641 } }
W20250626 14:00:12.455564 8679 tablet.cc:2378] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 14:00:12.456657 8708 leader_election.cc:290] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f08b723abda4c519a9a04ac98418a8f (127.2.107.66:36747), 28dad11a06a14cc38086f0218694ac16 (127.2.107.65:39641)
I20250626 14:00:12.468597 8367 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f46a98c934042f98865dfbb088ddeea" candidate_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28dad11a06a14cc38086f0218694ac16" is_pre_election: true
I20250626 14:00:12.468595 8500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f46a98c934042f98865dfbb088ddeea" candidate_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f08b723abda4c519a9a04ac98418a8f" is_pre_election: true
I20250626 14:00:12.469514 8367 raft_consensus.cc:2466] T 7f46a98c934042f98865dfbb088ddeea P 28dad11a06a14cc38086f0218694ac16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d062bd8ea8b54c90a06a01953c0a21e8 in term 0.
I20250626 14:00:12.469514 8500 raft_consensus.cc:2466] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d062bd8ea8b54c90a06a01953c0a21e8 in term 0.
I20250626 14:00:12.470817 8568 leader_election.cc:304] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28dad11a06a14cc38086f0218694ac16, d062bd8ea8b54c90a06a01953c0a21e8; no voters:
I20250626 14:00:12.471645 8708 raft_consensus.cc:2802] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 14:00:12.471975 8708 raft_consensus.cc:491] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:00:12.472251 8708 raft_consensus.cc:3058] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:00:12.477227 8708 raft_consensus.cc:513] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 42909 } } peers { permanent_uuid: "4f08b723abda4c519a9a04ac98418a8f" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 36747 } } peers { permanent_uuid: "28dad11a06a14cc38086f0218694ac16" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 39641 } }
I20250626 14:00:12.478842 8708 leader_election.cc:290] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8 [CANDIDATE]: Term 1 election: Requested vote from peers 4f08b723abda4c519a9a04ac98418a8f (127.2.107.66:36747), 28dad11a06a14cc38086f0218694ac16 (127.2.107.65:39641)
I20250626 14:00:12.479490 8500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f46a98c934042f98865dfbb088ddeea" candidate_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f08b723abda4c519a9a04ac98418a8f"
I20250626 14:00:12.479781 8367 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f46a98c934042f98865dfbb088ddeea" candidate_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28dad11a06a14cc38086f0218694ac16"
I20250626 14:00:12.480029 8500 raft_consensus.cc:3058] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:00:12.480279 8367 raft_consensus.cc:3058] T 7f46a98c934042f98865dfbb088ddeea P 28dad11a06a14cc38086f0218694ac16 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:00:12.486979 8367 raft_consensus.cc:2466] T 7f46a98c934042f98865dfbb088ddeea P 28dad11a06a14cc38086f0218694ac16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d062bd8ea8b54c90a06a01953c0a21e8 in term 1.
I20250626 14:00:12.486982 8500 raft_consensus.cc:2466] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d062bd8ea8b54c90a06a01953c0a21e8 in term 1.
I20250626 14:00:12.488230 8568 leader_election.cc:304] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28dad11a06a14cc38086f0218694ac16, d062bd8ea8b54c90a06a01953c0a21e8; no voters:
I20250626 14:00:12.488888 8708 raft_consensus.cc:2802] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:00:12.490409 8708 raft_consensus.cc:695] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8 [term 1 LEADER]: Becoming Leader. State: Replica: d062bd8ea8b54c90a06a01953c0a21e8, State: Running, Role: LEADER
I20250626 14:00:12.491271 8708 consensus_queue.cc:237] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 42909 } } peers { permanent_uuid: "4f08b723abda4c519a9a04ac98418a8f" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 36747 } } peers { permanent_uuid: "28dad11a06a14cc38086f0218694ac16" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 39641 } }
I20250626 14:00:12.503419 8222 catalog_manager.cc:5582] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to d062bd8ea8b54c90a06a01953c0a21e8 (127.2.107.67). New cstate: current_term: 1 leader_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 42909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f08b723abda4c519a9a04ac98418a8f" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 36747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28dad11a06a14cc38086f0218694ac16" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 39641 } health_report { overall_health: UNKNOWN } } }
I20250626 14:00:12.654501 8367 raft_consensus.cc:1273] T 7f46a98c934042f98865dfbb088ddeea P 28dad11a06a14cc38086f0218694ac16 [term 1 FOLLOWER]: Refusing update from remote peer d062bd8ea8b54c90a06a01953c0a21e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 14:00:12.654882 8500 raft_consensus.cc:1273] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f [term 1 FOLLOWER]: Refusing update from remote peer d062bd8ea8b54c90a06a01953c0a21e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 14:00:12.656049 8712 consensus_queue.cc:1035] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28dad11a06a14cc38086f0218694ac16" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 39641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 14:00:12.657061 8708 consensus_queue.cc:1035] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f08b723abda4c519a9a04ac98418a8f" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 36747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:00:12.679531 8719 mvcc.cc:204] Tried to move back new op lower bound from 7171876506216230912 to 7171876505572376576. Current Snapshot: MvccSnapshot[applied={T|T < 7171876506216230912}]
I20250626 14:00:12.706516 8721 mvcc.cc:204] Tried to move back new op lower bound from 7171876506216230912 to 7171876505572376576. Current Snapshot: MvccSnapshot[applied={T|T < 7171876506216230912}]
I20250626 14:00:17.942005 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 8190
W20250626 14:00:18.278573 8756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 14:00:18.279224 8756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 14:00:18.313345 8756 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250626 14:00:18.767665 8678 heartbeater.cc:646] Failed to heartbeat to 127.2.107.126:40413 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.107.126:40413: connect: Connection refused (error 111)
W20250626 14:00:18.803083 8412 heartbeater.cc:646] Failed to heartbeat to 127.2.107.126:40413 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.107.126:40413: connect: Connection refused (error 111)
W20250626 14:00:18.830735 8545 heartbeater.cc:646] Failed to heartbeat to 127.2.107.126:40413 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.107.126:40413: connect: Connection refused (error 111)
W20250626 14:00:19.552877 8756 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.197s user 0.541s sys 0.651s
W20250626 14:00:19.553277 8756 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.197s user 0.545s sys 0.651s
I20250626 14:00:19.681236 8756 minidump.cc:252] Setting minidump size limit to 20M
I20250626 14:00:19.683808 8756 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 14:00:19.685346 8756 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 14:00:19.698973 8786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 14:00:19.701525 8787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 14:00:19.904168 8789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 14:00:19.904410 8756 server_base.cc:1048] running on GCE node
I20250626 14:00:19.905787 8756 hybrid_clock.cc:584] initializing the hybrid clock with 'system' time source
I20250626 14:00:19.906332 8756 hybrid_clock.cc:648] HybridClock initialized: now 1750946419906303 us; error 274016 us; skew 500 ppm
I20250626 14:00:19.907227 8756 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 14:00:19.912597 8756 webserver.cc:469] Webserver started at http://0.0.0.0:45401/ using document root <none> and password file <none>
I20250626 14:00:19.913751 8756 fs_manager.cc:362] Metadata directory not provided
I20250626 14:00:19.914036 8756 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 14:00:19.914542 8756 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 14:00:19.920881 8756 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/master-0/data/instance:
uuid: "538dac2d162c4a62a2ca9fdd3ae02f38"
format_stamp: "Formatted at 2025-06-26 14:00:19 on dist-test-slave-7p9x"
I20250626 14:00:19.922556 8756 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/master-0/wal/instance:
uuid: "538dac2d162c4a62a2ca9fdd3ae02f38"
format_stamp: "Formatted at 2025-06-26 14:00:19 on dist-test-slave-7p9x"
I20250626 14:00:19.930495 8756 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20250626 14:00:19.937021 8797 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 14:00:19.938334 8756 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250626 14:00:19.938745 8756 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/master-0/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/master-0/wal
uuid: "538dac2d162c4a62a2ca9fdd3ae02f38"
format_stamp: "Formatted at 2025-06-26 14:00:19 on dist-test-slave-7p9x"
I20250626 14:00:19.939182 8756 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 14:00:20.090314 8756 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 14:00:20.091709 8756 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 14:00:20.092208 8756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 14:00:20.097625 8756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:00:20.112516 8756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38: Bootstrap starting.
I20250626 14:00:20.117619 8756 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38: Neither blocks nor log segments found. Creating new log.
I20250626 14:00:20.119340 8756 log.cc:826] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38: Log is configured to *not* fsync() on all Append() calls
I20250626 14:00:20.123512 8756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38: No bootstrap required, opened a new log
I20250626 14:00:20.140224 8756 raft_consensus.cc:357] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538dac2d162c4a62a2ca9fdd3ae02f38" member_type: VOTER }
I20250626 14:00:20.140750 8756 raft_consensus.cc:383] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:00:20.140985 8756 raft_consensus.cc:738] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 538dac2d162c4a62a2ca9fdd3ae02f38, State: Initialized, Role: FOLLOWER
I20250626 14:00:20.141755 8756 consensus_queue.cc:260] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538dac2d162c4a62a2ca9fdd3ae02f38" member_type: VOTER }
I20250626 14:00:20.142254 8756 raft_consensus.cc:397] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 14:00:20.142498 8756 raft_consensus.cc:491] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 14:00:20.142788 8756 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:00:20.146772 8756 raft_consensus.cc:513] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538dac2d162c4a62a2ca9fdd3ae02f38" member_type: VOTER }
I20250626 14:00:20.147419 8756 leader_election.cc:304] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 538dac2d162c4a62a2ca9fdd3ae02f38; no voters:
I20250626 14:00:20.149042 8756 leader_election.cc:290] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 14:00:20.149358 8804 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:00:20.151922 8804 raft_consensus.cc:695] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38 [term 1 LEADER]: Becoming Leader. State: Replica: 538dac2d162c4a62a2ca9fdd3ae02f38, State: Running, Role: LEADER
I20250626 14:00:20.152695 8804 consensus_queue.cc:237] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538dac2d162c4a62a2ca9fdd3ae02f38" member_type: VOTER }
I20250626 14:00:20.161075 8805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 538dac2d162c4a62a2ca9fdd3ae02f38. Latest consensus state: current_term: 1 leader_uuid: "538dac2d162c4a62a2ca9fdd3ae02f38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538dac2d162c4a62a2ca9fdd3ae02f38" member_type: VOTER } }
I20250626 14:00:20.161652 8805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38 [sys.catalog]: This master's current role is: LEADER
I20250626 14:00:20.162055 8806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "538dac2d162c4a62a2ca9fdd3ae02f38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538dac2d162c4a62a2ca9fdd3ae02f38" member_type: VOTER } }
I20250626 14:00:20.162449 8806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38 [sys.catalog]: This master's current role is: LEADER
I20250626 14:00:20.172901 8756 tablet_replica.cc:331] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38: stopping tablet replica
I20250626 14:00:20.173450 8756 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38 [term 1 LEADER]: Raft consensus shutting down.
I20250626 14:00:20.173899 8756 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 14:00:20.175866 8756 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250626 14:00:20.176342 8756 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250626 14:00:20.203197 8756 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
I20250626 14:00:21.232630 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 8283
I20250626 14:00:21.263617 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 8416
I20250626 14:00:21.292353 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 8549
I20250626 14:00:21.329442 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.107.126:40413
--webserver_interface=127.2.107.126
--webserver_port=42067
--builtin_ntp_servers=127.2.107.84:46535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.107.126:40413 with env {}
W20250626 14:00:21.632412 8814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 14:00:21.633025 8814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 14:00:21.633502 8814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 14:00:21.664605 8814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 14:00:21.664968 8814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 14:00:21.665431 8814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 14:00:21.665833 8814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 14:00:21.701081 8814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:46535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.107.126:40413
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.107.126:40413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.107.126
--webserver_port=42067
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 14:00:21.702461 8814 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 14:00:21.704560 8814 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 14:00:21.719841 8821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 14:00:21.719867 8820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 14:00:21.721783 8823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 14:00:21.721879 8814 server_base.cc:1048] running on GCE node
I20250626 14:00:22.829394 8814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 14:00:22.832376 8814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 14:00:22.833809 8814 hybrid_clock.cc:648] HybridClock initialized: now 1750946422833717 us; error 104 us; skew 500 ppm
I20250626 14:00:22.834610 8814 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 14:00:22.845927 8814 webserver.cc:469] Webserver started at http://127.2.107.126:42067/ using document root <none> and password file <none>
I20250626 14:00:22.847445 8814 fs_manager.cc:362] Metadata directory not provided
I20250626 14:00:22.847842 8814 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 14:00:22.861953 8814 fs_manager.cc:714] Time spent opening directory manager: real 0.009s user 0.004s sys 0.004s
I20250626 14:00:22.868122 8830 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 14:00:22.869619 8814 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.001s
I20250626 14:00:22.870188 8814 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/master-0/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/master-0/wal
uuid: "538dac2d162c4a62a2ca9fdd3ae02f38"
format_stamp: "Formatted at 2025-06-26 14:00:19 on dist-test-slave-7p9x"
I20250626 14:00:22.873092 8814 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 14:00:22.955322 8814 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 14:00:22.956817 8814 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 14:00:22.957298 8814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 14:00:23.027092 8814 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.126:40413
I20250626 14:00:23.027199 8881 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.126:40413 every 8 connection(s)
I20250626 14:00:23.029903 8814 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/master-0/data/info.pb
I20250626 14:00:23.030876 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 8814
I20250626 14:00:23.032967 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.107.65:39641
--local_ip_for_outbound_sockets=127.2.107.65
--tserver_master_addrs=127.2.107.126:40413
--webserver_port=34617
--webserver_interface=127.2.107.65
--builtin_ntp_servers=127.2.107.84:46535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250626 14:00:23.041421 8882 sys_catalog.cc:263] Verifying existing consensus state
I20250626 14:00:23.058871 8882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38: Bootstrap starting.
I20250626 14:00:23.072860 8882 log.cc:826] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38: Log is configured to *not* fsync() on all Append() calls
I20250626 14:00:23.088542 8882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 14:00:23.089607 8882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38: Bootstrap complete.
I20250626 14:00:23.118489 8882 raft_consensus.cc:357] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538dac2d162c4a62a2ca9fdd3ae02f38" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 40413 } }
I20250626 14:00:23.119166 8882 raft_consensus.cc:738] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 538dac2d162c4a62a2ca9fdd3ae02f38, State: Initialized, Role: FOLLOWER
I20250626 14:00:23.119916 8882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538dac2d162c4a62a2ca9fdd3ae02f38" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 40413 } }
I20250626 14:00:23.120447 8882 raft_consensus.cc:397] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 14:00:23.120808 8882 raft_consensus.cc:491] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 14:00:23.121202 8882 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38 [term 1 FOLLOWER]: Advancing to term 2
I20250626 14:00:23.125452 8882 raft_consensus.cc:513] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538dac2d162c4a62a2ca9fdd3ae02f38" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 40413 } }
I20250626 14:00:23.126144 8882 leader_election.cc:304] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 538dac2d162c4a62a2ca9fdd3ae02f38; no voters:
I20250626 14:00:23.128005 8882 leader_election.cc:290] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250626 14:00:23.128446 8886 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 14:00:23.131343 8886 raft_consensus.cc:695] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38 [term 2 LEADER]: Becoming Leader. State: Replica: 538dac2d162c4a62a2ca9fdd3ae02f38, State: Running, Role: LEADER
I20250626 14:00:23.132256 8886 consensus_queue.cc:237] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538dac2d162c4a62a2ca9fdd3ae02f38" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 40413 } }
I20250626 14:00:23.132652 8882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 14:00:23.143950 8888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 538dac2d162c4a62a2ca9fdd3ae02f38. Latest consensus state: current_term: 2 leader_uuid: "538dac2d162c4a62a2ca9fdd3ae02f38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538dac2d162c4a62a2ca9fdd3ae02f38" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 40413 } } }
I20250626 14:00:23.144942 8888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38 [sys.catalog]: This master's current role is: LEADER
I20250626 14:00:23.145974 8887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "538dac2d162c4a62a2ca9fdd3ae02f38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538dac2d162c4a62a2ca9fdd3ae02f38" member_type: VOTER last_known_addr { host: "127.2.107.126" port: 40413 } } }
I20250626 14:00:23.146561 8887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38 [sys.catalog]: This master's current role is: LEADER
I20250626 14:00:23.148443 8895 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 14:00:23.161039 8895 catalog_manager.cc:671] Loaded metadata for table pre_rebuild [id=7a4235e9e55a4377a0c744fe48eed350]
I20250626 14:00:23.168315 8895 tablet_loader.cc:96] loaded metadata for tablet 7f46a98c934042f98865dfbb088ddeea (table pre_rebuild [id=7a4235e9e55a4377a0c744fe48eed350])
I20250626 14:00:23.169955 8895 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 14:00:23.187676 8895 catalog_manager.cc:1349] Generated new cluster ID: 1bb8f87c81ba4de58b0b8e0b2aa3535f
I20250626 14:00:23.187970 8895 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 14:00:23.224702 8895 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 14:00:23.227021 8895 catalog_manager.cc:1506] Loading token signing keys...
I20250626 14:00:23.243485 8895 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38: Generated new TSK 0
I20250626 14:00:23.244596 8895 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250626 14:00:23.408223 8884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 14:00:23.408735 8884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 14:00:23.409296 8884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 14:00:23.440953 8884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 14:00:23.441876 8884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.107.65
I20250626 14:00:23.479673 8884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:46535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.107.65:39641
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.107.65
--webserver_port=34617
--tserver_master_addrs=127.2.107.126:40413
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.107.65
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 14:00:23.481069 8884 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 14:00:23.483449 8884 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 14:00:23.501302 8911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 14:00:23.501765 8913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 14:00:23.502802 8884 server_base.cc:1048] running on GCE node
W20250626 14:00:23.503536 8910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 14:00:24.599023 8884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 14:00:24.611675 8884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 14:00:24.613219 8884 hybrid_clock.cc:648] HybridClock initialized: now 1750946424613174 us; error 70 us; skew 500 ppm
I20250626 14:00:24.614075 8884 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 14:00:24.620893 8884 webserver.cc:469] Webserver started at http://127.2.107.65:34617/ using document root <none> and password file <none>
I20250626 14:00:24.621850 8884 fs_manager.cc:362] Metadata directory not provided
I20250626 14:00:24.622215 8884 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 14:00:24.631242 8884 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.001s sys 0.004s
I20250626 14:00:24.636363 8921 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 14:00:24.637563 8884 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250626 14:00:24.637876 8884 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-0/wal
uuid: "28dad11a06a14cc38086f0218694ac16"
format_stamp: "Formatted at 2025-06-26 14:00:08 on dist-test-slave-7p9x"
I20250626 14:00:24.639811 8884 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 14:00:24.685479 8884 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 14:00:24.686929 8884 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 14:00:24.687361 8884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 14:00:24.690312 8884 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 14:00:24.696789 8928 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250626 14:00:24.704483 8884 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250626 14:00:24.704710 8884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.002s sys 0.001s
I20250626 14:00:24.705016 8884 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250626 14:00:24.709743 8884 ts_tablet_manager.cc:610] Registered 1 tablets
I20250626 14:00:24.710005 8884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.005s sys 0.000s
I20250626 14:00:24.710450 8928 tablet_bootstrap.cc:492] T 7f46a98c934042f98865dfbb088ddeea P 28dad11a06a14cc38086f0218694ac16: Bootstrap starting.
I20250626 14:00:24.886229 8884 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.65:39641
I20250626 14:00:24.886413 9034 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.65:39641 every 8 connection(s)
I20250626 14:00:24.889928 8884 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-0/data/info.pb
I20250626 14:00:24.899791 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 8884
I20250626 14:00:24.901952 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.107.66:36747
--local_ip_for_outbound_sockets=127.2.107.66
--tserver_master_addrs=127.2.107.126:40413
--webserver_port=42395
--webserver_interface=127.2.107.66
--builtin_ntp_servers=127.2.107.84:46535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250626 14:00:24.935223 9035 heartbeater.cc:344] Connected to a master server at 127.2.107.126:40413
I20250626 14:00:24.935760 9035 heartbeater.cc:461] Registering TS with master...
I20250626 14:00:24.937088 9035 heartbeater.cc:507] Master 127.2.107.126:40413 requested a full tablet report, sending...
I20250626 14:00:24.942342 8847 ts_manager.cc:194] Registered new tserver with Master: 28dad11a06a14cc38086f0218694ac16 (127.2.107.65:39641)
I20250626 14:00:24.950736 8847 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.107.65:34675
I20250626 14:00:24.994978 8928 log.cc:826] T 7f46a98c934042f98865dfbb088ddeea P 28dad11a06a14cc38086f0218694ac16: Log is configured to *not* fsync() on all Append() calls
W20250626 14:00:25.283463 9039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 14:00:25.283995 9039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 14:00:25.284511 9039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 14:00:25.317749 9039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 14:00:25.318624 9039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.107.66
I20250626 14:00:25.355764 9039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:46535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.107.66:36747
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.107.66
--webserver_port=42395
--tserver_master_addrs=127.2.107.126:40413
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.107.66
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 14:00:25.357096 9039 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 14:00:25.359184 9039 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 14:00:25.377218 9047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 14:00:25.955858 9035 heartbeater.cc:499] Master 127.2.107.126:40413 was elected leader, sending a full tablet report...
W20250626 14:00:25.379200 9046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 14:00:25.379046 9049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 14:00:25.379281 9039 server_base.cc:1048] running on GCE node
I20250626 14:00:26.490800 9039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 14:00:26.493455 9039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 14:00:26.494930 9039 hybrid_clock.cc:648] HybridClock initialized: now 1750946426494880 us; error 67 us; skew 500 ppm
I20250626 14:00:26.495786 9039 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 14:00:26.502662 9039 webserver.cc:469] Webserver started at http://127.2.107.66:42395/ using document root <none> and password file <none>
I20250626 14:00:26.503588 9039 fs_manager.cc:362] Metadata directory not provided
I20250626 14:00:26.503800 9039 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 14:00:26.513541 9039 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.002s sys 0.004s
I20250626 14:00:26.518633 9056 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 14:00:26.519718 9039 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250626 14:00:26.520027 9039 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-1/wal
uuid: "4f08b723abda4c519a9a04ac98418a8f"
format_stamp: "Formatted at 2025-06-26 14:00:10 on dist-test-slave-7p9x"
I20250626 14:00:26.522104 9039 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 14:00:26.573945 9039 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 14:00:26.575409 9039 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 14:00:26.575816 9039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 14:00:26.578508 9039 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 14:00:26.584563 9063 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250626 14:00:26.595674 9039 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250626 14:00:26.595937 9039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.002s sys 0.000s
I20250626 14:00:26.596335 9039 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250626 14:00:26.601029 9039 ts_tablet_manager.cc:610] Registered 1 tablets
I20250626 14:00:26.601276 9039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.006s sys 0.000s
I20250626 14:00:26.601749 9063 tablet_bootstrap.cc:492] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f: Bootstrap starting.
I20250626 14:00:26.778966 9039 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.66:36747
I20250626 14:00:26.779151 9169 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.66:36747 every 8 connection(s)
I20250626 14:00:26.782721 9039 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-1/data/info.pb
I20250626 14:00:26.783893 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 9039
I20250626 14:00:26.785727 2477 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
/tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.107.67:42909
--local_ip_for_outbound_sockets=127.2.107.67
--tserver_master_addrs=127.2.107.126:40413
--webserver_port=37217
--webserver_interface=127.2.107.67
--builtin_ntp_servers=127.2.107.84:46535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250626 14:00:26.855345 9170 heartbeater.cc:344] Connected to a master server at 127.2.107.126:40413
I20250626 14:00:26.855813 9170 heartbeater.cc:461] Registering TS with master...
I20250626 14:00:26.857021 9170 heartbeater.cc:507] Master 127.2.107.126:40413 requested a full tablet report, sending...
I20250626 14:00:26.860888 8847 ts_manager.cc:194] Registered new tserver with Master: 4f08b723abda4c519a9a04ac98418a8f (127.2.107.66:36747)
I20250626 14:00:26.863889 8847 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.107.66:33555
I20250626 14:00:26.900871 9063 log.cc:826] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f: Log is configured to *not* fsync() on all Append() calls
I20250626 14:00:27.138309 8928 tablet_bootstrap.cc:492] T 7f46a98c934042f98865dfbb088ddeea P 28dad11a06a14cc38086f0218694ac16: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 14:00:27.139088 8928 tablet_bootstrap.cc:492] T 7f46a98c934042f98865dfbb088ddeea P 28dad11a06a14cc38086f0218694ac16: Bootstrap complete.
I20250626 14:00:27.140262 8928 ts_tablet_manager.cc:1397] T 7f46a98c934042f98865dfbb088ddeea P 28dad11a06a14cc38086f0218694ac16: Time spent bootstrapping tablet: real 2.430s user 2.345s sys 0.040s
I20250626 14:00:27.151700 8928 raft_consensus.cc:357] T 7f46a98c934042f98865dfbb088ddeea P 28dad11a06a14cc38086f0218694ac16 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 42909 } } peers { permanent_uuid: "4f08b723abda4c519a9a04ac98418a8f" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 36747 } } peers { permanent_uuid: "28dad11a06a14cc38086f0218694ac16" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 39641 } }
I20250626 14:00:27.153904 8928 raft_consensus.cc:738] T 7f46a98c934042f98865dfbb088ddeea P 28dad11a06a14cc38086f0218694ac16 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28dad11a06a14cc38086f0218694ac16, State: Initialized, Role: FOLLOWER
I20250626 14:00:27.154708 8928 consensus_queue.cc:260] T 7f46a98c934042f98865dfbb088ddeea P 28dad11a06a14cc38086f0218694ac16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 42909 } } peers { permanent_uuid: "4f08b723abda4c519a9a04ac98418a8f" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 36747 } } peers { permanent_uuid: "28dad11a06a14cc38086f0218694ac16" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 39641 } }
I20250626 14:00:27.158035 8928 ts_tablet_manager.cc:1428] T 7f46a98c934042f98865dfbb088ddeea P 28dad11a06a14cc38086f0218694ac16: Time spent starting tablet: real 0.018s user 0.020s sys 0.000s
W20250626 14:00:27.244091 9171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 14:00:27.244755 9171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 14:00:27.245568 9171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 14:00:27.287180 9171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 14:00:27.288061 9171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.107.67
I20250626 14:00:27.324092 9171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.107.84:46535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.107.67:42909
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.107.67
--webserver_port=37217
--tserver_master_addrs=127.2.107.126:40413
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.107.67
--log_dir=/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 14:00:27.325436 9171 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 14:00:27.327067 9171 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 14:00:27.345026 9182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 14:00:27.868151 9170 heartbeater.cc:499] Master 127.2.107.126:40413 was elected leader, sending a full tablet report...
W20250626 14:00:27.345726 9183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 14:00:27.347255 9185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 14:00:28.451613 9184 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 14:00:28.451673 9171 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 14:00:28.457407 9171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 14:00:28.460116 9171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 14:00:28.461669 9171 hybrid_clock.cc:648] HybridClock initialized: now 1750946428461638 us; error 35 us; skew 500 ppm
I20250626 14:00:28.462869 9171 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 14:00:28.485373 9171 webserver.cc:469] Webserver started at http://127.2.107.67:37217/ using document root <none> and password file <none>
I20250626 14:00:28.486704 9171 fs_manager.cc:362] Metadata directory not provided
I20250626 14:00:28.487017 9171 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 14:00:28.498307 9171 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.007s sys 0.000s
I20250626 14:00:28.504369 9192 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 14:00:28.505769 9171 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20250626 14:00:28.506183 9171 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-2/wal
uuid: "d062bd8ea8b54c90a06a01953c0a21e8"
format_stamp: "Formatted at 2025-06-26 14:00:11 on dist-test-slave-7p9x"
I20250626 14:00:28.508955 9171 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 14:00:28.581902 9171 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 14:00:28.583858 9171 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 14:00:28.584427 9171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 14:00:28.587733 9171 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 14:00:28.595621 9199 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250626 14:00:28.603329 9171 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250626 14:00:28.603669 9171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.002s sys 0.000s
I20250626 14:00:28.604004 9171 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250626 14:00:28.611405 9171 ts_tablet_manager.cc:610] Registered 1 tablets
I20250626 14:00:28.611682 9171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.008s user 0.006s sys 0.000s
I20250626 14:00:28.612046 9199 tablet_bootstrap.cc:492] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8: Bootstrap starting.
I20250626 14:00:28.811443 9171 rpc_server.cc:307] RPC server started. Bound to: 127.2.107.67:42909
I20250626 14:00:28.811511 9305 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.107.67:42909 every 8 connection(s)
I20250626 14:00:28.815541 9171 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-2/data/info.pb
I20250626 14:00:28.818079 2477 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu as pid 9171
I20250626 14:00:28.862882 9306 heartbeater.cc:344] Connected to a master server at 127.2.107.126:40413
I20250626 14:00:28.863431 9306 heartbeater.cc:461] Registering TS with master...
I20250626 14:00:28.864704 9306 heartbeater.cc:507] Master 127.2.107.126:40413 requested a full tablet report, sending...
I20250626 14:00:28.869953 8847 ts_manager.cc:194] Registered new tserver with Master: d062bd8ea8b54c90a06a01953c0a21e8 (127.2.107.67:42909)
I20250626 14:00:28.872535 2477 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 14:00:28.872903 8847 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.107.67:42567
I20250626 14:00:28.874972 9312 raft_consensus.cc:491] T 7f46a98c934042f98865dfbb088ddeea P 28dad11a06a14cc38086f0218694ac16 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:00:28.875733 9312 raft_consensus.cc:513] T 7f46a98c934042f98865dfbb088ddeea P 28dad11a06a14cc38086f0218694ac16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 42909 } } peers { permanent_uuid: "4f08b723abda4c519a9a04ac98418a8f" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 36747 } } peers { permanent_uuid: "28dad11a06a14cc38086f0218694ac16" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 39641 } }
I20250626 14:00:28.917328 9199 log.cc:826] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8: Log is configured to *not* fsync() on all Append() calls
I20250626 14:00:28.919587 9312 leader_election.cc:290] T 7f46a98c934042f98865dfbb088ddeea P 28dad11a06a14cc38086f0218694ac16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d062bd8ea8b54c90a06a01953c0a21e8 (127.2.107.67:42909), 4f08b723abda4c519a9a04ac98418a8f (127.2.107.66:36747)
I20250626 14:00:28.963586 9261 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f46a98c934042f98865dfbb088ddeea" candidate_uuid: "28dad11a06a14cc38086f0218694ac16" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" is_pre_election: true
I20250626 14:00:28.968037 9125 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f46a98c934042f98865dfbb088ddeea" candidate_uuid: "28dad11a06a14cc38086f0218694ac16" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "4f08b723abda4c519a9a04ac98418a8f" is_pre_election: true
W20250626 14:00:28.982913 8925 leader_election.cc:343] T 7f46a98c934042f98865dfbb088ddeea P 28dad11a06a14cc38086f0218694ac16 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d062bd8ea8b54c90a06a01953c0a21e8 (127.2.107.67:42909): Illegal state: must be running to vote when last-logged opid is not known
W20250626 14:00:28.984279 8923 leader_election.cc:343] T 7f46a98c934042f98865dfbb088ddeea P 28dad11a06a14cc38086f0218694ac16 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4f08b723abda4c519a9a04ac98418a8f (127.2.107.66:36747): Illegal state: must be running to vote when last-logged opid is not known
I20250626 14:00:28.984725 8923 leader_election.cc:304] T 7f46a98c934042f98865dfbb088ddeea P 28dad11a06a14cc38086f0218694ac16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28dad11a06a14cc38086f0218694ac16; no voters: 4f08b723abda4c519a9a04ac98418a8f, d062bd8ea8b54c90a06a01953c0a21e8
I20250626 14:00:28.986125 9312 raft_consensus.cc:2747] T 7f46a98c934042f98865dfbb088ddeea P 28dad11a06a14cc38086f0218694ac16 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 14:00:29.244196 9063 tablet_bootstrap.cc:492] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 14:00:29.244984 9063 tablet_bootstrap.cc:492] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f: Bootstrap complete.
I20250626 14:00:29.246119 9063 ts_tablet_manager.cc:1397] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f: Time spent bootstrapping tablet: real 2.645s user 2.507s sys 0.076s
I20250626 14:00:29.251171 9063 raft_consensus.cc:357] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 42909 } } peers { permanent_uuid: "4f08b723abda4c519a9a04ac98418a8f" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 36747 } } peers { permanent_uuid: "28dad11a06a14cc38086f0218694ac16" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 39641 } }
I20250626 14:00:29.253264 9063 raft_consensus.cc:738] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f08b723abda4c519a9a04ac98418a8f, State: Initialized, Role: FOLLOWER
I20250626 14:00:29.254442 9063 consensus_queue.cc:260] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 42909 } } peers { permanent_uuid: "4f08b723abda4c519a9a04ac98418a8f" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 36747 } } peers { permanent_uuid: "28dad11a06a14cc38086f0218694ac16" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 39641 } }
I20250626 14:00:29.257875 9063 ts_tablet_manager.cc:1428] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20250626 14:00:29.879463 9306 heartbeater.cc:499] Master 127.2.107.126:40413 was elected leader, sending a full tablet report...
I20250626 14:00:30.885249 9199 tablet_bootstrap.cc:492] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 14:00:30.886039 9199 tablet_bootstrap.cc:492] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8: Bootstrap complete.
I20250626 14:00:30.887902 9199 ts_tablet_manager.cc:1397] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8: Time spent bootstrapping tablet: real 2.276s user 2.190s sys 0.055s
I20250626 14:00:30.893174 9199 raft_consensus.cc:357] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 42909 } } peers { permanent_uuid: "4f08b723abda4c519a9a04ac98418a8f" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 36747 } } peers { permanent_uuid: "28dad11a06a14cc38086f0218694ac16" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 39641 } }
I20250626 14:00:30.895104 9199 raft_consensus.cc:738] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d062bd8ea8b54c90a06a01953c0a21e8, State: Initialized, Role: FOLLOWER
I20250626 14:00:30.895809 9199 consensus_queue.cc:260] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 42909 } } peers { permanent_uuid: "4f08b723abda4c519a9a04ac98418a8f" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 36747 } } peers { permanent_uuid: "28dad11a06a14cc38086f0218694ac16" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 39641 } }
I20250626 14:00:30.898182 9199 ts_tablet_manager.cc:1428] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8: Time spent starting tablet: real 0.010s user 0.008s sys 0.004s
I20250626 14:00:31.070889 9325 raft_consensus.cc:491] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:00:31.071375 9325 raft_consensus.cc:513] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 42909 } } peers { permanent_uuid: "4f08b723abda4c519a9a04ac98418a8f" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 36747 } } peers { permanent_uuid: "28dad11a06a14cc38086f0218694ac16" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 39641 } }
I20250626 14:00:31.073523 9325 leader_election.cc:290] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d062bd8ea8b54c90a06a01953c0a21e8 (127.2.107.67:42909), 28dad11a06a14cc38086f0218694ac16 (127.2.107.65:39641)
I20250626 14:00:31.085664 9261 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f46a98c934042f98865dfbb088ddeea" candidate_uuid: "4f08b723abda4c519a9a04ac98418a8f" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" is_pre_election: true
I20250626 14:00:31.086202 9261 raft_consensus.cc:2466] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f08b723abda4c519a9a04ac98418a8f in term 1.
I20250626 14:00:31.087203 8990 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f46a98c934042f98865dfbb088ddeea" candidate_uuid: "4f08b723abda4c519a9a04ac98418a8f" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "28dad11a06a14cc38086f0218694ac16" is_pre_election: true
I20250626 14:00:31.087244 9060 leader_election.cc:304] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f08b723abda4c519a9a04ac98418a8f, d062bd8ea8b54c90a06a01953c0a21e8; no voters:
I20250626 14:00:31.087755 8990 raft_consensus.cc:2466] T 7f46a98c934042f98865dfbb088ddeea P 28dad11a06a14cc38086f0218694ac16 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f08b723abda4c519a9a04ac98418a8f in term 1.
I20250626 14:00:31.087924 9325 raft_consensus.cc:2802] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 14:00:31.088248 9325 raft_consensus.cc:491] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:00:31.088583 9325 raft_consensus.cc:3058] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f [term 1 FOLLOWER]: Advancing to term 2
I20250626 14:00:31.095381 9325 raft_consensus.cc:513] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 42909 } } peers { permanent_uuid: "4f08b723abda4c519a9a04ac98418a8f" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 36747 } } peers { permanent_uuid: "28dad11a06a14cc38086f0218694ac16" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 39641 } }
I20250626 14:00:31.096791 9325 leader_election.cc:290] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f [CANDIDATE]: Term 2 election: Requested vote from peers d062bd8ea8b54c90a06a01953c0a21e8 (127.2.107.67:42909), 28dad11a06a14cc38086f0218694ac16 (127.2.107.65:39641)
I20250626 14:00:31.097764 9261 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f46a98c934042f98865dfbb088ddeea" candidate_uuid: "4f08b723abda4c519a9a04ac98418a8f" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "d062bd8ea8b54c90a06a01953c0a21e8"
I20250626 14:00:31.097829 8990 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f46a98c934042f98865dfbb088ddeea" candidate_uuid: "4f08b723abda4c519a9a04ac98418a8f" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "28dad11a06a14cc38086f0218694ac16"
I20250626 14:00:31.098176 9261 raft_consensus.cc:3058] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8 [term 1 FOLLOWER]: Advancing to term 2
I20250626 14:00:31.098311 8990 raft_consensus.cc:3058] T 7f46a98c934042f98865dfbb088ddeea P 28dad11a06a14cc38086f0218694ac16 [term 1 FOLLOWER]: Advancing to term 2
I20250626 14:00:31.104120 9261 raft_consensus.cc:2466] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f08b723abda4c519a9a04ac98418a8f in term 2.
I20250626 14:00:31.104529 8990 raft_consensus.cc:2466] T 7f46a98c934042f98865dfbb088ddeea P 28dad11a06a14cc38086f0218694ac16 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f08b723abda4c519a9a04ac98418a8f in term 2.
I20250626 14:00:31.104980 9060 leader_election.cc:304] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f08b723abda4c519a9a04ac98418a8f, d062bd8ea8b54c90a06a01953c0a21e8; no voters:
I20250626 14:00:31.105737 9325 raft_consensus.cc:2802] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f [term 2 FOLLOWER]: Leader election won for term 2
I20250626 14:00:31.107267 9325 raft_consensus.cc:695] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f [term 2 LEADER]: Becoming Leader. State: Replica: 4f08b723abda4c519a9a04ac98418a8f, State: Running, Role: LEADER
I20250626 14:00:31.108100 9325 consensus_queue.cc:237] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 42909 } } peers { permanent_uuid: "4f08b723abda4c519a9a04ac98418a8f" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 36747 } } peers { permanent_uuid: "28dad11a06a14cc38086f0218694ac16" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 39641 } }
I20250626 14:00:31.117966 8847 catalog_manager.cc:5582] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f reported cstate change: term changed from 0 to 2, leader changed from <none> to 4f08b723abda4c519a9a04ac98418a8f (127.2.107.66), VOTER 28dad11a06a14cc38086f0218694ac16 (127.2.107.65) added, VOTER 4f08b723abda4c519a9a04ac98418a8f (127.2.107.66) added, VOTER d062bd8ea8b54c90a06a01953c0a21e8 (127.2.107.67) added. New cstate: current_term: 2 leader_uuid: "4f08b723abda4c519a9a04ac98418a8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 42909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f08b723abda4c519a9a04ac98418a8f" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 36747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28dad11a06a14cc38086f0218694ac16" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 39641 } health_report { overall_health: UNKNOWN } } }
I20250626 14:00:31.537794 9261 raft_consensus.cc:1273] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8 [term 2 FOLLOWER]: Refusing update from remote peer 4f08b723abda4c519a9a04ac98418a8f: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250626 14:00:31.539081 9325 consensus_queue.cc:1035] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 42909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
I20250626 14:00:31.551797 8990 raft_consensus.cc:1273] T 7f46a98c934042f98865dfbb088ddeea P 28dad11a06a14cc38086f0218694ac16 [term 2 FOLLOWER]: Refusing update from remote peer 4f08b723abda4c519a9a04ac98418a8f: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250626 14:00:31.553184 9325 consensus_queue.cc:1035] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "28dad11a06a14cc38086f0218694ac16" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 39641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
I20250626 14:00:31.574312 9125 consensus_queue.cc:237] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 2.206, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 42909 } } peers { permanent_uuid: "4f08b723abda4c519a9a04ac98418a8f" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 36747 } }
I20250626 14:00:31.578466 9261 raft_consensus.cc:1273] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8 [term 2 FOLLOWER]: Refusing update from remote peer 4f08b723abda4c519a9a04ac98418a8f: Log matching property violated. Preceding OpId in replica: term: 2 index: 206. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
I20250626 14:00:31.579496 9325 consensus_queue.cc:1035] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 42909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.000s
I20250626 14:00:31.584619 9330 raft_consensus.cc:2953] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f [term 2 LEADER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 28dad11a06a14cc38086f0218694ac16 (127.2.107.65) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 42909 } } peers { permanent_uuid: "4f08b723abda4c519a9a04ac98418a8f" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 36747 } } }
I20250626 14:00:31.585975 9261 raft_consensus.cc:2953] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8 [term 2 FOLLOWER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 28dad11a06a14cc38086f0218694ac16 (127.2.107.65) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 42909 } } peers { permanent_uuid: "4f08b723abda4c519a9a04ac98418a8f" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 36747 } } }
I20250626 14:00:31.593776 8832 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 7f46a98c934042f98865dfbb088ddeea with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 14:00:31.597113 8846 catalog_manager.cc:5582] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f reported cstate change: config changed from index -1 to 207, VOTER 28dad11a06a14cc38086f0218694ac16 (127.2.107.65) evicted. New cstate: current_term: 2 leader_uuid: "4f08b723abda4c519a9a04ac98418a8f" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 42909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f08b723abda4c519a9a04ac98418a8f" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 36747 } health_report { overall_health: HEALTHY } } }
I20250626 14:00:31.608408 9125 consensus_queue.cc:237] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 2.207, Last appended by leader: 205, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "4f08b723abda4c519a9a04ac98418a8f" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 36747 } }
I20250626 14:00:31.610675 9330 raft_consensus.cc:2953] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f [term 2 LEADER]: Committing config change with OpId 2.208: config changed from index 207 to 208, VOTER d062bd8ea8b54c90a06a01953c0a21e8 (127.2.107.67) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "4f08b723abda4c519a9a04ac98418a8f" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 36747 } } }
I20250626 14:00:31.616108 8970 tablet_service.cc:1515] Processing DeleteTablet for tablet 7f46a98c934042f98865dfbb088ddeea with delete_type TABLET_DATA_TOMBSTONED (TS 28dad11a06a14cc38086f0218694ac16 not found in new config with opid_index 207) from {username='slave'} at 127.0.0.1:34222
I20250626 14:00:31.617089 8832 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 7f46a98c934042f98865dfbb088ddeea with cas_config_opid_index 207: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 14:00:31.618736 9347 tablet_replica.cc:331] stopping tablet replica
I20250626 14:00:31.619302 9347 raft_consensus.cc:2241] T 7f46a98c934042f98865dfbb088ddeea P 28dad11a06a14cc38086f0218694ac16 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 14:00:31.619732 9347 raft_consensus.cc:2270] T 7f46a98c934042f98865dfbb088ddeea P 28dad11a06a14cc38086f0218694ac16 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 14:00:31.620185 8846 catalog_manager.cc:5582] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f reported cstate change: config changed from index 207 to 208, VOTER d062bd8ea8b54c90a06a01953c0a21e8 (127.2.107.67) evicted. New cstate: current_term: 2 leader_uuid: "4f08b723abda4c519a9a04ac98418a8f" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "4f08b723abda4c519a9a04ac98418a8f" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 36747 } health_report { overall_health: HEALTHY } } }
I20250626 14:00:31.642608 9347 ts_tablet_manager.cc:1905] T 7f46a98c934042f98865dfbb088ddeea P 28dad11a06a14cc38086f0218694ac16: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 14:00:31.653927 9347 ts_tablet_manager.cc:1918] T 7f46a98c934042f98865dfbb088ddeea P 28dad11a06a14cc38086f0218694ac16: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.206
I20250626 14:00:31.654330 9347 log.cc:1199] T 7f46a98c934042f98865dfbb088ddeea P 28dad11a06a14cc38086f0218694ac16: Deleting WAL directory at /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-0/wal/wals/7f46a98c934042f98865dfbb088ddeea
I20250626 14:00:31.652858 9241 tablet_service.cc:1515] Processing DeleteTablet for tablet 7f46a98c934042f98865dfbb088ddeea with delete_type TABLET_DATA_TOMBSTONED (TS d062bd8ea8b54c90a06a01953c0a21e8 not found in new config with opid_index 208) from {username='slave'} at 127.0.0.1:43426
I20250626 14:00:31.656085 8833 catalog_manager.cc:4928] TS 28dad11a06a14cc38086f0218694ac16 (127.2.107.65:39641): tablet 7f46a98c934042f98865dfbb088ddeea (table pre_rebuild [id=7a4235e9e55a4377a0c744fe48eed350]) successfully deleted
I20250626 14:00:31.658136 9350 tablet_replica.cc:331] stopping tablet replica
I20250626 14:00:31.658720 9350 raft_consensus.cc:2241] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 14:00:31.659184 9350 raft_consensus.cc:2270] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 14:00:31.679464 9350 ts_tablet_manager.cc:1905] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 14:00:31.689735 9350 ts_tablet_manager.cc:1918] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.207
I20250626 14:00:31.690034 9350 log.cc:1199] T 7f46a98c934042f98865dfbb088ddeea P d062bd8ea8b54c90a06a01953c0a21e8: Deleting WAL directory at /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-2/wal/wals/7f46a98c934042f98865dfbb088ddeea
I20250626 14:00:31.691643 8834 catalog_manager.cc:4928] TS d062bd8ea8b54c90a06a01953c0a21e8 (127.2.107.67:42909): tablet 7f46a98c934042f98865dfbb088ddeea (table pre_rebuild [id=7a4235e9e55a4377a0c744fe48eed350]) successfully deleted
W20250626 14:00:32.043516 2477 scanner-internal.cc:458] Time spent opening tablet: real 3.032s user 0.008s sys 0.002s
I20250626 14:00:32.608359 8970 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 14:00:32.608481 9105 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250626 14:00:32.618980 9241 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
538dac2d162c4a62a2ca9fdd3ae02f38 | 127.2.107.126:40413 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.2.107.84:46535 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
28dad11a06a14cc38086f0218694ac16 | 127.2.107.65:39641 | HEALTHY | <none> | 0 | 0
4f08b723abda4c519a9a04ac98418a8f | 127.2.107.66:36747 | HEALTHY | <none> | 1 | 0
d062bd8ea8b54c90a06a01953c0a21e8 | 127.2.107.67:42909 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.2.107.65 | experimental | 127.2.107.65:39641
local_ip_for_outbound_sockets | 127.2.107.66 | experimental | 127.2.107.66:36747
local_ip_for_outbound_sockets | 127.2.107.67 | experimental | 127.2.107.67:42909
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-0/data/info.pb | hidden | 127.2.107.65:39641
server_dump_info_path | /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-1/data/info.pb | hidden | 127.2.107.66:36747
server_dump_info_path | /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-2/data/info.pb | hidden | 127.2.107.67:42909
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.2.107.84:46535 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------+----+---------+---------------+---------+------------+------------------+-------------
pre_rebuild | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 0
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 1
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250626 14:00:32.906608 2477 log_verifier.cc:126] Checking tablet 7f46a98c934042f98865dfbb088ddeea
I20250626 14:00:33.184644 2477 log_verifier.cc:177] Verified matching terms for 208 ops in tablet 7f46a98c934042f98865dfbb088ddeea
I20250626 14:00:33.187148 8847 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43368:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250626 14:00:33.187759 8847 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43368:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250626 14:00:33.199172 8847 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 538dac2d162c4a62a2ca9fdd3ae02f38: Sending DeleteTablet for 1 replicas of tablet 7f46a98c934042f98865dfbb088ddeea
I20250626 14:00:33.201157 2477 test_util.cc:276] Using random seed: -64228395
I20250626 14:00:33.201120 9105 tablet_service.cc:1515] Processing DeleteTablet for tablet 7f46a98c934042f98865dfbb088ddeea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 14:00:33 UTC) from {username='slave'} at 127.0.0.1:43796
I20250626 14:00:33.205811 9385 tablet_replica.cc:331] stopping tablet replica
I20250626 14:00:33.207564 9385 raft_consensus.cc:2241] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f [term 2 LEADER]: Raft consensus shutting down.
I20250626 14:00:33.208186 9385 raft_consensus.cc:2270] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 14:00:33.240027 8847 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43406:
name: "post_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 14:00:33.242610 9385 ts_tablet_manager.cc:1905] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250626 14:00:33.243463 8847 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table post_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 14:00:33.254388 9385 ts_tablet_manager.cc:1918] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.208
I20250626 14:00:33.254951 9385 log.cc:1199] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f: Deleting WAL directory at /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-1/wal/wals/7f46a98c934042f98865dfbb088ddeea
I20250626 14:00:33.256032 9385 ts_tablet_manager.cc:1939] T 7f46a98c934042f98865dfbb088ddeea P 4f08b723abda4c519a9a04ac98418a8f: Deleting consensus metadata
I20250626 14:00:33.259111 8832 catalog_manager.cc:4928] TS 4f08b723abda4c519a9a04ac98418a8f (127.2.107.66:36747): tablet 7f46a98c934042f98865dfbb088ddeea (table pre_rebuild [id=7a4235e9e55a4377a0c744fe48eed350]) successfully deleted
I20250626 14:00:33.270829 9241 tablet_service.cc:1468] Processing CreateTablet for tablet 05f84e2137da47809452200c32c3582b (DEFAULT_TABLE table=post_rebuild [id=1ad2d22a82cd4eed846d8715a2463b0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:00:33.271446 8970 tablet_service.cc:1468] Processing CreateTablet for tablet 05f84e2137da47809452200c32c3582b (DEFAULT_TABLE table=post_rebuild [id=1ad2d22a82cd4eed846d8715a2463b0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:00:33.272178 9241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05f84e2137da47809452200c32c3582b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:00:33.272678 8970 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05f84e2137da47809452200c32c3582b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:00:33.272583 9105 tablet_service.cc:1468] Processing CreateTablet for tablet 05f84e2137da47809452200c32c3582b (DEFAULT_TABLE table=post_rebuild [id=1ad2d22a82cd4eed846d8715a2463b0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 14:00:33.273816 9105 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05f84e2137da47809452200c32c3582b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 14:00:33.293756 9392 tablet_bootstrap.cc:492] T 05f84e2137da47809452200c32c3582b P d062bd8ea8b54c90a06a01953c0a21e8: Bootstrap starting.
I20250626 14:00:33.294492 9394 tablet_bootstrap.cc:492] T 05f84e2137da47809452200c32c3582b P 4f08b723abda4c519a9a04ac98418a8f: Bootstrap starting.
I20250626 14:00:33.300544 9392 tablet_bootstrap.cc:654] T 05f84e2137da47809452200c32c3582b P d062bd8ea8b54c90a06a01953c0a21e8: Neither blocks nor log segments found. Creating new log.
I20250626 14:00:33.300663 9394 tablet_bootstrap.cc:654] T 05f84e2137da47809452200c32c3582b P 4f08b723abda4c519a9a04ac98418a8f: Neither blocks nor log segments found. Creating new log.
I20250626 14:00:33.301826 9393 tablet_bootstrap.cc:492] T 05f84e2137da47809452200c32c3582b P 28dad11a06a14cc38086f0218694ac16: Bootstrap starting.
I20250626 14:00:33.308408 9393 tablet_bootstrap.cc:654] T 05f84e2137da47809452200c32c3582b P 28dad11a06a14cc38086f0218694ac16: Neither blocks nor log segments found. Creating new log.
I20250626 14:00:33.312158 9394 tablet_bootstrap.cc:492] T 05f84e2137da47809452200c32c3582b P 4f08b723abda4c519a9a04ac98418a8f: No bootstrap required, opened a new log
I20250626 14:00:33.312639 9394 ts_tablet_manager.cc:1397] T 05f84e2137da47809452200c32c3582b P 4f08b723abda4c519a9a04ac98418a8f: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.004s
I20250626 14:00:33.313367 9392 tablet_bootstrap.cc:492] T 05f84e2137da47809452200c32c3582b P d062bd8ea8b54c90a06a01953c0a21e8: No bootstrap required, opened a new log
I20250626 14:00:33.313845 9392 ts_tablet_manager.cc:1397] T 05f84e2137da47809452200c32c3582b P d062bd8ea8b54c90a06a01953c0a21e8: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.007s
I20250626 14:00:33.314122 9393 tablet_bootstrap.cc:492] T 05f84e2137da47809452200c32c3582b P 28dad11a06a14cc38086f0218694ac16: No bootstrap required, opened a new log
I20250626 14:00:33.314607 9393 ts_tablet_manager.cc:1397] T 05f84e2137da47809452200c32c3582b P 28dad11a06a14cc38086f0218694ac16: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20250626 14:00:33.315270 9394 raft_consensus.cc:357] T 05f84e2137da47809452200c32c3582b P 4f08b723abda4c519a9a04ac98418a8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 42909 } } peers { permanent_uuid: "28dad11a06a14cc38086f0218694ac16" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 39641 } } peers { permanent_uuid: "4f08b723abda4c519a9a04ac98418a8f" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 36747 } }
I20250626 14:00:33.316021 9394 raft_consensus.cc:383] T 05f84e2137da47809452200c32c3582b P 4f08b723abda4c519a9a04ac98418a8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:00:33.316349 9394 raft_consensus.cc:738] T 05f84e2137da47809452200c32c3582b P 4f08b723abda4c519a9a04ac98418a8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f08b723abda4c519a9a04ac98418a8f, State: Initialized, Role: FOLLOWER
I20250626 14:00:33.316687 9392 raft_consensus.cc:357] T 05f84e2137da47809452200c32c3582b P d062bd8ea8b54c90a06a01953c0a21e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 42909 } } peers { permanent_uuid: "28dad11a06a14cc38086f0218694ac16" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 39641 } } peers { permanent_uuid: "4f08b723abda4c519a9a04ac98418a8f" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 36747 } }
I20250626 14:00:33.317126 9393 raft_consensus.cc:357] T 05f84e2137da47809452200c32c3582b P 28dad11a06a14cc38086f0218694ac16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 42909 } } peers { permanent_uuid: "28dad11a06a14cc38086f0218694ac16" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 39641 } } peers { permanent_uuid: "4f08b723abda4c519a9a04ac98418a8f" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 36747 } }
I20250626 14:00:33.317538 9392 raft_consensus.cc:383] T 05f84e2137da47809452200c32c3582b P d062bd8ea8b54c90a06a01953c0a21e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:00:33.317198 9394 consensus_queue.cc:260] T 05f84e2137da47809452200c32c3582b P 4f08b723abda4c519a9a04ac98418a8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 42909 } } peers { permanent_uuid: "28dad11a06a14cc38086f0218694ac16" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 39641 } } peers { permanent_uuid: "4f08b723abda4c519a9a04ac98418a8f" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 36747 } }
I20250626 14:00:33.317823 9393 raft_consensus.cc:383] T 05f84e2137da47809452200c32c3582b P 28dad11a06a14cc38086f0218694ac16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 14:00:33.317936 9392 raft_consensus.cc:738] T 05f84e2137da47809452200c32c3582b P d062bd8ea8b54c90a06a01953c0a21e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d062bd8ea8b54c90a06a01953c0a21e8, State: Initialized, Role: FOLLOWER
I20250626 14:00:33.318109 9393 raft_consensus.cc:738] T 05f84e2137da47809452200c32c3582b P 28dad11a06a14cc38086f0218694ac16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28dad11a06a14cc38086f0218694ac16, State: Initialized, Role: FOLLOWER
I20250626 14:00:33.318756 9393 consensus_queue.cc:260] T 05f84e2137da47809452200c32c3582b P 28dad11a06a14cc38086f0218694ac16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 42909 } } peers { permanent_uuid: "28dad11a06a14cc38086f0218694ac16" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 39641 } } peers { permanent_uuid: "4f08b723abda4c519a9a04ac98418a8f" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 36747 } }
I20250626 14:00:33.318744 9392 consensus_queue.cc:260] T 05f84e2137da47809452200c32c3582b P d062bd8ea8b54c90a06a01953c0a21e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 42909 } } peers { permanent_uuid: "28dad11a06a14cc38086f0218694ac16" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 39641 } } peers { permanent_uuid: "4f08b723abda4c519a9a04ac98418a8f" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 36747 } }
I20250626 14:00:33.324661 9393 ts_tablet_manager.cc:1428] T 05f84e2137da47809452200c32c3582b P 28dad11a06a14cc38086f0218694ac16: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20250626 14:00:33.328069 9392 ts_tablet_manager.cc:1428] T 05f84e2137da47809452200c32c3582b P d062bd8ea8b54c90a06a01953c0a21e8: Time spent starting tablet: real 0.014s user 0.005s sys 0.006s
I20250626 14:00:33.330737 9394 ts_tablet_manager.cc:1428] T 05f84e2137da47809452200c32c3582b P 4f08b723abda4c519a9a04ac98418a8f: Time spent starting tablet: real 0.018s user 0.008s sys 0.004s
W20250626 14:00:33.355206 9172 tablet.cc:2378] T 05f84e2137da47809452200c32c3582b P 4f08b723abda4c519a9a04ac98418a8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250626 14:00:33.356606 9309 tablet.cc:2378] T 05f84e2137da47809452200c32c3582b P d062bd8ea8b54c90a06a01953c0a21e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250626 14:00:33.418730 9036 tablet.cc:2378] T 05f84e2137da47809452200c32c3582b P 28dad11a06a14cc38086f0218694ac16: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 14:00:33.543797 9400 raft_consensus.cc:491] T 05f84e2137da47809452200c32c3582b P d062bd8ea8b54c90a06a01953c0a21e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 14:00:33.544201 9400 raft_consensus.cc:513] T 05f84e2137da47809452200c32c3582b P d062bd8ea8b54c90a06a01953c0a21e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 42909 } } peers { permanent_uuid: "28dad11a06a14cc38086f0218694ac16" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 39641 } } peers { permanent_uuid: "4f08b723abda4c519a9a04ac98418a8f" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 36747 } }
I20250626 14:00:33.546679 9400 leader_election.cc:290] T 05f84e2137da47809452200c32c3582b P d062bd8ea8b54c90a06a01953c0a21e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28dad11a06a14cc38086f0218694ac16 (127.2.107.65:39641), 4f08b723abda4c519a9a04ac98418a8f (127.2.107.66:36747)
I20250626 14:00:33.562429 8990 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05f84e2137da47809452200c32c3582b" candidate_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28dad11a06a14cc38086f0218694ac16" is_pre_election: true
I20250626 14:00:33.562850 9125 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05f84e2137da47809452200c32c3582b" candidate_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f08b723abda4c519a9a04ac98418a8f" is_pre_election: true
I20250626 14:00:33.563316 8990 raft_consensus.cc:2466] T 05f84e2137da47809452200c32c3582b P 28dad11a06a14cc38086f0218694ac16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d062bd8ea8b54c90a06a01953c0a21e8 in term 0.
I20250626 14:00:33.563446 9125 raft_consensus.cc:2466] T 05f84e2137da47809452200c32c3582b P 4f08b723abda4c519a9a04ac98418a8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d062bd8ea8b54c90a06a01953c0a21e8 in term 0.
I20250626 14:00:33.564800 9195 leader_election.cc:304] T 05f84e2137da47809452200c32c3582b P d062bd8ea8b54c90a06a01953c0a21e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28dad11a06a14cc38086f0218694ac16, d062bd8ea8b54c90a06a01953c0a21e8; no voters:
I20250626 14:00:33.565572 9400 raft_consensus.cc:2802] T 05f84e2137da47809452200c32c3582b P d062bd8ea8b54c90a06a01953c0a21e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 14:00:33.565914 9400 raft_consensus.cc:491] T 05f84e2137da47809452200c32c3582b P d062bd8ea8b54c90a06a01953c0a21e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 14:00:33.566231 9400 raft_consensus.cc:3058] T 05f84e2137da47809452200c32c3582b P d062bd8ea8b54c90a06a01953c0a21e8 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:00:33.573119 9400 raft_consensus.cc:513] T 05f84e2137da47809452200c32c3582b P d062bd8ea8b54c90a06a01953c0a21e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 42909 } } peers { permanent_uuid: "28dad11a06a14cc38086f0218694ac16" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 39641 } } peers { permanent_uuid: "4f08b723abda4c519a9a04ac98418a8f" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 36747 } }
I20250626 14:00:33.575124 9400 leader_election.cc:290] T 05f84e2137da47809452200c32c3582b P d062bd8ea8b54c90a06a01953c0a21e8 [CANDIDATE]: Term 1 election: Requested vote from peers 28dad11a06a14cc38086f0218694ac16 (127.2.107.65:39641), 4f08b723abda4c519a9a04ac98418a8f (127.2.107.66:36747)
I20250626 14:00:33.575994 8990 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05f84e2137da47809452200c32c3582b" candidate_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28dad11a06a14cc38086f0218694ac16"
I20250626 14:00:33.576169 9125 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05f84e2137da47809452200c32c3582b" candidate_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f08b723abda4c519a9a04ac98418a8f"
I20250626 14:00:33.576413 8990 raft_consensus.cc:3058] T 05f84e2137da47809452200c32c3582b P 28dad11a06a14cc38086f0218694ac16 [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:00:33.576706 9125 raft_consensus.cc:3058] T 05f84e2137da47809452200c32c3582b P 4f08b723abda4c519a9a04ac98418a8f [term 0 FOLLOWER]: Advancing to term 1
I20250626 14:00:33.581351 8990 raft_consensus.cc:2466] T 05f84e2137da47809452200c32c3582b P 28dad11a06a14cc38086f0218694ac16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d062bd8ea8b54c90a06a01953c0a21e8 in term 1.
I20250626 14:00:33.582510 9195 leader_election.cc:304] T 05f84e2137da47809452200c32c3582b P d062bd8ea8b54c90a06a01953c0a21e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28dad11a06a14cc38086f0218694ac16, d062bd8ea8b54c90a06a01953c0a21e8; no voters:
I20250626 14:00:33.583161 9400 raft_consensus.cc:2802] T 05f84e2137da47809452200c32c3582b P d062bd8ea8b54c90a06a01953c0a21e8 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 14:00:33.583279 9125 raft_consensus.cc:2466] T 05f84e2137da47809452200c32c3582b P 4f08b723abda4c519a9a04ac98418a8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d062bd8ea8b54c90a06a01953c0a21e8 in term 1.
I20250626 14:00:33.585507 9400 raft_consensus.cc:695] T 05f84e2137da47809452200c32c3582b P d062bd8ea8b54c90a06a01953c0a21e8 [term 1 LEADER]: Becoming Leader. State: Replica: d062bd8ea8b54c90a06a01953c0a21e8, State: Running, Role: LEADER
I20250626 14:00:33.586519 9400 consensus_queue.cc:237] T 05f84e2137da47809452200c32c3582b P d062bd8ea8b54c90a06a01953c0a21e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 42909 } } peers { permanent_uuid: "28dad11a06a14cc38086f0218694ac16" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 39641 } } peers { permanent_uuid: "4f08b723abda4c519a9a04ac98418a8f" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 36747 } }
I20250626 14:00:33.595335 8846 catalog_manager.cc:5582] T 05f84e2137da47809452200c32c3582b P d062bd8ea8b54c90a06a01953c0a21e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to d062bd8ea8b54c90a06a01953c0a21e8 (127.2.107.67). New cstate: current_term: 1 leader_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d062bd8ea8b54c90a06a01953c0a21e8" member_type: VOTER last_known_addr { host: "127.2.107.67" port: 42909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28dad11a06a14cc38086f0218694ac16" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 39641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f08b723abda4c519a9a04ac98418a8f" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 36747 } health_report { overall_health: UNKNOWN } } }
I20250626 14:00:33.754237 8990 raft_consensus.cc:1273] T 05f84e2137da47809452200c32c3582b P 28dad11a06a14cc38086f0218694ac16 [term 1 FOLLOWER]: Refusing update from remote peer d062bd8ea8b54c90a06a01953c0a21e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 14:00:33.754484 9125 raft_consensus.cc:1273] T 05f84e2137da47809452200c32c3582b P 4f08b723abda4c519a9a04ac98418a8f [term 1 FOLLOWER]: Refusing update from remote peer d062bd8ea8b54c90a06a01953c0a21e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 14:00:33.755815 9400 consensus_queue.cc:1035] T 05f84e2137da47809452200c32c3582b P d062bd8ea8b54c90a06a01953c0a21e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28dad11a06a14cc38086f0218694ac16" member_type: VOTER last_known_addr { host: "127.2.107.65" port: 39641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 14:00:33.756681 9406 consensus_queue.cc:1035] T 05f84e2137da47809452200c32c3582b P d062bd8ea8b54c90a06a01953c0a21e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f08b723abda4c519a9a04ac98418a8f" member_type: VOTER last_known_addr { host: "127.2.107.66" port: 36747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 14:00:33.800413 9414 mvcc.cc:204] Tried to move back new op lower bound from 7171876592640606208 to 7171876591979417600. Current Snapshot: MvccSnapshot[applied={T|T < 7171876592640606208}]
W20250626 14:00:34.957626 9409 meta_cache.cc:1261] Time spent looking up entry by key: real 0.051s user 0.001s sys 0.000s
I20250626 14:00:38.752429 9241 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250626 14:00:38.752754 8970 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 14:00:38.754657 9105 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
538dac2d162c4a62a2ca9fdd3ae02f38 | 127.2.107.126:40413 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.2.107.84:46535 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
28dad11a06a14cc38086f0218694ac16 | 127.2.107.65:39641 | HEALTHY | <none> | 0 | 0
4f08b723abda4c519a9a04ac98418a8f | 127.2.107.66:36747 | HEALTHY | <none> | 0 | 0
d062bd8ea8b54c90a06a01953c0a21e8 | 127.2.107.67:42909 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.2.107.65 | experimental | 127.2.107.65:39641
local_ip_for_outbound_sockets | 127.2.107.66 | experimental | 127.2.107.66:36747
local_ip_for_outbound_sockets | 127.2.107.67 | experimental | 127.2.107.67:42909
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-0/data/info.pb | hidden | 127.2.107.65:39641
server_dump_info_path | /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-1/data/info.pb | hidden | 127.2.107.66:36747
server_dump_info_path | /tmp/dist-test-taskTkoGgi/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750946263504804-2477-0/minicluster-data/ts-2/data/info.pb | hidden | 127.2.107.67:42909
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.2.107.84:46535 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
--------------+----+---------+---------------+---------+------------+------------------+-------------
post_rebuild | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250626 14:00:39.033869 2477 log_verifier.cc:126] Checking tablet 05f84e2137da47809452200c32c3582b
I20250626 14:00:39.767927 2477 log_verifier.cc:177] Verified matching terms for 205 ops in tablet 05f84e2137da47809452200c32c3582b
I20250626 14:00:39.768904 2477 log_verifier.cc:126] Checking tablet 7f46a98c934042f98865dfbb088ddeea
I20250626 14:00:39.769220 2477 log_verifier.cc:177] Verified matching terms for 0 ops in tablet 7f46a98c934042f98865dfbb088ddeea
I20250626 14:00:39.793922 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 8884
I20250626 14:00:39.827059 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 9039
W20250626 14:00:39.859778 9194 connection.cc:537] client connection to 127.2.107.66:36747 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250626 14:00:39.860226 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 9171
W20250626 14:00:39.860200 9194 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250626 14:00:39.894460 2477 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskTkoGgi/build/tsan/bin/kudu with pid 8814
2025-06-26T14:00:39Z chronyd exiting
[ OK ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0 (34971 ms)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest (34971 ms total)
[----------] Global test environment tear-down
[==========] 9 tests from 5 test suites ran. (176376 ms total)
[ PASSED ] 8 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AdminCliTest.TestRebuildTables
1 FAILED TEST