Note: This is test shard 6 of 8.
[==========] Running 9 tests from 5 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from AdminCliTest
[ RUN ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250814 13:56:59.754186 11794 test_util.cc:276] Using random seed: -1515429304
W20250814 13:57:00.923910 11794 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.126s user 0.413s sys 0.710s
W20250814 13:57:00.924222 11794 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.126s user 0.413s sys 0.710s
I20250814 13:57:00.937697 11794 ts_itest-base.cc:115] Starting cluster with:
I20250814 13:57:00.937904 11794 ts_itest-base.cc:116] --------------
I20250814 13:57:00.938074 11794 ts_itest-base.cc:117] 4 tablet servers
I20250814 13:57:00.938247 11794 ts_itest-base.cc:118] 3 replicas per TS
I20250814 13:57:00.938396 11794 ts_itest-base.cc:119] --------------
2025-08-14T13:57:00Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-14T13:57:00Z Disabled control of system clock
I20250814 13:57:00.974872 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.190:43609
--webserver_interface=127.11.132.190
--webserver_port=0
--builtin_ntp_servers=127.11.132.148:40333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.132.190:43609 with env {}
W20250814 13:57:01.259913 11808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:57:01.260438 11808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:57:01.260828 11808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:57:01.289497 11808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250814 13:57:01.289760 11808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:57:01.289943 11808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250814 13:57:01.290117 11808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250814 13:57:01.322335 11808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:40333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.132.190:43609
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.190:43609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.11.132.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:57:01.323555 11808 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:57:01.325028 11808 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:57:01.334903 11815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:57:02.738569 11813 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 11808
W20250814 13:57:03.050591 11808 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.716s user 0.622s sys 1.093s
W20250814 13:57:01.334896 11814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:57:03.053097 11808 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.718s user 0.622s sys 1.093s
W20250814 13:57:03.053272 11817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 13:57:03.053714 11808 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250814 13:57:03.053820 11816 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1716 milliseconds
I20250814 13:57:03.058140 11808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:57:03.060436 11808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:57:03.061744 11808 hybrid_clock.cc:648] HybridClock initialized: now 1755179823061719 us; error 35 us; skew 500 ppm
I20250814 13:57:03.062474 11808 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:57:03.068059 11808 webserver.cc:480] Webserver started at http://127.11.132.190:46041/ using document root <none> and password file <none>
I20250814 13:57:03.068894 11808 fs_manager.cc:362] Metadata directory not provided
I20250814 13:57:03.069115 11808 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:57:03.069511 11808 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 13:57:03.073650 11808 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "90cf1783ebdd4afe8f5b95f54d7665a8"
format_stamp: "Formatted at 2025-08-14 13:57:03 on dist-test-slave-q1tc"
I20250814 13:57:03.074633 11808 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "90cf1783ebdd4afe8f5b95f54d7665a8"
format_stamp: "Formatted at 2025-08-14 13:57:03 on dist-test-slave-q1tc"
I20250814 13:57:03.081233 11808 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.006s
I20250814 13:57:03.086308 11824 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:57:03.087155 11808 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250814 13:57:03.087458 11808 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
uuid: "90cf1783ebdd4afe8f5b95f54d7665a8"
format_stamp: "Formatted at 2025-08-14 13:57:03 on dist-test-slave-q1tc"
I20250814 13:57:03.087772 11808 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:57:03.145198 11808 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:57:03.146570 11808 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:57:03.146988 11808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:57:03.212770 11808 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.190:43609
I20250814 13:57:03.212836 11875 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.190:43609 every 8 connection(s)
I20250814 13:57:03.215363 11808 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250814 13:57:03.218920 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 11808
I20250814 13:57:03.219482 11794 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250814 13:57:03.221244 11876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:57:03.240540 11876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8: Bootstrap starting.
I20250814 13:57:03.247263 11876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8: Neither blocks nor log segments found. Creating new log.
I20250814 13:57:03.249464 11876 log.cc:826] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8: Log is configured to *not* fsync() on all Append() calls
I20250814 13:57:03.254329 11876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8: No bootstrap required, opened a new log
I20250814 13:57:03.271517 11876 raft_consensus.cc:357] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90cf1783ebdd4afe8f5b95f54d7665a8" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 43609 } }
I20250814 13:57:03.272095 11876 raft_consensus.cc:383] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 13:57:03.272289 11876 raft_consensus.cc:738] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90cf1783ebdd4afe8f5b95f54d7665a8, State: Initialized, Role: FOLLOWER
I20250814 13:57:03.272913 11876 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90cf1783ebdd4afe8f5b95f54d7665a8" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 43609 } }
I20250814 13:57:03.273428 11876 raft_consensus.cc:397] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 13:57:03.273681 11876 raft_consensus.cc:491] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 13:57:03.273985 11876 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8 [term 0 FOLLOWER]: Advancing to term 1
I20250814 13:57:03.278533 11876 raft_consensus.cc:513] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90cf1783ebdd4afe8f5b95f54d7665a8" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 43609 } }
I20250814 13:57:03.279136 11876 leader_election.cc:304] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90cf1783ebdd4afe8f5b95f54d7665a8; no voters:
I20250814 13:57:03.280694 11876 leader_election.cc:290] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250814 13:57:03.281316 11881 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8 [term 1 FOLLOWER]: Leader election won for term 1
I20250814 13:57:03.283488 11881 raft_consensus.cc:695] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8 [term 1 LEADER]: Becoming Leader. State: Replica: 90cf1783ebdd4afe8f5b95f54d7665a8, State: Running, Role: LEADER
I20250814 13:57:03.284402 11881 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90cf1783ebdd4afe8f5b95f54d7665a8" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 43609 } }
I20250814 13:57:03.286144 11876 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8 [sys.catalog]: configured and running, proceeding with master startup.
I20250814 13:57:03.293941 11883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90cf1783ebdd4afe8f5b95f54d7665a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90cf1783ebdd4afe8f5b95f54d7665a8" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 43609 } } }
I20250814 13:57:03.294148 11882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90cf1783ebdd4afe8f5b95f54d7665a8. Latest consensus state: current_term: 1 leader_uuid: "90cf1783ebdd4afe8f5b95f54d7665a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90cf1783ebdd4afe8f5b95f54d7665a8" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 43609 } } }
I20250814 13:57:03.294576 11883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8 [sys.catalog]: This master's current role is: LEADER
I20250814 13:57:03.294621 11882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8 [sys.catalog]: This master's current role is: LEADER
I20250814 13:57:03.297431 11888 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250814 13:57:03.308743 11888 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250814 13:57:03.324600 11888 catalog_manager.cc:1349] Generated new cluster ID: 1a5c953b8fc440278b2f62080c871a92
I20250814 13:57:03.324887 11888 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250814 13:57:03.344128 11888 catalog_manager.cc:1372] Generated new certificate authority record
I20250814 13:57:03.346057 11888 catalog_manager.cc:1506] Loading token signing keys...
I20250814 13:57:03.363888 11888 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8: Generated new TSK 0
I20250814 13:57:03.364856 11888 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250814 13:57:03.375873 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.132.129:0
--local_ip_for_outbound_sockets=127.11.132.129
--webserver_interface=127.11.132.129
--webserver_port=0
--tserver_master_addrs=127.11.132.190:43609
--builtin_ntp_servers=127.11.132.148:40333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250814 13:57:03.660012 11900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:57:03.660476 11900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:57:03.660920 11900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:57:03.690640 11900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:57:03.691442 11900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.132.129
I20250814 13:57:03.723757 11900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:40333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.132.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.11.132.129
--webserver_port=0
--tserver_master_addrs=127.11.132.190:43609
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.132.129
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:57:03.725056 11900 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:57:03.726572 11900 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:57:03.738081 11906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:57:03.740954 11907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:57:04.902990 11909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:57:04.905836 11908 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1161 milliseconds
I20250814 13:57:04.905932 11900 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 13:57:04.907258 11900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:57:04.909974 11900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:57:04.911435 11900 hybrid_clock.cc:648] HybridClock initialized: now 1755179824911381 us; error 40 us; skew 500 ppm
I20250814 13:57:04.912456 11900 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:57:04.919705 11900 webserver.cc:480] Webserver started at http://127.11.132.129:36983/ using document root <none> and password file <none>
I20250814 13:57:04.920908 11900 fs_manager.cc:362] Metadata directory not provided
I20250814 13:57:04.921185 11900 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:57:04.921742 11900 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 13:57:04.928141 11900 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "071c488311ff42dc9d7dbed1bd39ea9f"
format_stamp: "Formatted at 2025-08-14 13:57:04 on dist-test-slave-q1tc"
I20250814 13:57:04.929582 11900 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "071c488311ff42dc9d7dbed1bd39ea9f"
format_stamp: "Formatted at 2025-08-14 13:57:04 on dist-test-slave-q1tc"
W20250814 13:57:04.934083 11872 debug-util.cc:398] Leaking SignalData structure 0x7b080006f060 after lost signal to thread 11809
W20250814 13:57:04.934546 11872 debug-util.cc:398] Leaking SignalData structure 0x7b080006f7a0 after lost signal to thread 11875
I20250814 13:57:04.938755 11900 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20250814 13:57:04.947335 11916 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:57:04.948468 11900 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20250814 13:57:04.948869 11900 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "071c488311ff42dc9d7dbed1bd39ea9f"
format_stamp: "Formatted at 2025-08-14 13:57:04 on dist-test-slave-q1tc"
I20250814 13:57:04.949339 11900 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:57:05.028128 11900 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:57:05.029974 11900 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:57:05.030580 11900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:57:05.033892 11900 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 13:57:05.039513 11900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 13:57:05.039839 11900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250814 13:57:05.040191 11900 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 13:57:05.040431 11900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 13:57:05.207450 11900 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.129:41199
I20250814 13:57:05.207847 12028 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.129:41199 every 8 connection(s)
I20250814 13:57:05.210850 11900 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250814 13:57:05.218530 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 11900
I20250814 13:57:05.219069 11794 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250814 13:57:05.229223 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.132.130:0
--local_ip_for_outbound_sockets=127.11.132.130
--webserver_interface=127.11.132.130
--webserver_port=0
--tserver_master_addrs=127.11.132.190:43609
--builtin_ntp_servers=127.11.132.148:40333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250814 13:57:05.270704 12029 heartbeater.cc:344] Connected to a master server at 127.11.132.190:43609
I20250814 13:57:05.271204 12029 heartbeater.cc:461] Registering TS with master...
I20250814 13:57:05.272485 12029 heartbeater.cc:507] Master 127.11.132.190:43609 requested a full tablet report, sending...
I20250814 13:57:05.275745 11841 ts_manager.cc:194] Registered new tserver with Master: 071c488311ff42dc9d7dbed1bd39ea9f (127.11.132.129:41199)
I20250814 13:57:05.278393 11841 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.132.129:54465
W20250814 13:57:05.675729 12033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:57:05.676347 12033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:57:05.677122 12033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:57:05.732946 12033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:57:05.734321 12033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.132.130
I20250814 13:57:05.793448 12033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:40333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.132.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.11.132.130
--webserver_port=0
--tserver_master_addrs=127.11.132.190:43609
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.132.130
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:57:05.795087 12033 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:57:05.796977 12033 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:57:05.810642 12039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 13:57:06.283417 12029 heartbeater.cc:499] Master 127.11.132.190:43609 was elected leader, sending a full tablet report...
W20250814 13:57:07.212445 12038 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 12033
W20250814 13:57:07.440451 12038 kernel_stack_watchdog.cc:198] Thread 12033 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 397ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250814 13:57:05.812175 12040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:57:07.446110 12033 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.633s user 0.000s sys 0.004s
W20250814 13:57:07.446440 12041 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1633 milliseconds
W20250814 13:57:07.446563 12033 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.634s user 0.001s sys 0.004s
W20250814 13:57:07.448057 12042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 13:57:07.448125 12033 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 13:57:07.449721 12033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:57:07.452567 12033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:57:07.454054 12033 hybrid_clock.cc:648] HybridClock initialized: now 1755179827454006 us; error 47 us; skew 500 ppm
I20250814 13:57:07.455138 12033 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:57:07.460974 12033 webserver.cc:480] Webserver started at http://127.11.132.130:40065/ using document root <none> and password file <none>
I20250814 13:57:07.462090 12033 fs_manager.cc:362] Metadata directory not provided
I20250814 13:57:07.462289 12033 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:57:07.462695 12033 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 13:57:07.466949 12033 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "f2319463db2f4e2080f3b073ae605cbb"
format_stamp: "Formatted at 2025-08-14 13:57:07 on dist-test-slave-q1tc"
I20250814 13:57:07.467990 12033 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "f2319463db2f4e2080f3b073ae605cbb"
format_stamp: "Formatted at 2025-08-14 13:57:07 on dist-test-slave-q1tc"
I20250814 13:57:07.474944 12033 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20250814 13:57:07.480271 12049 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:57:07.481314 12033 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250814 13:57:07.481590 12033 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "f2319463db2f4e2080f3b073ae605cbb"
format_stamp: "Formatted at 2025-08-14 13:57:07 on dist-test-slave-q1tc"
I20250814 13:57:07.481884 12033 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:57:07.545276 12033 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:57:07.546627 12033 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:57:07.547016 12033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:57:07.549353 12033 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 13:57:07.552948 12033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 13:57:07.553217 12033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250814 13:57:07.553467 12033 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 13:57:07.553629 12033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 13:57:07.682267 12033 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.130:42573
I20250814 13:57:07.682415 12161 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.130:42573 every 8 connection(s)
I20250814 13:57:07.684798 12033 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250814 13:57:07.691474 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 12033
I20250814 13:57:07.691958 11794 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250814 13:57:07.698554 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.132.131:0
--local_ip_for_outbound_sockets=127.11.132.131
--webserver_interface=127.11.132.131
--webserver_port=0
--tserver_master_addrs=127.11.132.190:43609
--builtin_ntp_servers=127.11.132.148:40333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250814 13:57:07.706542 12162 heartbeater.cc:344] Connected to a master server at 127.11.132.190:43609
I20250814 13:57:07.706959 12162 heartbeater.cc:461] Registering TS with master...
I20250814 13:57:07.707911 12162 heartbeater.cc:507] Master 127.11.132.190:43609 requested a full tablet report, sending...
I20250814 13:57:07.710042 11841 ts_manager.cc:194] Registered new tserver with Master: f2319463db2f4e2080f3b073ae605cbb (127.11.132.130:42573)
I20250814 13:57:07.711791 11841 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.132.130:42607
W20250814 13:57:07.983882 12166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:57:07.984324 12166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:57:07.984800 12166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:57:08.013669 12166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:57:08.014587 12166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.132.131
I20250814 13:57:08.050288 12166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:40333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.132.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.11.132.131
--webserver_port=0
--tserver_master_addrs=127.11.132.190:43609
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.132.131
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:57:08.051506 12166 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:57:08.052871 12166 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:57:08.063505 12172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 13:57:08.714977 12162 heartbeater.cc:499] Master 127.11.132.190:43609 was elected leader, sending a full tablet report...
W20250814 13:57:08.064054 12173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:57:09.227574 12175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:57:09.230633 12174 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1162 milliseconds
I20250814 13:57:09.230728 12166 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 13:57:09.232014 12166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:57:09.234513 12166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:57:09.235931 12166 hybrid_clock.cc:648] HybridClock initialized: now 1755179829235864 us; error 62 us; skew 500 ppm
I20250814 13:57:09.236632 12166 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:57:09.243280 12166 webserver.cc:480] Webserver started at http://127.11.132.131:36259/ using document root <none> and password file <none>
I20250814 13:57:09.244369 12166 fs_manager.cc:362] Metadata directory not provided
I20250814 13:57:09.244633 12166 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:57:09.245400 12166 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 13:57:09.251705 12166 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "4b40806ed51a4879bb1918fb595551f5"
format_stamp: "Formatted at 2025-08-14 13:57:09 on dist-test-slave-q1tc"
I20250814 13:57:09.252753 12166 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "4b40806ed51a4879bb1918fb595551f5"
format_stamp: "Formatted at 2025-08-14 13:57:09 on dist-test-slave-q1tc"
I20250814 13:57:09.259912 12166 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20250814 13:57:09.265463 12182 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:57:09.266595 12166 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250814 13:57:09.266906 12166 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "4b40806ed51a4879bb1918fb595551f5"
format_stamp: "Formatted at 2025-08-14 13:57:09 on dist-test-slave-q1tc"
I20250814 13:57:09.267192 12166 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:57:09.314671 12166 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:57:09.315944 12166 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:57:09.316305 12166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:57:09.319001 12166 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 13:57:09.323024 12166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 13:57:09.323211 12166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 13:57:09.323398 12166 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 13:57:09.323547 12166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 13:57:09.450011 12166 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.131:41961
I20250814 13:57:09.450109 12294 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.131:41961 every 8 connection(s)
I20250814 13:57:09.452387 12166 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250814 13:57:09.455781 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 12166
I20250814 13:57:09.456255 11794 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250814 13:57:09.462496 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.132.132:0
--local_ip_for_outbound_sockets=127.11.132.132
--webserver_interface=127.11.132.132
--webserver_port=0
--tserver_master_addrs=127.11.132.190:43609
--builtin_ntp_servers=127.11.132.148:40333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250814 13:57:09.477180 12295 heartbeater.cc:344] Connected to a master server at 127.11.132.190:43609
I20250814 13:57:09.477702 12295 heartbeater.cc:461] Registering TS with master...
I20250814 13:57:09.479104 12295 heartbeater.cc:507] Master 127.11.132.190:43609 requested a full tablet report, sending...
I20250814 13:57:09.481685 11841 ts_manager.cc:194] Registered new tserver with Master: 4b40806ed51a4879bb1918fb595551f5 (127.11.132.131:41961)
I20250814 13:57:09.483582 11841 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.132.131:33259
W20250814 13:57:09.769686 12299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:57:09.770087 12299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:57:09.770478 12299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:57:09.799757 12299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:57:09.800418 12299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.132.132
I20250814 13:57:09.831168 12299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:40333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.132.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.11.132.132
--webserver_port=0
--tserver_master_addrs=127.11.132.190:43609
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.132.132
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:57:09.832306 12299 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:57:09.833837 12299 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:57:09.843597 12306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 13:57:10.487046 12295 heartbeater.cc:499] Master 127.11.132.190:43609 was elected leader, sending a full tablet report...
W20250814 13:57:09.845434 12307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:57:09.848544 12309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:57:10.958879 12308 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250814 13:57:10.959005 12299 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 13:57:10.962371 12299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:57:10.965090 12299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:57:10.966535 12299 hybrid_clock.cc:648] HybridClock initialized: now 1755179830966493 us; error 65 us; skew 500 ppm
I20250814 13:57:10.967226 12299 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:57:10.972862 12299 webserver.cc:480] Webserver started at http://127.11.132.132:36697/ using document root <none> and password file <none>
I20250814 13:57:10.973760 12299 fs_manager.cc:362] Metadata directory not provided
I20250814 13:57:10.973943 12299 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:57:10.974364 12299 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 13:57:10.978550 12299 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "5e01eb3a7be0476abe51f9b9efb101a7"
format_stamp: "Formatted at 2025-08-14 13:57:10 on dist-test-slave-q1tc"
I20250814 13:57:10.979593 12299 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "5e01eb3a7be0476abe51f9b9efb101a7"
format_stamp: "Formatted at 2025-08-14 13:57:10 on dist-test-slave-q1tc"
I20250814 13:57:10.986797 12299 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20250814 13:57:10.992050 12316 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:57:10.993001 12299 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250814 13:57:10.993341 12299 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "5e01eb3a7be0476abe51f9b9efb101a7"
format_stamp: "Formatted at 2025-08-14 13:57:10 on dist-test-slave-q1tc"
I20250814 13:57:10.993708 12299 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:57:11.040968 12299 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:57:11.042418 12299 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:57:11.042832 12299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:57:11.045243 12299 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 13:57:11.049189 12299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 13:57:11.049388 12299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 13:57:11.049623 12299 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 13:57:11.049777 12299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 13:57:11.178628 12299 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.132:38297
I20250814 13:57:11.178740 12428 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.132:38297 every 8 connection(s)
I20250814 13:57:11.181191 12299 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250814 13:57:11.189831 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 12299
I20250814 13:57:11.190526 11794 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250814 13:57:11.202539 12429 heartbeater.cc:344] Connected to a master server at 127.11.132.190:43609
I20250814 13:57:11.202910 12429 heartbeater.cc:461] Registering TS with master...
I20250814 13:57:11.203866 12429 heartbeater.cc:507] Master 127.11.132.190:43609 requested a full tablet report, sending...
I20250814 13:57:11.205821 11841 ts_manager.cc:194] Registered new tserver with Master: 5e01eb3a7be0476abe51f9b9efb101a7 (127.11.132.132:38297)
I20250814 13:57:11.206979 11841 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.132.132:51611
I20250814 13:57:11.211767 11794 external_mini_cluster.cc:949] 4 TS(s) registered with all masters
I20250814 13:57:11.261471 11841 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33704:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250814 13:57:11.338927 12364 tablet_service.cc:1468] Processing CreateTablet for tablet ab93eceae8144cc0b69dbb5a3813f8fe (DEFAULT_TABLE table=TestTable [id=c5a1facec8404251b48cb810ec2ae09e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 13:57:11.338943 12230 tablet_service.cc:1468] Processing CreateTablet for tablet ab93eceae8144cc0b69dbb5a3813f8fe (DEFAULT_TABLE table=TestTable [id=c5a1facec8404251b48cb810ec2ae09e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 13:57:11.340577 12230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab93eceae8144cc0b69dbb5a3813f8fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:57:11.340560 12364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab93eceae8144cc0b69dbb5a3813f8fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:57:11.350025 11964 tablet_service.cc:1468] Processing CreateTablet for tablet ab93eceae8144cc0b69dbb5a3813f8fe (DEFAULT_TABLE table=TestTable [id=c5a1facec8404251b48cb810ec2ae09e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 13:57:11.352681 11964 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab93eceae8144cc0b69dbb5a3813f8fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:57:11.362033 12449 tablet_bootstrap.cc:492] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7: Bootstrap starting.
I20250814 13:57:11.366190 12450 tablet_bootstrap.cc:492] T ab93eceae8144cc0b69dbb5a3813f8fe P 4b40806ed51a4879bb1918fb595551f5: Bootstrap starting.
I20250814 13:57:11.372359 12449 tablet_bootstrap.cc:654] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7: Neither blocks nor log segments found. Creating new log.
I20250814 13:57:11.373714 12450 tablet_bootstrap.cc:654] T ab93eceae8144cc0b69dbb5a3813f8fe P 4b40806ed51a4879bb1918fb595551f5: Neither blocks nor log segments found. Creating new log.
I20250814 13:57:11.374958 12449 log.cc:826] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7: Log is configured to *not* fsync() on all Append() calls
I20250814 13:57:11.375967 12450 log.cc:826] T ab93eceae8144cc0b69dbb5a3813f8fe P 4b40806ed51a4879bb1918fb595551f5: Log is configured to *not* fsync() on all Append() calls
I20250814 13:57:11.380263 12449 tablet_bootstrap.cc:492] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7: No bootstrap required, opened a new log
I20250814 13:57:11.380867 12449 ts_tablet_manager.cc:1397] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.005s
I20250814 13:57:11.386046 12453 tablet_bootstrap.cc:492] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f: Bootstrap starting.
I20250814 13:57:11.386343 12450 tablet_bootstrap.cc:492] T ab93eceae8144cc0b69dbb5a3813f8fe P 4b40806ed51a4879bb1918fb595551f5: No bootstrap required, opened a new log
I20250814 13:57:11.386860 12450 ts_tablet_manager.cc:1397] T ab93eceae8144cc0b69dbb5a3813f8fe P 4b40806ed51a4879bb1918fb595551f5: Time spent bootstrapping tablet: real 0.021s user 0.016s sys 0.004s
I20250814 13:57:11.393329 12453 tablet_bootstrap.cc:654] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f: Neither blocks nor log segments found. Creating new log.
I20250814 13:57:11.395632 12453 log.cc:826] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f: Log is configured to *not* fsync() on all Append() calls
I20250814 13:57:11.404700 12449 raft_consensus.cc:357] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b40806ed51a4879bb1918fb595551f5" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 41961 } } peers { permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" member_type: VOTER last_known_addr { host: "127.11.132.132" port: 38297 } } peers { permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 } }
I20250814 13:57:11.405710 12449 raft_consensus.cc:383] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 13:57:11.406033 12449 raft_consensus.cc:738] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e01eb3a7be0476abe51f9b9efb101a7, State: Initialized, Role: FOLLOWER
I20250814 13:57:11.406920 12449 consensus_queue.cc:260] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b40806ed51a4879bb1918fb595551f5" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 41961 } } peers { permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" member_type: VOTER last_known_addr { host: "127.11.132.132" port: 38297 } } peers { permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 } }
I20250814 13:57:11.410133 12429 heartbeater.cc:499] Master 127.11.132.190:43609 was elected leader, sending a full tablet report...
I20250814 13:57:11.411412 12449 ts_tablet_manager.cc:1428] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7: Time spent starting tablet: real 0.030s user 0.025s sys 0.002s
I20250814 13:57:11.412485 12453 tablet_bootstrap.cc:492] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f: No bootstrap required, opened a new log
I20250814 13:57:11.412967 12453 ts_tablet_manager.cc:1397] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.012s
I20250814 13:57:11.420146 12450 raft_consensus.cc:357] T ab93eceae8144cc0b69dbb5a3813f8fe P 4b40806ed51a4879bb1918fb595551f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b40806ed51a4879bb1918fb595551f5" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 41961 } } peers { permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" member_type: VOTER last_known_addr { host: "127.11.132.132" port: 38297 } } peers { permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 } }
I20250814 13:57:11.421033 12450 raft_consensus.cc:383] T ab93eceae8144cc0b69dbb5a3813f8fe P 4b40806ed51a4879bb1918fb595551f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 13:57:11.421423 12450 raft_consensus.cc:738] T ab93eceae8144cc0b69dbb5a3813f8fe P 4b40806ed51a4879bb1918fb595551f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b40806ed51a4879bb1918fb595551f5, State: Initialized, Role: FOLLOWER
I20250814 13:57:11.422420 12450 consensus_queue.cc:260] T ab93eceae8144cc0b69dbb5a3813f8fe P 4b40806ed51a4879bb1918fb595551f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b40806ed51a4879bb1918fb595551f5" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 41961 } } peers { permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" member_type: VOTER last_known_addr { host: "127.11.132.132" port: 38297 } } peers { permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 } }
I20250814 13:57:11.430059 12450 ts_tablet_manager.cc:1428] T ab93eceae8144cc0b69dbb5a3813f8fe P 4b40806ed51a4879bb1918fb595551f5: Time spent starting tablet: real 0.043s user 0.032s sys 0.004s
W20250814 13:57:11.435688 12430 tablet.cc:2378] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250814 13:57:11.438313 12453 raft_consensus.cc:357] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b40806ed51a4879bb1918fb595551f5" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 41961 } } peers { permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" member_type: VOTER last_known_addr { host: "127.11.132.132" port: 38297 } } peers { permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 } }
I20250814 13:57:11.438930 12453 raft_consensus.cc:383] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 13:57:11.439119 12453 raft_consensus.cc:738] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 071c488311ff42dc9d7dbed1bd39ea9f, State: Initialized, Role: FOLLOWER
I20250814 13:57:11.440052 12453 consensus_queue.cc:260] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b40806ed51a4879bb1918fb595551f5" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 41961 } } peers { permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" member_type: VOTER last_known_addr { host: "127.11.132.132" port: 38297 } } peers { permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 } }
I20250814 13:57:11.443209 12453 ts_tablet_manager.cc:1428] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f: Time spent starting tablet: real 0.030s user 0.019s sys 0.009s
W20250814 13:57:11.459451 12296 tablet.cc:2378] T ab93eceae8144cc0b69dbb5a3813f8fe P 4b40806ed51a4879bb1918fb595551f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250814 13:57:11.483776 12030 tablet.cc:2378] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250814 13:57:11.533897 12456 raft_consensus.cc:491] T ab93eceae8144cc0b69dbb5a3813f8fe P 4b40806ed51a4879bb1918fb595551f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250814 13:57:11.534371 12456 raft_consensus.cc:513] T ab93eceae8144cc0b69dbb5a3813f8fe P 4b40806ed51a4879bb1918fb595551f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b40806ed51a4879bb1918fb595551f5" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 41961 } } peers { permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" member_type: VOTER last_known_addr { host: "127.11.132.132" port: 38297 } } peers { permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 } }
I20250814 13:57:11.536767 12456 leader_election.cc:290] T ab93eceae8144cc0b69dbb5a3813f8fe P 4b40806ed51a4879bb1918fb595551f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e01eb3a7be0476abe51f9b9efb101a7 (127.11.132.132:38297), 071c488311ff42dc9d7dbed1bd39ea9f (127.11.132.129:41199)
I20250814 13:57:11.541530 12455 raft_consensus.cc:491] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250814 13:57:11.542114 12455 raft_consensus.cc:513] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b40806ed51a4879bb1918fb595551f5" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 41961 } } peers { permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" member_type: VOTER last_known_addr { host: "127.11.132.132" port: 38297 } } peers { permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 } }
I20250814 13:57:11.551292 12384 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ab93eceae8144cc0b69dbb5a3813f8fe" candidate_uuid: "4b40806ed51a4879bb1918fb595551f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" is_pre_election: true
I20250814 13:57:11.552001 12384 raft_consensus.cc:2466] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b40806ed51a4879bb1918fb595551f5 in term 0.
I20250814 13:57:11.553303 12186 leader_election.cc:304] T ab93eceae8144cc0b69dbb5a3813f8fe P 4b40806ed51a4879bb1918fb595551f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b40806ed51a4879bb1918fb595551f5, 5e01eb3a7be0476abe51f9b9efb101a7; no voters:
I20250814 13:57:11.553241 11984 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ab93eceae8144cc0b69dbb5a3813f8fe" candidate_uuid: "4b40806ed51a4879bb1918fb595551f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" is_pre_election: true
I20250814 13:57:11.554139 11984 raft_consensus.cc:2466] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b40806ed51a4879bb1918fb595551f5 in term 0.
I20250814 13:57:11.554567 12456 raft_consensus.cc:2802] T ab93eceae8144cc0b69dbb5a3813f8fe P 4b40806ed51a4879bb1918fb595551f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250814 13:57:11.554924 12456 raft_consensus.cc:491] T ab93eceae8144cc0b69dbb5a3813f8fe P 4b40806ed51a4879bb1918fb595551f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250814 13:57:11.554982 12455 leader_election.cc:290] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b40806ed51a4879bb1918fb595551f5 (127.11.132.131:41961), 071c488311ff42dc9d7dbed1bd39ea9f (127.11.132.129:41199)
I20250814 13:57:11.555258 12456 raft_consensus.cc:3058] T ab93eceae8144cc0b69dbb5a3813f8fe P 4b40806ed51a4879bb1918fb595551f5 [term 0 FOLLOWER]: Advancing to term 1
I20250814 13:57:11.557363 12250 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ab93eceae8144cc0b69dbb5a3813f8fe" candidate_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b40806ed51a4879bb1918fb595551f5" is_pre_election: true
I20250814 13:57:11.560917 12456 raft_consensus.cc:513] T ab93eceae8144cc0b69dbb5a3813f8fe P 4b40806ed51a4879bb1918fb595551f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b40806ed51a4879bb1918fb595551f5" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 41961 } } peers { permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" member_type: VOTER last_known_addr { host: "127.11.132.132" port: 38297 } } peers { permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 } }
I20250814 13:57:11.562031 12250 raft_consensus.cc:2391] T ab93eceae8144cc0b69dbb5a3813f8fe P 4b40806ed51a4879bb1918fb595551f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e01eb3a7be0476abe51f9b9efb101a7 in current term 1: Already voted for candidate 4b40806ed51a4879bb1918fb595551f5 in this term.
I20250814 13:57:11.563207 11984 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ab93eceae8144cc0b69dbb5a3813f8fe" candidate_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" is_pre_election: true
I20250814 13:57:11.563455 12384 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ab93eceae8144cc0b69dbb5a3813f8fe" candidate_uuid: "4b40806ed51a4879bb1918fb595551f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e01eb3a7be0476abe51f9b9efb101a7"
I20250814 13:57:11.563795 11984 raft_consensus.cc:2466] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e01eb3a7be0476abe51f9b9efb101a7 in term 0.
I20250814 13:57:11.564006 12384 raft_consensus.cc:3058] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [term 0 FOLLOWER]: Advancing to term 1
I20250814 13:57:11.563977 11983 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ab93eceae8144cc0b69dbb5a3813f8fe" candidate_uuid: "4b40806ed51a4879bb1918fb595551f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "071c488311ff42dc9d7dbed1bd39ea9f"
I20250814 13:57:11.564594 11983 raft_consensus.cc:3058] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f [term 0 FOLLOWER]: Advancing to term 1
I20250814 13:57:11.565331 12320 leader_election.cc:304] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 071c488311ff42dc9d7dbed1bd39ea9f, 5e01eb3a7be0476abe51f9b9efb101a7; no voters:
I20250814 13:57:11.568148 12456 leader_election.cc:290] T ab93eceae8144cc0b69dbb5a3813f8fe P 4b40806ed51a4879bb1918fb595551f5 [CANDIDATE]: Term 1 election: Requested vote from peers 5e01eb3a7be0476abe51f9b9efb101a7 (127.11.132.132:38297), 071c488311ff42dc9d7dbed1bd39ea9f (127.11.132.129:41199)
I20250814 13:57:11.570902 12384 raft_consensus.cc:2466] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b40806ed51a4879bb1918fb595551f5 in term 1.
I20250814 13:57:11.571357 12455 raft_consensus.cc:2762] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20250814 13:57:11.571323 11983 raft_consensus.cc:2466] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b40806ed51a4879bb1918fb595551f5 in term 1.
I20250814 13:57:11.571897 12186 leader_election.cc:304] T ab93eceae8144cc0b69dbb5a3813f8fe P 4b40806ed51a4879bb1918fb595551f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b40806ed51a4879bb1918fb595551f5, 5e01eb3a7be0476abe51f9b9efb101a7; no voters:
I20250814 13:57:11.572510 12456 raft_consensus.cc:2802] T ab93eceae8144cc0b69dbb5a3813f8fe P 4b40806ed51a4879bb1918fb595551f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250814 13:57:11.574440 12456 raft_consensus.cc:695] T ab93eceae8144cc0b69dbb5a3813f8fe P 4b40806ed51a4879bb1918fb595551f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4b40806ed51a4879bb1918fb595551f5, State: Running, Role: LEADER
I20250814 13:57:11.575413 12456 consensus_queue.cc:237] T ab93eceae8144cc0b69dbb5a3813f8fe P 4b40806ed51a4879bb1918fb595551f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b40806ed51a4879bb1918fb595551f5" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 41961 } } peers { permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" member_type: VOTER last_known_addr { host: "127.11.132.132" port: 38297 } } peers { permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 } }
I20250814 13:57:11.583933 11840 catalog_manager.cc:5582] T ab93eceae8144cc0b69dbb5a3813f8fe P 4b40806ed51a4879bb1918fb595551f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b40806ed51a4879bb1918fb595551f5 (127.11.132.131). New cstate: current_term: 1 leader_uuid: "4b40806ed51a4879bb1918fb595551f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b40806ed51a4879bb1918fb595551f5" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 41961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" member_type: VOTER last_known_addr { host: "127.11.132.132" port: 38297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 } health_report { overall_health: UNKNOWN } } }
I20250814 13:57:11.634877 11794 external_mini_cluster.cc:949] 4 TS(s) registered with all masters
I20250814 13:57:11.638222 11794 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 071c488311ff42dc9d7dbed1bd39ea9f to finish bootstrapping
I20250814 13:57:11.651170 11794 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 4b40806ed51a4879bb1918fb595551f5 to finish bootstrapping
I20250814 13:57:11.660265 11794 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 5e01eb3a7be0476abe51f9b9efb101a7 to finish bootstrapping
I20250814 13:57:11.669791 11794 kudu-admin-test.cc:709] Waiting for Master to see the current replicas...
I20250814 13:57:11.672843 11794 kudu-admin-test.cc:716] Tablet locations:
tablet_locations {
tablet_id: "ab93eceae8144cc0b69dbb5a3813f8fe"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: LEADER
}
interned_replicas {
ts_info_idx: 1
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "4b40806ed51a4879bb1918fb595551f5"
rpc_addresses {
host: "127.11.132.131"
port: 41961
}
}
ts_infos {
permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7"
rpc_addresses {
host: "127.11.132.132"
port: 38297
}
}
ts_infos {
permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f"
rpc_addresses {
host: "127.11.132.129"
port: 41199
}
}
I20250814 13:57:12.009351 12456 consensus_queue.cc:1035] T ab93eceae8144cc0b69dbb5a3813f8fe P 4b40806ed51a4879bb1918fb595551f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250814 13:57:12.023769 12456 consensus_queue.cc:1035] T ab93eceae8144cc0b69dbb5a3813f8fe P 4b40806ed51a4879bb1918fb595551f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" member_type: VOTER last_known_addr { host: "127.11.132.132" port: 38297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250814 13:57:12.037413 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 12166
I20250814 13:57:12.062836 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 11808
I20250814 13:57:12.088919 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.190:43609
--webserver_interface=127.11.132.190
--webserver_port=46041
--builtin_ntp_servers=127.11.132.148:40333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.132.190:43609 with env {}
W20250814 13:57:12.405961 12472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:57:12.406541 12472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:57:12.406984 12472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:57:12.437774 12472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250814 13:57:12.438109 12472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:57:12.438380 12472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250814 13:57:12.438621 12472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250814 13:57:12.472908 12472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:40333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.132.190:43609
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.190:43609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.11.132.190
--webserver_port=46041
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:57:12.474270 12472 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:57:12.475776 12472 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:57:12.486057 12478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:57:12.734336 12162 heartbeater.cc:646] Failed to heartbeat to 127.11.132.190:43609 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.132.190:43609: connect: Connection refused (error 111)
W20250814 13:57:13.040258 12029 heartbeater.cc:646] Failed to heartbeat to 127.11.132.190:43609 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.132.190:43609: connect: Connection refused (error 111)
W20250814 13:57:13.051960 12429 heartbeater.cc:646] Failed to heartbeat to 127.11.132.190:43609 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.132.190:43609: connect: Connection refused (error 111)
I20250814 13:57:13.525614 12487 raft_consensus.cc:491] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b40806ed51a4879bb1918fb595551f5)
I20250814 13:57:13.526290 12487 raft_consensus.cc:513] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b40806ed51a4879bb1918fb595551f5" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 41961 } } peers { permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" member_type: VOTER last_known_addr { host: "127.11.132.132" port: 38297 } } peers { permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 } }
I20250814 13:57:13.529949 12487 leader_election.cc:290] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b40806ed51a4879bb1918fb595551f5 (127.11.132.131:41961), 5e01eb3a7be0476abe51f9b9efb101a7 (127.11.132.132:38297)
W20250814 13:57:13.533354 11918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.131:41961: connect: Connection refused (error 111)
I20250814 13:57:13.537370 12488 raft_consensus.cc:491] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b40806ed51a4879bb1918fb595551f5)
I20250814 13:57:13.539286 12488 raft_consensus.cc:513] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b40806ed51a4879bb1918fb595551f5" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 41961 } } peers { permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" member_type: VOTER last_known_addr { host: "127.11.132.132" port: 38297 } } peers { permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 } }
W20250814 13:57:13.541589 11918 leader_election.cc:336] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4b40806ed51a4879bb1918fb595551f5 (127.11.132.131:41961): Network error: Client connection negotiation failed: client connection to 127.11.132.131:41961: connect: Connection refused (error 111)
W20250814 13:57:13.543629 12318 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.131:41961: connect: Connection refused (error 111)
I20250814 13:57:13.551040 12488 leader_election.cc:290] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b40806ed51a4879bb1918fb595551f5 (127.11.132.131:41961), 071c488311ff42dc9d7dbed1bd39ea9f (127.11.132.129:41199)
I20250814 13:57:13.550984 11983 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ab93eceae8144cc0b69dbb5a3813f8fe" candidate_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" is_pre_election: true
I20250814 13:57:13.551528 11983 raft_consensus.cc:2466] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e01eb3a7be0476abe51f9b9efb101a7 in term 1.
I20250814 13:57:13.552942 12320 leader_election.cc:304] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 071c488311ff42dc9d7dbed1bd39ea9f, 5e01eb3a7be0476abe51f9b9efb101a7; no voters:
I20250814 13:57:13.553894 12488 raft_consensus.cc:2802] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250814 13:57:13.554265 12488 raft_consensus.cc:491] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4b40806ed51a4879bb1918fb595551f5)
I20250814 13:57:13.554642 12488 raft_consensus.cc:3058] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [term 1 FOLLOWER]: Advancing to term 2
I20250814 13:57:13.556447 12384 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ab93eceae8144cc0b69dbb5a3813f8fe" candidate_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" is_pre_election: true
W20250814 13:57:13.559155 12318 leader_election.cc:336] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4b40806ed51a4879bb1918fb595551f5 (127.11.132.131:41961): Network error: Client connection negotiation failed: client connection to 127.11.132.131:41961: connect: Connection refused (error 111)
I20250814 13:57:13.561583 12488 raft_consensus.cc:513] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b40806ed51a4879bb1918fb595551f5" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 41961 } } peers { permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" member_type: VOTER last_known_addr { host: "127.11.132.132" port: 38297 } } peers { permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 } }
I20250814 13:57:13.562772 12384 raft_consensus.cc:2391] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 071c488311ff42dc9d7dbed1bd39ea9f in current term 2: Already voted for candidate 5e01eb3a7be0476abe51f9b9efb101a7 in this term.
I20250814 13:57:13.564075 12488 leader_election.cc:290] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [CANDIDATE]: Term 2 election: Requested vote from peers 4b40806ed51a4879bb1918fb595551f5 (127.11.132.131:41961), 071c488311ff42dc9d7dbed1bd39ea9f (127.11.132.129:41199)
I20250814 13:57:13.564276 11983 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ab93eceae8144cc0b69dbb5a3813f8fe" candidate_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "071c488311ff42dc9d7dbed1bd39ea9f"
I20250814 13:57:13.564801 11983 raft_consensus.cc:3058] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f [term 1 FOLLOWER]: Advancing to term 2
I20250814 13:57:13.567477 11920 leader_election.cc:304] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 071c488311ff42dc9d7dbed1bd39ea9f; no voters: 4b40806ed51a4879bb1918fb595551f5, 5e01eb3a7be0476abe51f9b9efb101a7
I20250814 13:57:13.571094 11983 raft_consensus.cc:2466] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e01eb3a7be0476abe51f9b9efb101a7 in term 2.
I20250814 13:57:13.571640 12487 raft_consensus.cc:2747] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250814 13:57:13.574431 12320 leader_election.cc:304] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 071c488311ff42dc9d7dbed1bd39ea9f, 5e01eb3a7be0476abe51f9b9efb101a7; no voters:
W20250814 13:57:13.575002 12318 leader_election.cc:336] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4b40806ed51a4879bb1918fb595551f5 (127.11.132.131:41961): Network error: Client connection negotiation failed: client connection to 127.11.132.131:41961: connect: Connection refused (error 111)
I20250814 13:57:13.575273 12488 raft_consensus.cc:2802] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [term 2 FOLLOWER]: Leader election won for term 2
I20250814 13:57:13.577106 12488 raft_consensus.cc:695] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [term 2 LEADER]: Becoming Leader. State: Replica: 5e01eb3a7be0476abe51f9b9efb101a7, State: Running, Role: LEADER
I20250814 13:57:13.578014 12488 consensus_queue.cc:237] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b40806ed51a4879bb1918fb595551f5" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 41961 } } peers { permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" member_type: VOTER last_known_addr { host: "127.11.132.132" port: 38297 } } peers { permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 } }
W20250814 13:57:12.487079 12479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:57:13.709133 12481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:57:13.711493 12480 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1220 milliseconds
I20250814 13:57:13.711583 12472 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 13:57:13.712736 12472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:57:13.715410 12472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:57:13.716821 12472 hybrid_clock.cc:648] HybridClock initialized: now 1755179833716784 us; error 54 us; skew 500 ppm
I20250814 13:57:13.717578 12472 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:57:13.722986 12472 webserver.cc:480] Webserver started at http://127.11.132.190:46041/ using document root <none> and password file <none>
I20250814 13:57:13.723803 12472 fs_manager.cc:362] Metadata directory not provided
I20250814 13:57:13.723999 12472 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:57:13.731474 12472 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20250814 13:57:13.735708 12500 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:57:13.737027 12472 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20250814 13:57:13.737452 12472 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
uuid: "90cf1783ebdd4afe8f5b95f54d7665a8"
format_stamp: "Formatted at 2025-08-14 13:57:03 on dist-test-slave-q1tc"
I20250814 13:57:13.739593 12472 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:57:13.796741 12472 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:57:13.798060 12472 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:57:13.798403 12472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:57:13.864034 12472 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.190:43609
I20250814 13:57:13.864048 12552 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.190:43609 every 8 connection(s)
I20250814 13:57:13.866901 12472 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250814 13:57:13.868728 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 12472
I20250814 13:57:13.869483 11794 kudu-admin-test.cc:735] Forcing unsafe config change on tserver 5e01eb3a7be0476abe51f9b9efb101a7
I20250814 13:57:13.880995 12553 sys_catalog.cc:263] Verifying existing consensus state
I20250814 13:57:13.885519 12553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8: Bootstrap starting.
I20250814 13:57:13.921939 12553 log.cc:826] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8: Log is configured to *not* fsync() on all Append() calls
I20250814 13:57:13.944584 12553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 13:57:13.945433 12553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8: Bootstrap complete.
I20250814 13:57:13.966056 11983 raft_consensus.cc:1273] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f [term 2 FOLLOWER]: Refusing update from remote peer 5e01eb3a7be0476abe51f9b9efb101a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250814 13:57:13.967823 12488 consensus_queue.cc:1035] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250814 13:57:13.967440 12553 raft_consensus.cc:357] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90cf1783ebdd4afe8f5b95f54d7665a8" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 43609 } }
I20250814 13:57:13.969712 12553 raft_consensus.cc:738] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90cf1783ebdd4afe8f5b95f54d7665a8, State: Initialized, Role: FOLLOWER
I20250814 13:57:13.970583 12553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90cf1783ebdd4afe8f5b95f54d7665a8" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 43609 } }
I20250814 13:57:13.971972 12553 raft_consensus.cc:397] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 13:57:13.972316 12553 raft_consensus.cc:491] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 13:57:13.972790 12553 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8 [term 1 FOLLOWER]: Advancing to term 2
I20250814 13:57:13.982681 12553 raft_consensus.cc:513] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90cf1783ebdd4afe8f5b95f54d7665a8" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 43609 } }
I20250814 13:57:13.983479 12553 leader_election.cc:304] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90cf1783ebdd4afe8f5b95f54d7665a8; no voters:
I20250814 13:57:14.020977 12553 leader_election.cc:290] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250814 13:57:14.021502 12559 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8 [term 2 FOLLOWER]: Leader election won for term 2
I20250814 13:57:14.028650 12559 raft_consensus.cc:695] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8 [term 2 LEADER]: Becoming Leader. State: Replica: 90cf1783ebdd4afe8f5b95f54d7665a8, State: Running, Role: LEADER
I20250814 13:57:14.029877 12559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90cf1783ebdd4afe8f5b95f54d7665a8" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 43609 } }
I20250814 13:57:14.036681 12553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8 [sys.catalog]: configured and running, proceeding with master startup.
I20250814 13:57:14.044770 12029 heartbeater.cc:344] Connected to a master server at 127.11.132.190:43609
I20250814 13:57:14.051769 12564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90cf1783ebdd4afe8f5b95f54d7665a8. Latest consensus state: current_term: 2 leader_uuid: "90cf1783ebdd4afe8f5b95f54d7665a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90cf1783ebdd4afe8f5b95f54d7665a8" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 43609 } } }
I20250814 13:57:14.054080 12564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8 [sys.catalog]: This master's current role is: LEADER
W20250814 13:57:14.054352 12318 consensus_peers.cc:489] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 -> Peer 4b40806ed51a4879bb1918fb595551f5 (127.11.132.131:41961): Couldn't send request to peer 4b40806ed51a4879bb1918fb595551f5. Status: Network error: Client connection negotiation failed: client connection to 127.11.132.131:41961: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250814 13:57:14.056489 12429 heartbeater.cc:344] Connected to a master server at 127.11.132.190:43609
I20250814 13:57:14.057662 12565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "90cf1783ebdd4afe8f5b95f54d7665a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90cf1783ebdd4afe8f5b95f54d7665a8" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 43609 } } }
I20250814 13:57:14.058285 12565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8 [sys.catalog]: This master's current role is: LEADER
I20250814 13:57:14.066843 12570 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250814 13:57:14.079396 12570 catalog_manager.cc:671] Loaded metadata for table TestTable [id=c5a1facec8404251b48cb810ec2ae09e]
I20250814 13:57:14.088231 12570 tablet_loader.cc:96] loaded metadata for tablet ab93eceae8144cc0b69dbb5a3813f8fe (table TestTable [id=c5a1facec8404251b48cb810ec2ae09e])
I20250814 13:57:14.089977 12570 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250814 13:57:14.095381 12570 catalog_manager.cc:1261] Loaded cluster ID: 1a5c953b8fc440278b2f62080c871a92
I20250814 13:57:14.095767 12570 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250814 13:57:14.104331 12570 catalog_manager.cc:1506] Loading token signing keys...
I20250814 13:57:14.109606 12570 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 90cf1783ebdd4afe8f5b95f54d7665a8: Loaded TSK: 0
I20250814 13:57:14.111335 12570 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250814 13:57:14.114419 12580 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20250814 13:57:14.249905 12555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:57:14.250630 12555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:57:14.282462 12555 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250814 13:57:14.785780 12162 heartbeater.cc:344] Connected to a master server at 127.11.132.190:43609
I20250814 13:57:14.793242 12518 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "f2319463db2f4e2080f3b073ae605cbb" instance_seqno: 1755179827652706) as {username='slave'} at 127.11.132.130:34373; Asking this server to re-register.
I20250814 13:57:14.795156 12162 heartbeater.cc:461] Registering TS with master...
I20250814 13:57:14.795964 12162 heartbeater.cc:507] Master 127.11.132.190:43609 requested a full tablet report, sending...
I20250814 13:57:14.799034 12517 ts_manager.cc:194] Registered new tserver with Master: f2319463db2f4e2080f3b073ae605cbb (127.11.132.130:42573)
I20250814 13:57:15.056121 12517 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" instance_seqno: 1755179825169809) as {username='slave'} at 127.11.132.129:55679; Asking this server to re-register.
I20250814 13:57:15.057996 12029 heartbeater.cc:461] Registering TS with master...
I20250814 13:57:15.058708 12029 heartbeater.cc:507] Master 127.11.132.190:43609 requested a full tablet report, sending...
I20250814 13:57:15.061640 12517 ts_manager.cc:194] Registered new tserver with Master: 071c488311ff42dc9d7dbed1bd39ea9f (127.11.132.129:41199)
I20250814 13:57:15.065855 12517 catalog_manager.cc:5582] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f reported cstate change: term changed from 1 to 2, leader changed from 4b40806ed51a4879bb1918fb595551f5 (127.11.132.131) to 5e01eb3a7be0476abe51f9b9efb101a7 (127.11.132.132). New cstate: current_term: 2 leader_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b40806ed51a4879bb1918fb595551f5" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 41961 } } peers { permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" member_type: VOTER last_known_addr { host: "127.11.132.132" port: 38297 } } peers { permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 } } }
I20250814 13:57:15.070464 12518 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" instance_seqno: 1755179831148658) as {username='slave'} at 127.11.132.132:56895; Asking this server to re-register.
I20250814 13:57:15.072203 12429 heartbeater.cc:461] Registering TS with master...
I20250814 13:57:15.073135 12429 heartbeater.cc:507] Master 127.11.132.190:43609 requested a full tablet report, sending...
I20250814 13:57:15.076815 12518 ts_manager.cc:194] Registered new tserver with Master: 5e01eb3a7be0476abe51f9b9efb101a7 (127.11.132.132:38297)
W20250814 13:57:15.726357 12585 debug-util.cc:398] Leaking SignalData structure 0x7b08000347a0 after lost signal to thread 12555
W20250814 13:57:15.727005 12585 kernel_stack_watchdog.cc:198] Thread 12555 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 402ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250814 13:57:15.734943 12555 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.413s user 0.486s sys 0.918s
W20250814 13:57:15.850934 12555 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.529s user 0.487s sys 0.928s
I20250814 13:57:15.899591 12384 tablet_service.cc:1905] Received UnsafeChangeConfig RPC: dest_uuid: "5e01eb3a7be0476abe51f9b9efb101a7"
tablet_id: "ab93eceae8144cc0b69dbb5a3813f8fe"
caller_id: "kudu-tools"
new_config {
peers {
permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f"
}
peers {
permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7"
}
}
from {username='slave'} at 127.0.0.1:59496
W20250814 13:57:15.900537 12384 raft_consensus.cc:2216] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [term 2 LEADER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b40806ed51a4879bb1918fb595551f5" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 41961 } } peers { permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" member_type: VOTER last_known_addr { host: "127.11.132.132" port: 38297 } } peers { permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 } }NEW CONFIG: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" member_type: VOTER last_known_addr { host: "127.11.132.132" port: 38297 } } peers { permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 } } unsafe_config_change: true
I20250814 13:57:15.901394 12384 raft_consensus.cc:3053] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [term 2 LEADER]: Stepping down as leader of term 2
I20250814 13:57:15.901618 12384 raft_consensus.cc:738] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5e01eb3a7be0476abe51f9b9efb101a7, State: Running, Role: LEADER
I20250814 13:57:15.902215 12384 consensus_queue.cc:260] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b40806ed51a4879bb1918fb595551f5" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 41961 } } peers { permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" member_type: VOTER last_known_addr { host: "127.11.132.132" port: 38297 } } peers { permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 } }
I20250814 13:57:15.903065 12384 raft_consensus.cc:3058] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [term 2 FOLLOWER]: Advancing to term 3
I20250814 13:57:17.304715 12612 raft_consensus.cc:491] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5e01eb3a7be0476abe51f9b9efb101a7)
I20250814 13:57:17.305158 12612 raft_consensus.cc:513] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b40806ed51a4879bb1918fb595551f5" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 41961 } } peers { permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" member_type: VOTER last_known_addr { host: "127.11.132.132" port: 38297 } } peers { permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 } }
I20250814 13:57:17.306792 12612 leader_election.cc:290] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4b40806ed51a4879bb1918fb595551f5 (127.11.132.131:41961), 5e01eb3a7be0476abe51f9b9efb101a7 (127.11.132.132:38297)
I20250814 13:57:17.307921 12384 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ab93eceae8144cc0b69dbb5a3813f8fe" candidate_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" is_pre_election: true
W20250814 13:57:17.311172 11918 leader_election.cc:336] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4b40806ed51a4879bb1918fb595551f5 (127.11.132.131:41961): Network error: Client connection negotiation failed: client connection to 127.11.132.131:41961: connect: Connection refused (error 111)
I20250814 13:57:17.311488 11918 leader_election.cc:304] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 071c488311ff42dc9d7dbed1bd39ea9f; no voters: 4b40806ed51a4879bb1918fb595551f5, 5e01eb3a7be0476abe51f9b9efb101a7
I20250814 13:57:17.312042 12612 raft_consensus.cc:2747] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250814 13:57:17.409829 12615 raft_consensus.cc:491] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools)
I20250814 13:57:17.410199 12615 raft_consensus.cc:513] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" member_type: VOTER last_known_addr { host: "127.11.132.132" port: 38297 } } peers { permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 } } unsafe_config_change: true
I20250814 13:57:17.411186 12615 leader_election.cc:290] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 071c488311ff42dc9d7dbed1bd39ea9f (127.11.132.129:41199)
I20250814 13:57:17.412040 11983 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ab93eceae8144cc0b69dbb5a3813f8fe" candidate_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" is_pre_election: true
I20250814 13:57:17.412494 11983 raft_consensus.cc:2466] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e01eb3a7be0476abe51f9b9efb101a7 in term 2.
I20250814 13:57:17.413360 12320 leader_election.cc:304] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 071c488311ff42dc9d7dbed1bd39ea9f, 5e01eb3a7be0476abe51f9b9efb101a7; no voters:
I20250814 13:57:17.413975 12615 raft_consensus.cc:2802] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250814 13:57:17.414234 12615 raft_consensus.cc:491] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [term 3 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools)
I20250814 13:57:17.414475 12615 raft_consensus.cc:3058] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [term 3 FOLLOWER]: Advancing to term 4
I20250814 13:57:17.418375 12615 raft_consensus.cc:513] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" member_type: VOTER last_known_addr { host: "127.11.132.132" port: 38297 } } peers { permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 } } unsafe_config_change: true
I20250814 13:57:17.419258 12615 leader_election.cc:290] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [CANDIDATE]: Term 4 election: Requested vote from peers 071c488311ff42dc9d7dbed1bd39ea9f (127.11.132.129:41199)
I20250814 13:57:17.420142 11983 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ab93eceae8144cc0b69dbb5a3813f8fe" candidate_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "071c488311ff42dc9d7dbed1bd39ea9f"
I20250814 13:57:17.420504 11983 raft_consensus.cc:3058] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f [term 2 FOLLOWER]: Advancing to term 4
I20250814 13:57:17.424569 11983 raft_consensus.cc:2466] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e01eb3a7be0476abe51f9b9efb101a7 in term 4.
I20250814 13:57:17.425328 12320 leader_election.cc:304] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 071c488311ff42dc9d7dbed1bd39ea9f, 5e01eb3a7be0476abe51f9b9efb101a7; no voters:
I20250814 13:57:17.425858 12615 raft_consensus.cc:2802] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [term 4 FOLLOWER]: Leader election won for term 4
I20250814 13:57:17.426503 12615 raft_consensus.cc:695] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [term 4 LEADER]: Becoming Leader. State: Replica: 5e01eb3a7be0476abe51f9b9efb101a7, State: Running, Role: LEADER
I20250814 13:57:17.427053 12615 consensus_queue.cc:237] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" member_type: VOTER last_known_addr { host: "127.11.132.132" port: 38297 } } peers { permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 } } unsafe_config_change: true
I20250814 13:57:17.431789 12518 catalog_manager.cc:5582] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 reported cstate change: term changed from 2 to 4, now has a pending config: VOTER 5e01eb3a7be0476abe51f9b9efb101a7 (127.11.132.132), VOTER 071c488311ff42dc9d7dbed1bd39ea9f (127.11.132.129). New cstate: current_term: 4 leader_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b40806ed51a4879bb1918fb595551f5" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 41961 } } peers { permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" member_type: VOTER last_known_addr { host: "127.11.132.132" port: 38297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" member_type: VOTER last_known_addr { host: "127.11.132.132" port: 38297 } } peers { permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 } } unsafe_config_change: true }
I20250814 13:57:17.942699 11983 raft_consensus.cc:1273] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f [term 4 FOLLOWER]: Refusing update from remote peer 5e01eb3a7be0476abe51f9b9efb101a7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250814 13:57:17.943714 12622 consensus_queue.cc:1035] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250814 13:57:17.950630 12625 raft_consensus.cc:2953] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [term 4 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER 4b40806ed51a4879bb1918fb595551f5 (127.11.132.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" member_type: VOTER last_known_addr { host: "127.11.132.132" port: 38297 } } peers { permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 } } unsafe_config_change: true }
I20250814 13:57:17.951550 11983 raft_consensus.cc:2953] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f [term 4 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER 4b40806ed51a4879bb1918fb595551f5 (127.11.132.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" member_type: VOTER last_known_addr { host: "127.11.132.132" port: 38297 } } peers { permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 } } unsafe_config_change: true }
I20250814 13:57:17.960199 12518 catalog_manager.cc:5582] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 reported cstate change: config changed from index -1 to 3, VOTER 4b40806ed51a4879bb1918fb595551f5 (127.11.132.131) evicted, no longer has a pending config: VOTER 5e01eb3a7be0476abe51f9b9efb101a7 (127.11.132.132), VOTER 071c488311ff42dc9d7dbed1bd39ea9f (127.11.132.129). New cstate: current_term: 4 leader_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" member_type: VOTER last_known_addr { host: "127.11.132.132" port: 38297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
W20250814 13:57:17.968024 12518 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet ab93eceae8144cc0b69dbb5a3813f8fe on TS 4b40806ed51a4879bb1918fb595551f5: Not found: failed to reset TS proxy: Could not find TS for UUID 4b40806ed51a4879bb1918fb595551f5
I20250814 13:57:17.981446 12384 consensus_queue.cc:237] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" member_type: VOTER last_known_addr { host: "127.11.132.132" port: 38297 } } peers { permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 } } peers { permanent_uuid: "f2319463db2f4e2080f3b073ae605cbb" member_type: NON_VOTER last_known_addr { host: "127.11.132.130" port: 42573 } attrs { promote: true } } unsafe_config_change: true
I20250814 13:57:17.985244 11983 raft_consensus.cc:1273] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f [term 4 FOLLOWER]: Refusing update from remote peer 5e01eb3a7be0476abe51f9b9efb101a7: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250814 13:57:17.986680 12625 consensus_queue.cc:1035] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250814 13:57:17.991467 12622 raft_consensus.cc:2953] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER f2319463db2f4e2080f3b073ae605cbb (127.11.132.130) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" member_type: VOTER last_known_addr { host: "127.11.132.132" port: 38297 } } peers { permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 } } peers { permanent_uuid: "f2319463db2f4e2080f3b073ae605cbb" member_type: NON_VOTER last_known_addr { host: "127.11.132.130" port: 42573 } attrs { promote: true } } unsafe_config_change: true }
I20250814 13:57:17.992919 11983 raft_consensus.cc:2953] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER f2319463db2f4e2080f3b073ae605cbb (127.11.132.130) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" member_type: VOTER last_known_addr { host: "127.11.132.132" port: 38297 } } peers { permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 } } peers { permanent_uuid: "f2319463db2f4e2080f3b073ae605cbb" member_type: NON_VOTER last_known_addr { host: "127.11.132.130" port: 42573 } attrs { promote: true } } unsafe_config_change: true }
W20250814 13:57:17.995961 12317 consensus_peers.cc:489] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 -> Peer f2319463db2f4e2080f3b073ae605cbb (127.11.132.130:42573): Couldn't send request to peer f2319463db2f4e2080f3b073ae605cbb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ab93eceae8144cc0b69dbb5a3813f8fe. This is attempt 1: this message will repeat every 5th retry.
I20250814 13:57:17.998127 12505 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ab93eceae8144cc0b69dbb5a3813f8fe with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250814 13:57:18.000691 12517 catalog_manager.cc:5582] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 reported cstate change: config changed from index 3 to 5, NON_VOTER f2319463db2f4e2080f3b073ae605cbb (127.11.132.130) added. New cstate: current_term: 4 leader_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" member_type: VOTER last_known_addr { host: "127.11.132.132" port: 38297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2319463db2f4e2080f3b073ae605cbb" member_type: NON_VOTER last_known_addr { host: "127.11.132.130" port: 42573 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true }
W20250814 13:57:18.021111 12503 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet ab93eceae8144cc0b69dbb5a3813f8fe on TS 4b40806ed51a4879bb1918fb595551f5 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 4b40806ed51a4879bb1918fb595551f5
I20250814 13:57:18.610697 12636 ts_tablet_manager.cc:927] T ab93eceae8144cc0b69dbb5a3813f8fe P f2319463db2f4e2080f3b073ae605cbb: Initiating tablet copy from peer 5e01eb3a7be0476abe51f9b9efb101a7 (127.11.132.132:38297)
I20250814 13:57:18.612761 12636 tablet_copy_client.cc:323] T ab93eceae8144cc0b69dbb5a3813f8fe P f2319463db2f4e2080f3b073ae605cbb: tablet copy: Beginning tablet copy session from remote peer at address 127.11.132.132:38297
I20250814 13:57:18.622543 12404 tablet_copy_service.cc:140] P 5e01eb3a7be0476abe51f9b9efb101a7: Received BeginTabletCopySession request for tablet ab93eceae8144cc0b69dbb5a3813f8fe from peer f2319463db2f4e2080f3b073ae605cbb ({username='slave'} at 127.11.132.130:34465)
I20250814 13:57:18.622941 12404 tablet_copy_service.cc:161] P 5e01eb3a7be0476abe51f9b9efb101a7: Beginning new tablet copy session on tablet ab93eceae8144cc0b69dbb5a3813f8fe from peer f2319463db2f4e2080f3b073ae605cbb at {username='slave'} at 127.11.132.130:34465: session id = f2319463db2f4e2080f3b073ae605cbb-ab93eceae8144cc0b69dbb5a3813f8fe
I20250814 13:57:18.627611 12404 tablet_copy_source_session.cc:215] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7: Tablet Copy: opened 0 blocks and 1 log segments
I20250814 13:57:18.632025 12636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab93eceae8144cc0b69dbb5a3813f8fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:57:18.649098 12636 tablet_copy_client.cc:806] T ab93eceae8144cc0b69dbb5a3813f8fe P f2319463db2f4e2080f3b073ae605cbb: tablet copy: Starting download of 0 data blocks...
I20250814 13:57:18.649523 12636 tablet_copy_client.cc:670] T ab93eceae8144cc0b69dbb5a3813f8fe P f2319463db2f4e2080f3b073ae605cbb: tablet copy: Starting download of 1 WAL segments...
I20250814 13:57:18.652513 12636 tablet_copy_client.cc:538] T ab93eceae8144cc0b69dbb5a3813f8fe P f2319463db2f4e2080f3b073ae605cbb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250814 13:57:18.657788 12636 tablet_bootstrap.cc:492] T ab93eceae8144cc0b69dbb5a3813f8fe P f2319463db2f4e2080f3b073ae605cbb: Bootstrap starting.
I20250814 13:57:18.668830 12636 log.cc:826] T ab93eceae8144cc0b69dbb5a3813f8fe P f2319463db2f4e2080f3b073ae605cbb: Log is configured to *not* fsync() on all Append() calls
I20250814 13:57:18.678664 12636 tablet_bootstrap.cc:492] T ab93eceae8144cc0b69dbb5a3813f8fe P f2319463db2f4e2080f3b073ae605cbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 13:57:18.679293 12636 tablet_bootstrap.cc:492] T ab93eceae8144cc0b69dbb5a3813f8fe P f2319463db2f4e2080f3b073ae605cbb: Bootstrap complete.
I20250814 13:57:18.679795 12636 ts_tablet_manager.cc:1397] T ab93eceae8144cc0b69dbb5a3813f8fe P f2319463db2f4e2080f3b073ae605cbb: Time spent bootstrapping tablet: real 0.022s user 0.019s sys 0.000s
I20250814 13:57:18.696483 12636 raft_consensus.cc:357] T ab93eceae8144cc0b69dbb5a3813f8fe P f2319463db2f4e2080f3b073ae605cbb [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" member_type: VOTER last_known_addr { host: "127.11.132.132" port: 38297 } } peers { permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 } } peers { permanent_uuid: "f2319463db2f4e2080f3b073ae605cbb" member_type: NON_VOTER last_known_addr { host: "127.11.132.130" port: 42573 } attrs { promote: true } } unsafe_config_change: true
I20250814 13:57:18.697333 12636 raft_consensus.cc:738] T ab93eceae8144cc0b69dbb5a3813f8fe P f2319463db2f4e2080f3b073ae605cbb [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: f2319463db2f4e2080f3b073ae605cbb, State: Initialized, Role: LEARNER
I20250814 13:57:18.697880 12636 consensus_queue.cc:260] T ab93eceae8144cc0b69dbb5a3813f8fe P f2319463db2f4e2080f3b073ae605cbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" member_type: VOTER last_known_addr { host: "127.11.132.132" port: 38297 } } peers { permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 } } peers { permanent_uuid: "f2319463db2f4e2080f3b073ae605cbb" member_type: NON_VOTER last_known_addr { host: "127.11.132.130" port: 42573 } attrs { promote: true } } unsafe_config_change: true
I20250814 13:57:18.701198 12636 ts_tablet_manager.cc:1428] T ab93eceae8144cc0b69dbb5a3813f8fe P f2319463db2f4e2080f3b073ae605cbb: Time spent starting tablet: real 0.021s user 0.022s sys 0.002s
I20250814 13:57:18.702898 12404 tablet_copy_service.cc:342] P 5e01eb3a7be0476abe51f9b9efb101a7: Request end of tablet copy session f2319463db2f4e2080f3b073ae605cbb-ab93eceae8144cc0b69dbb5a3813f8fe received from {username='slave'} at 127.11.132.130:34465
I20250814 13:57:18.703311 12404 tablet_copy_service.cc:434] P 5e01eb3a7be0476abe51f9b9efb101a7: ending tablet copy session f2319463db2f4e2080f3b073ae605cbb-ab93eceae8144cc0b69dbb5a3813f8fe on tablet ab93eceae8144cc0b69dbb5a3813f8fe with peer f2319463db2f4e2080f3b073ae605cbb
W20250814 13:57:19.188184 12503 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet ab93eceae8144cc0b69dbb5a3813f8fe on TS 4b40806ed51a4879bb1918fb595551f5 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 4b40806ed51a4879bb1918fb595551f5
I20250814 13:57:19.193435 12117 raft_consensus.cc:1215] T ab93eceae8144cc0b69dbb5a3813f8fe P f2319463db2f4e2080f3b073ae605cbb [term 4 LEARNER]: Deduplicated request from leader. Original: 4.4->[4.5-4.5] Dedup: 4.5->[]
I20250814 13:57:19.749287 12643 raft_consensus.cc:1062] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7: attempting to promote NON_VOTER f2319463db2f4e2080f3b073ae605cbb to VOTER
I20250814 13:57:19.751469 12643 consensus_queue.cc:237] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" member_type: VOTER last_known_addr { host: "127.11.132.132" port: 38297 } } peers { permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 } } peers { permanent_uuid: "f2319463db2f4e2080f3b073ae605cbb" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42573 } attrs { promote: false } } unsafe_config_change: true
I20250814 13:57:19.764531 12116 raft_consensus.cc:1273] T ab93eceae8144cc0b69dbb5a3813f8fe P f2319463db2f4e2080f3b073ae605cbb [term 4 LEARNER]: Refusing update from remote peer 5e01eb3a7be0476abe51f9b9efb101a7: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250814 13:57:19.764751 11983 raft_consensus.cc:1273] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f [term 4 FOLLOWER]: Refusing update from remote peer 5e01eb3a7be0476abe51f9b9efb101a7: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250814 13:57:19.765852 12645 consensus_queue.cc:1035] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2319463db2f4e2080f3b073ae605cbb" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42573 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250814 13:57:19.766640 12643 consensus_queue.cc:1035] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250814 13:57:19.772523 12645 raft_consensus.cc:2953] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, f2319463db2f4e2080f3b073ae605cbb (127.11.132.130) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" member_type: VOTER last_known_addr { host: "127.11.132.132" port: 38297 } } peers { permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 } } peers { permanent_uuid: "f2319463db2f4e2080f3b073ae605cbb" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42573 } attrs { promote: false } } unsafe_config_change: true }
I20250814 13:57:19.775485 11983 raft_consensus.cc:2953] T ab93eceae8144cc0b69dbb5a3813f8fe P 071c488311ff42dc9d7dbed1bd39ea9f [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, f2319463db2f4e2080f3b073ae605cbb (127.11.132.130) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" member_type: VOTER last_known_addr { host: "127.11.132.132" port: 38297 } } peers { permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 } } peers { permanent_uuid: "f2319463db2f4e2080f3b073ae605cbb" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42573 } attrs { promote: false } } unsafe_config_change: true }
I20250814 13:57:19.775069 12116 raft_consensus.cc:2953] T ab93eceae8144cc0b69dbb5a3813f8fe P f2319463db2f4e2080f3b073ae605cbb [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, f2319463db2f4e2080f3b073ae605cbb (127.11.132.130) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" member_type: VOTER last_known_addr { host: "127.11.132.132" port: 38297 } } peers { permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 } } peers { permanent_uuid: "f2319463db2f4e2080f3b073ae605cbb" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42573 } attrs { promote: false } } unsafe_config_change: true }
I20250814 13:57:19.785745 12518 catalog_manager.cc:5582] T ab93eceae8144cc0b69dbb5a3813f8fe P 5e01eb3a7be0476abe51f9b9efb101a7 reported cstate change: config changed from index 5 to 6, f2319463db2f4e2080f3b073ae605cbb (127.11.132.130) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7" member_type: VOTER last_known_addr { host: "127.11.132.132" port: 38297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2319463db2f4e2080f3b073ae605cbb" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42573 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
I20250814 13:57:19.880410 11794 kudu-admin-test.cc:751] Waiting for Master to see new config...
I20250814 13:57:19.894837 11794 kudu-admin-test.cc:756] Tablet locations:
tablet_locations {
tablet_id: "ab93eceae8144cc0b69dbb5a3813f8fe"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: LEADER
}
interned_replicas {
ts_info_idx: 1
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "5e01eb3a7be0476abe51f9b9efb101a7"
rpc_addresses {
host: "127.11.132.132"
port: 38297
}
}
ts_infos {
permanent_uuid: "071c488311ff42dc9d7dbed1bd39ea9f"
rpc_addresses {
host: "127.11.132.129"
port: 41199
}
}
ts_infos {
permanent_uuid: "f2319463db2f4e2080f3b073ae605cbb"
rpc_addresses {
host: "127.11.132.130"
port: 42573
}
}
I20250814 13:57:19.899314 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 11900
I20250814 13:57:19.922930 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 12033
I20250814 13:57:19.945914 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 12299
I20250814 13:57:19.971500 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 12472
2025-08-14T13:57:19Z chronyd exiting
[ OK ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes (20280 ms)
[ RUN ] AdminCliTest.TestGracefulSpecificLeaderStepDown
I20250814 13:57:20.032261 11794 test_util.cc:276] Using random seed: -1495151120
I20250814 13:57:20.037873 11794 ts_itest-base.cc:115] Starting cluster with:
I20250814 13:57:20.038046 11794 ts_itest-base.cc:116] --------------
I20250814 13:57:20.038221 11794 ts_itest-base.cc:117] 3 tablet servers
I20250814 13:57:20.038401 11794 ts_itest-base.cc:118] 3 replicas per TS
I20250814 13:57:20.038537 11794 ts_itest-base.cc:119] --------------
2025-08-14T13:57:20Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-14T13:57:20Z Disabled control of system clock
I20250814 13:57:20.072108 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.190:36311
--webserver_interface=127.11.132.190
--webserver_port=0
--builtin_ntp_servers=127.11.132.148:37773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.132.190:36311
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250814 13:57:20.350356 12666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:57:20.350888 12666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:57:20.351269 12666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:57:20.380236 12666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250814 13:57:20.380497 12666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:57:20.380700 12666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250814 13:57:20.380885 12666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250814 13:57:20.414749 12666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:37773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.11.132.190:36311
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.190:36311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.11.132.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:57:20.415977 12666 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:57:20.417519 12666 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:57:20.427172 12672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:57:20.428658 12673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:57:21.831461 12671 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 12666
W20250814 13:57:22.132567 12666 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.704s user 0.002s sys 0.000s
W20250814 13:57:22.132927 12666 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.705s user 0.002s sys 0.000s
W20250814 13:57:22.136629 12674 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1704 milliseconds
W20250814 13:57:22.137873 12675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 13:57:22.137930 12666 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 13:57:22.139082 12666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:57:22.141237 12666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:57:22.142541 12666 hybrid_clock.cc:648] HybridClock initialized: now 1755179842142504 us; error 49 us; skew 500 ppm
I20250814 13:57:22.143249 12666 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:57:22.148727 12666 webserver.cc:480] Webserver started at http://127.11.132.190:34207/ using document root <none> and password file <none>
I20250814 13:57:22.149537 12666 fs_manager.cc:362] Metadata directory not provided
I20250814 13:57:22.149718 12666 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:57:22.150079 12666 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 13:57:22.154201 12666 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "b2315988d7ae42a7884c32c74d4324e9"
format_stamp: "Formatted at 2025-08-14 13:57:22 on dist-test-slave-q1tc"
I20250814 13:57:22.155210 12666 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "b2315988d7ae42a7884c32c74d4324e9"
format_stamp: "Formatted at 2025-08-14 13:57:22 on dist-test-slave-q1tc"
I20250814 13:57:22.161559 12666 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250814 13:57:22.166630 12682 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:57:22.167495 12666 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250814 13:57:22.167773 12666 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
uuid: "b2315988d7ae42a7884c32c74d4324e9"
format_stamp: "Formatted at 2025-08-14 13:57:22 on dist-test-slave-q1tc"
I20250814 13:57:22.168047 12666 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:57:22.232642 12666 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:57:22.234200 12666 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:57:22.234643 12666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:57:22.302379 12666 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.190:36311
I20250814 13:57:22.302438 12733 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.190:36311 every 8 connection(s)
I20250814 13:57:22.305032 12666 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250814 13:57:22.310053 12734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:57:22.315446 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 12666
I20250814 13:57:22.315819 11794 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250814 13:57:22.327836 12734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2315988d7ae42a7884c32c74d4324e9: Bootstrap starting.
I20250814 13:57:22.335129 12734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2315988d7ae42a7884c32c74d4324e9: Neither blocks nor log segments found. Creating new log.
I20250814 13:57:22.337457 12734 log.cc:826] T 00000000000000000000000000000000 P b2315988d7ae42a7884c32c74d4324e9: Log is configured to *not* fsync() on all Append() calls
I20250814 13:57:22.341897 12734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2315988d7ae42a7884c32c74d4324e9: No bootstrap required, opened a new log
I20250814 13:57:22.358506 12734 raft_consensus.cc:357] T 00000000000000000000000000000000 P b2315988d7ae42a7884c32c74d4324e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2315988d7ae42a7884c32c74d4324e9" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 36311 } }
I20250814 13:57:22.359231 12734 raft_consensus.cc:383] T 00000000000000000000000000000000 P b2315988d7ae42a7884c32c74d4324e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 13:57:22.359443 12734 raft_consensus.cc:738] T 00000000000000000000000000000000 P b2315988d7ae42a7884c32c74d4324e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2315988d7ae42a7884c32c74d4324e9, State: Initialized, Role: FOLLOWER
I20250814 13:57:22.360028 12734 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2315988d7ae42a7884c32c74d4324e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2315988d7ae42a7884c32c74d4324e9" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 36311 } }
I20250814 13:57:22.360502 12734 raft_consensus.cc:397] T 00000000000000000000000000000000 P b2315988d7ae42a7884c32c74d4324e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 13:57:22.360749 12734 raft_consensus.cc:491] T 00000000000000000000000000000000 P b2315988d7ae42a7884c32c74d4324e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 13:57:22.361035 12734 raft_consensus.cc:3058] T 00000000000000000000000000000000 P b2315988d7ae42a7884c32c74d4324e9 [term 0 FOLLOWER]: Advancing to term 1
I20250814 13:57:22.364928 12734 raft_consensus.cc:513] T 00000000000000000000000000000000 P b2315988d7ae42a7884c32c74d4324e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2315988d7ae42a7884c32c74d4324e9" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 36311 } }
I20250814 13:57:22.365646 12734 leader_election.cc:304] T 00000000000000000000000000000000 P b2315988d7ae42a7884c32c74d4324e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2315988d7ae42a7884c32c74d4324e9; no voters:
I20250814 13:57:22.367300 12734 leader_election.cc:290] T 00000000000000000000000000000000 P b2315988d7ae42a7884c32c74d4324e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250814 13:57:22.368077 12739 raft_consensus.cc:2802] T 00000000000000000000000000000000 P b2315988d7ae42a7884c32c74d4324e9 [term 1 FOLLOWER]: Leader election won for term 1
I20250814 13:57:22.370208 12739 raft_consensus.cc:695] T 00000000000000000000000000000000 P b2315988d7ae42a7884c32c74d4324e9 [term 1 LEADER]: Becoming Leader. State: Replica: b2315988d7ae42a7884c32c74d4324e9, State: Running, Role: LEADER
I20250814 13:57:22.370959 12739 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2315988d7ae42a7884c32c74d4324e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2315988d7ae42a7884c32c74d4324e9" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 36311 } }
I20250814 13:57:22.371925 12734 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2315988d7ae42a7884c32c74d4324e9 [sys.catalog]: configured and running, proceeding with master startup.
I20250814 13:57:22.381315 12740 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2315988d7ae42a7884c32c74d4324e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2315988d7ae42a7884c32c74d4324e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2315988d7ae42a7884c32c74d4324e9" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 36311 } } }
I20250814 13:57:22.381975 12740 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2315988d7ae42a7884c32c74d4324e9 [sys.catalog]: This master's current role is: LEADER
I20250814 13:57:22.382009 12741 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2315988d7ae42a7884c32c74d4324e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2315988d7ae42a7884c32c74d4324e9. Latest consensus state: current_term: 1 leader_uuid: "b2315988d7ae42a7884c32c74d4324e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2315988d7ae42a7884c32c74d4324e9" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 36311 } } }
I20250814 13:57:22.382817 12741 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2315988d7ae42a7884c32c74d4324e9 [sys.catalog]: This master's current role is: LEADER
I20250814 13:57:22.386878 12748 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250814 13:57:22.398952 12748 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250814 13:57:22.415113 12748 catalog_manager.cc:1349] Generated new cluster ID: 0e07d0df5f0942d2a7549b8848cd84b8
I20250814 13:57:22.415369 12748 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250814 13:57:22.440830 12748 catalog_manager.cc:1372] Generated new certificate authority record
I20250814 13:57:22.442595 12748 catalog_manager.cc:1506] Loading token signing keys...
I20250814 13:57:22.458665 12748 catalog_manager.cc:5955] T 00000000000000000000000000000000 P b2315988d7ae42a7884c32c74d4324e9: Generated new TSK 0
I20250814 13:57:22.459415 12748 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250814 13:57:22.484146 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.132.129:0
--local_ip_for_outbound_sockets=127.11.132.129
--webserver_interface=127.11.132.129
--webserver_port=0
--tserver_master_addrs=127.11.132.190:36311
--builtin_ntp_servers=127.11.132.148:37773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250814 13:57:22.769385 12758 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250814 13:57:22.769997 12758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:57:22.770244 12758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:57:22.770700 12758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:57:22.801260 12758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:57:22.802115 12758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.132.129
I20250814 13:57:22.835970 12758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:37773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.132.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.11.132.129
--webserver_port=0
--tserver_master_addrs=127.11.132.190:36311
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.132.129
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:57:22.837271 12758 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:57:22.838742 12758 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:57:22.853246 12764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:57:22.854599 12765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:57:24.158601 12758 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.307s user 0.528s sys 0.770s
W20250814 13:57:24.159014 12758 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.307s user 0.528s sys 0.770s
I20250814 13:57:24.165419 12758 server_base.cc:1047] running on GCE node
W20250814 13:57:24.166966 12769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 13:57:24.168085 12758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:57:24.170643 12758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:57:24.172041 12758 hybrid_clock.cc:648] HybridClock initialized: now 1755179844171984 us; error 58 us; skew 500 ppm
I20250814 13:57:24.172806 12758 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:57:24.180203 12758 webserver.cc:480] Webserver started at http://127.11.132.129:34637/ using document root <none> and password file <none>
I20250814 13:57:24.181043 12758 fs_manager.cc:362] Metadata directory not provided
I20250814 13:57:24.181272 12758 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:57:24.181677 12758 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 13:57:24.185849 12758 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "23ef25c60f6648b99641850f79708d68"
format_stamp: "Formatted at 2025-08-14 13:57:24 on dist-test-slave-q1tc"
I20250814 13:57:24.186864 12758 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "23ef25c60f6648b99641850f79708d68"
format_stamp: "Formatted at 2025-08-14 13:57:24 on dist-test-slave-q1tc"
I20250814 13:57:24.194562 12758 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20250814 13:57:24.200625 12774 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:57:24.201833 12758 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250814 13:57:24.202139 12758 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "23ef25c60f6648b99641850f79708d68"
format_stamp: "Formatted at 2025-08-14 13:57:24 on dist-test-slave-q1tc"
I20250814 13:57:24.202466 12758 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:57:24.267457 12758 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:57:24.268779 12758 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:57:24.269199 12758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:57:24.271993 12758 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 13:57:24.276635 12758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 13:57:24.276826 12758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 13:57:24.277050 12758 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 13:57:24.277230 12758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250814 13:57:24.439822 12758 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.129:39197
I20250814 13:57:24.439972 12886 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.129:39197 every 8 connection(s)
I20250814 13:57:24.442246 12758 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250814 13:57:24.445906 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 12758
I20250814 13:57:24.446372 11794 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250814 13:57:24.453933 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.132.130:0
--local_ip_for_outbound_sockets=127.11.132.130
--webserver_interface=127.11.132.130
--webserver_port=0
--tserver_master_addrs=127.11.132.190:36311
--builtin_ntp_servers=127.11.132.148:37773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250814 13:57:24.481207 12887 heartbeater.cc:344] Connected to a master server at 127.11.132.190:36311
I20250814 13:57:24.481736 12887 heartbeater.cc:461] Registering TS with master...
I20250814 13:57:24.482728 12887 heartbeater.cc:507] Master 127.11.132.190:36311 requested a full tablet report, sending...
I20250814 13:57:24.485201 12699 ts_manager.cc:194] Registered new tserver with Master: 23ef25c60f6648b99641850f79708d68 (127.11.132.129:39197)
I20250814 13:57:24.487061 12699 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.132.129:41553
W20250814 13:57:24.760309 12891 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250814 13:57:24.760910 12891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:57:24.761185 12891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:57:24.761646 12891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:57:24.790535 12891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:57:24.791296 12891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.132.130
I20250814 13:57:24.823246 12891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:37773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.132.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.11.132.130
--webserver_port=0
--tserver_master_addrs=127.11.132.190:36311
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.132.130
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:57:24.824514 12891 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:57:24.825994 12891 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:57:24.837471 12897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 13:57:25.490394 12887 heartbeater.cc:499] Master 127.11.132.190:36311 was elected leader, sending a full tablet report...
W20250814 13:57:24.838410 12898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:57:25.970108 12900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:57:25.971668 12899 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1130 milliseconds
I20250814 13:57:25.971747 12891 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 13:57:25.972931 12891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:57:25.974989 12891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:57:25.976307 12891 hybrid_clock.cc:648] HybridClock initialized: now 1755179845976269 us; error 43 us; skew 500 ppm
I20250814 13:57:25.977030 12891 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:57:25.983153 12891 webserver.cc:480] Webserver started at http://127.11.132.130:38545/ using document root <none> and password file <none>
I20250814 13:57:25.983968 12891 fs_manager.cc:362] Metadata directory not provided
I20250814 13:57:25.984205 12891 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:57:25.984642 12891 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 13:57:25.988663 12891 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "11ced0d3c04e40099f16f2fa321c0f3b"
format_stamp: "Formatted at 2025-08-14 13:57:25 on dist-test-slave-q1tc"
I20250814 13:57:25.989668 12891 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "11ced0d3c04e40099f16f2fa321c0f3b"
format_stamp: "Formatted at 2025-08-14 13:57:25 on dist-test-slave-q1tc"
I20250814 13:57:25.996062 12891 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250814 13:57:26.001420 12907 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:57:26.002343 12891 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250814 13:57:26.002620 12891 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "11ced0d3c04e40099f16f2fa321c0f3b"
format_stamp: "Formatted at 2025-08-14 13:57:25 on dist-test-slave-q1tc"
I20250814 13:57:26.002893 12891 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:57:26.051952 12891 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:57:26.053354 12891 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:57:26.053772 12891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:57:26.056164 12891 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 13:57:26.060031 12891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 13:57:26.060240 12891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 13:57:26.060492 12891 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 13:57:26.060629 12891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 13:57:26.189190 12891 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.130:44889
I20250814 13:57:26.189289 13019 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.130:44889 every 8 connection(s)
I20250814 13:57:26.191691 12891 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250814 13:57:26.195262 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 12891
I20250814 13:57:26.195761 11794 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250814 13:57:26.202443 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.132.131:0
--local_ip_for_outbound_sockets=127.11.132.131
--webserver_interface=127.11.132.131
--webserver_port=0
--tserver_master_addrs=127.11.132.190:36311
--builtin_ntp_servers=127.11.132.148:37773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250814 13:57:26.212531 13020 heartbeater.cc:344] Connected to a master server at 127.11.132.190:36311
I20250814 13:57:26.213142 13020 heartbeater.cc:461] Registering TS with master...
I20250814 13:57:26.214437 13020 heartbeater.cc:507] Master 127.11.132.190:36311 requested a full tablet report, sending...
I20250814 13:57:26.216872 12699 ts_manager.cc:194] Registered new tserver with Master: 11ced0d3c04e40099f16f2fa321c0f3b (127.11.132.130:44889)
I20250814 13:57:26.218822 12699 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.132.130:52957
W20250814 13:57:26.496250 13024 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250814 13:57:26.496951 13024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:57:26.497323 13024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:57:26.498039 13024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:57:26.527437 13024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:57:26.528175 13024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.132.131
I20250814 13:57:26.561273 13024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:37773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.132.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.11.132.131
--webserver_port=0
--tserver_master_addrs=127.11.132.190:36311
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.132.131
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:57:26.562436 13024 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:57:26.563875 13024 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:57:26.575039 13030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 13:57:27.222906 13020 heartbeater.cc:499] Master 127.11.132.190:36311 was elected leader, sending a full tablet report...
W20250814 13:57:26.575497 13031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:57:27.708424 13032 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1131 milliseconds
W20250814 13:57:27.708631 13024 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.133s user 0.371s sys 0.749s
W20250814 13:57:27.708990 13024 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.134s user 0.371s sys 0.749s
W20250814 13:57:27.709204 13033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 13:57:27.709309 13024 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 13:57:27.710930 13024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:57:27.713693 13024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:57:27.715214 13024 hybrid_clock.cc:648] HybridClock initialized: now 1755179847715158 us; error 42 us; skew 500 ppm
I20250814 13:57:27.716326 13024 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:57:27.724666 13024 webserver.cc:480] Webserver started at http://127.11.132.131:45207/ using document root <none> and password file <none>
I20250814 13:57:27.726068 13024 fs_manager.cc:362] Metadata directory not provided
I20250814 13:57:27.726346 13024 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:57:27.726931 13024 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 13:57:27.733819 13024 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "44cf2c78e87e48e58984f181c88c5f2c"
format_stamp: "Formatted at 2025-08-14 13:57:27 on dist-test-slave-q1tc"
I20250814 13:57:27.735396 13024 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "44cf2c78e87e48e58984f181c88c5f2c"
format_stamp: "Formatted at 2025-08-14 13:57:27 on dist-test-slave-q1tc"
I20250814 13:57:27.745131 13024 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.010s sys 0.001s
I20250814 13:57:27.752581 13040 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:57:27.753824 13024 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250814 13:57:27.754209 13024 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "44cf2c78e87e48e58984f181c88c5f2c"
format_stamp: "Formatted at 2025-08-14 13:57:27 on dist-test-slave-q1tc"
I20250814 13:57:27.754657 13024 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:57:27.844889 13024 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:57:27.846280 13024 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:57:27.846640 13024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:57:27.849028 13024 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 13:57:27.853013 13024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 13:57:27.853245 13024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250814 13:57:27.853444 13024 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 13:57:27.853574 13024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 13:57:27.979444 13024 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.131:40389
I20250814 13:57:27.979537 13152 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.131:40389 every 8 connection(s)
I20250814 13:57:27.981945 13024 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250814 13:57:27.992579 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 13024
I20250814 13:57:27.993044 11794 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250814 13:57:28.017279 13153 heartbeater.cc:344] Connected to a master server at 127.11.132.190:36311
I20250814 13:57:28.017683 13153 heartbeater.cc:461] Registering TS with master...
I20250814 13:57:28.018622 13153 heartbeater.cc:507] Master 127.11.132.190:36311 requested a full tablet report, sending...
I20250814 13:57:28.020501 12699 ts_manager.cc:194] Registered new tserver with Master: 44cf2c78e87e48e58984f181c88c5f2c (127.11.132.131:40389)
I20250814 13:57:28.022024 12699 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.132.131:38993
I20250814 13:57:28.026562 11794 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250814 13:57:28.057453 12699 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57586:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250814 13:57:28.075038 12699 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250814 13:57:28.121419 13088 tablet_service.cc:1468] Processing CreateTablet for tablet 957ede6934bf4f58a343031ac21099b5 (DEFAULT_TABLE table=TestTable [id=94c534ef16d14329a79fa0e10b5e77be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 13:57:28.123008 13088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 957ede6934bf4f58a343031ac21099b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:57:28.123050 12955 tablet_service.cc:1468] Processing CreateTablet for tablet 957ede6934bf4f58a343031ac21099b5 (DEFAULT_TABLE table=TestTable [id=94c534ef16d14329a79fa0e10b5e77be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 13:57:28.124982 12822 tablet_service.cc:1468] Processing CreateTablet for tablet 957ede6934bf4f58a343031ac21099b5 (DEFAULT_TABLE table=TestTable [id=94c534ef16d14329a79fa0e10b5e77be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 13:57:28.126281 12955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 957ede6934bf4f58a343031ac21099b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:57:28.126672 12822 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 957ede6934bf4f58a343031ac21099b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:57:28.147125 13172 tablet_bootstrap.cc:492] T 957ede6934bf4f58a343031ac21099b5 P 23ef25c60f6648b99641850f79708d68: Bootstrap starting.
I20250814 13:57:28.150715 13173 tablet_bootstrap.cc:492] T 957ede6934bf4f58a343031ac21099b5 P 44cf2c78e87e48e58984f181c88c5f2c: Bootstrap starting.
I20250814 13:57:28.156309 13172 tablet_bootstrap.cc:654] T 957ede6934bf4f58a343031ac21099b5 P 23ef25c60f6648b99641850f79708d68: Neither blocks nor log segments found. Creating new log.
I20250814 13:57:28.158195 13173 tablet_bootstrap.cc:654] T 957ede6934bf4f58a343031ac21099b5 P 44cf2c78e87e48e58984f181c88c5f2c: Neither blocks nor log segments found. Creating new log.
I20250814 13:57:28.158862 13174 tablet_bootstrap.cc:492] T 957ede6934bf4f58a343031ac21099b5 P 11ced0d3c04e40099f16f2fa321c0f3b: Bootstrap starting.
I20250814 13:57:28.159070 13172 log.cc:826] T 957ede6934bf4f58a343031ac21099b5 P 23ef25c60f6648b99641850f79708d68: Log is configured to *not* fsync() on all Append() calls
I20250814 13:57:28.160629 13173 log.cc:826] T 957ede6934bf4f58a343031ac21099b5 P 44cf2c78e87e48e58984f181c88c5f2c: Log is configured to *not* fsync() on all Append() calls
I20250814 13:57:28.164887 13172 tablet_bootstrap.cc:492] T 957ede6934bf4f58a343031ac21099b5 P 23ef25c60f6648b99641850f79708d68: No bootstrap required, opened a new log
I20250814 13:57:28.165409 13172 ts_tablet_manager.cc:1397] T 957ede6934bf4f58a343031ac21099b5 P 23ef25c60f6648b99641850f79708d68: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.007s
I20250814 13:57:28.165835 13174 tablet_bootstrap.cc:654] T 957ede6934bf4f58a343031ac21099b5 P 11ced0d3c04e40099f16f2fa321c0f3b: Neither blocks nor log segments found. Creating new log.
I20250814 13:57:28.168087 13173 tablet_bootstrap.cc:492] T 957ede6934bf4f58a343031ac21099b5 P 44cf2c78e87e48e58984f181c88c5f2c: No bootstrap required, opened a new log
I20250814 13:57:28.168114 13174 log.cc:826] T 957ede6934bf4f58a343031ac21099b5 P 11ced0d3c04e40099f16f2fa321c0f3b: Log is configured to *not* fsync() on all Append() calls
I20250814 13:57:28.168495 13173 ts_tablet_manager.cc:1397] T 957ede6934bf4f58a343031ac21099b5 P 44cf2c78e87e48e58984f181c88c5f2c: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.008s
I20250814 13:57:28.173645 13174 tablet_bootstrap.cc:492] T 957ede6934bf4f58a343031ac21099b5 P 11ced0d3c04e40099f16f2fa321c0f3b: No bootstrap required, opened a new log
I20250814 13:57:28.174106 13174 ts_tablet_manager.cc:1397] T 957ede6934bf4f58a343031ac21099b5 P 11ced0d3c04e40099f16f2fa321c0f3b: Time spent bootstrapping tablet: real 0.016s user 0.014s sys 0.000s
I20250814 13:57:28.186523 13173 raft_consensus.cc:357] T 957ede6934bf4f58a343031ac21099b5 P 44cf2c78e87e48e58984f181c88c5f2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cf2c78e87e48e58984f181c88c5f2c" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 40389 } } peers { permanent_uuid: "11ced0d3c04e40099f16f2fa321c0f3b" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 44889 } } peers { permanent_uuid: "23ef25c60f6648b99641850f79708d68" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 39197 } }
I20250814 13:57:28.187489 13173 raft_consensus.cc:738] T 957ede6934bf4f58a343031ac21099b5 P 44cf2c78e87e48e58984f181c88c5f2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44cf2c78e87e48e58984f181c88c5f2c, State: Initialized, Role: FOLLOWER
I20250814 13:57:28.188357 13173 consensus_queue.cc:260] T 957ede6934bf4f58a343031ac21099b5 P 44cf2c78e87e48e58984f181c88c5f2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cf2c78e87e48e58984f181c88c5f2c" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 40389 } } peers { permanent_uuid: "11ced0d3c04e40099f16f2fa321c0f3b" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 44889 } } peers { permanent_uuid: "23ef25c60f6648b99641850f79708d68" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 39197 } }
I20250814 13:57:28.191936 13172 raft_consensus.cc:357] T 957ede6934bf4f58a343031ac21099b5 P 23ef25c60f6648b99641850f79708d68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cf2c78e87e48e58984f181c88c5f2c" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 40389 } } peers { permanent_uuid: "11ced0d3c04e40099f16f2fa321c0f3b" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 44889 } } peers { permanent_uuid: "23ef25c60f6648b99641850f79708d68" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 39197 } }
I20250814 13:57:28.192829 13172 raft_consensus.cc:738] T 957ede6934bf4f58a343031ac21099b5 P 23ef25c60f6648b99641850f79708d68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ef25c60f6648b99641850f79708d68, State: Initialized, Role: FOLLOWER
I20250814 13:57:28.193609 13172 consensus_queue.cc:260] T 957ede6934bf4f58a343031ac21099b5 P 23ef25c60f6648b99641850f79708d68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cf2c78e87e48e58984f181c88c5f2c" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 40389 } } peers { permanent_uuid: "11ced0d3c04e40099f16f2fa321c0f3b" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 44889 } } peers { permanent_uuid: "23ef25c60f6648b99641850f79708d68" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 39197 } }
I20250814 13:57:28.202000 13174 raft_consensus.cc:357] T 957ede6934bf4f58a343031ac21099b5 P 11ced0d3c04e40099f16f2fa321c0f3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cf2c78e87e48e58984f181c88c5f2c" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 40389 } } peers { permanent_uuid: "11ced0d3c04e40099f16f2fa321c0f3b" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 44889 } } peers { permanent_uuid: "23ef25c60f6648b99641850f79708d68" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 39197 } }
I20250814 13:57:28.203231 13174 raft_consensus.cc:738] T 957ede6934bf4f58a343031ac21099b5 P 11ced0d3c04e40099f16f2fa321c0f3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11ced0d3c04e40099f16f2fa321c0f3b, State: Initialized, Role: FOLLOWER
I20250814 13:57:28.204209 13174 consensus_queue.cc:260] T 957ede6934bf4f58a343031ac21099b5 P 11ced0d3c04e40099f16f2fa321c0f3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cf2c78e87e48e58984f181c88c5f2c" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 40389 } } peers { permanent_uuid: "11ced0d3c04e40099f16f2fa321c0f3b" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 44889 } } peers { permanent_uuid: "23ef25c60f6648b99641850f79708d68" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 39197 } }
I20250814 13:57:28.211387 13172 ts_tablet_manager.cc:1428] T 957ede6934bf4f58a343031ac21099b5 P 23ef25c60f6648b99641850f79708d68: Time spent starting tablet: real 0.046s user 0.038s sys 0.006s
I20250814 13:57:28.213032 13173 ts_tablet_manager.cc:1428] T 957ede6934bf4f58a343031ac21099b5 P 44cf2c78e87e48e58984f181c88c5f2c: Time spent starting tablet: real 0.044s user 0.024s sys 0.020s
I20250814 13:57:28.213140 13153 heartbeater.cc:499] Master 127.11.132.190:36311 was elected leader, sending a full tablet report...
W20250814 13:57:28.217074 12888 tablet.cc:2378] T 957ede6934bf4f58a343031ac21099b5 P 23ef25c60f6648b99641850f79708d68: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250814 13:57:28.219470 13174 ts_tablet_manager.cc:1428] T 957ede6934bf4f58a343031ac21099b5 P 11ced0d3c04e40099f16f2fa321c0f3b: Time spent starting tablet: real 0.045s user 0.034s sys 0.008s
W20250814 13:57:28.236307 13154 tablet.cc:2378] T 957ede6934bf4f58a343031ac21099b5 P 44cf2c78e87e48e58984f181c88c5f2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250814 13:57:28.253005 11794 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250814 13:57:28.256004 11794 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 23ef25c60f6648b99641850f79708d68 to finish bootstrapping
I20250814 13:57:28.268224 11794 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 11ced0d3c04e40099f16f2fa321c0f3b to finish bootstrapping
I20250814 13:57:28.277379 11794 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 44cf2c78e87e48e58984f181c88c5f2c to finish bootstrapping
I20250814 13:57:28.317596 12842 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "957ede6934bf4f58a343031ac21099b5"
dest_uuid: "23ef25c60f6648b99641850f79708d68"
from {username='slave'} at 127.0.0.1:38632
I20250814 13:57:28.318254 12842 raft_consensus.cc:491] T 957ede6934bf4f58a343031ac21099b5 P 23ef25c60f6648b99641850f79708d68 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250814 13:57:28.318627 12842 raft_consensus.cc:3058] T 957ede6934bf4f58a343031ac21099b5 P 23ef25c60f6648b99641850f79708d68 [term 0 FOLLOWER]: Advancing to term 1
I20250814 13:57:28.325026 12842 raft_consensus.cc:513] T 957ede6934bf4f58a343031ac21099b5 P 23ef25c60f6648b99641850f79708d68 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cf2c78e87e48e58984f181c88c5f2c" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 40389 } } peers { permanent_uuid: "11ced0d3c04e40099f16f2fa321c0f3b" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 44889 } } peers { permanent_uuid: "23ef25c60f6648b99641850f79708d68" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 39197 } }
I20250814 13:57:28.327822 12842 leader_election.cc:290] T 957ede6934bf4f58a343031ac21099b5 P 23ef25c60f6648b99641850f79708d68 [CANDIDATE]: Term 1 election: Requested vote from peers 44cf2c78e87e48e58984f181c88c5f2c (127.11.132.131:40389), 11ced0d3c04e40099f16f2fa321c0f3b (127.11.132.130:44889)
I20250814 13:57:28.338290 11794 cluster_itest_util.cc:257] Not converged past 1 yet: 0.0 0.0 0.0
I20250814 13:57:28.341797 12975 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "957ede6934bf4f58a343031ac21099b5" candidate_uuid: "23ef25c60f6648b99641850f79708d68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "11ced0d3c04e40099f16f2fa321c0f3b"
I20250814 13:57:28.342612 12975 raft_consensus.cc:3058] T 957ede6934bf4f58a343031ac21099b5 P 11ced0d3c04e40099f16f2fa321c0f3b [term 0 FOLLOWER]: Advancing to term 1
I20250814 13:57:28.343730 13108 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "957ede6934bf4f58a343031ac21099b5" candidate_uuid: "23ef25c60f6648b99641850f79708d68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "44cf2c78e87e48e58984f181c88c5f2c"
I20250814 13:57:28.344499 13108 raft_consensus.cc:3058] T 957ede6934bf4f58a343031ac21099b5 P 44cf2c78e87e48e58984f181c88c5f2c [term 0 FOLLOWER]: Advancing to term 1
I20250814 13:57:28.349277 12975 raft_consensus.cc:2466] T 957ede6934bf4f58a343031ac21099b5 P 11ced0d3c04e40099f16f2fa321c0f3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23ef25c60f6648b99641850f79708d68 in term 1.
I20250814 13:57:28.350462 12778 leader_election.cc:304] T 957ede6934bf4f58a343031ac21099b5 P 23ef25c60f6648b99641850f79708d68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11ced0d3c04e40099f16f2fa321c0f3b, 23ef25c60f6648b99641850f79708d68; no voters:
I20250814 13:57:28.350873 13108 raft_consensus.cc:2466] T 957ede6934bf4f58a343031ac21099b5 P 44cf2c78e87e48e58984f181c88c5f2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23ef25c60f6648b99641850f79708d68 in term 1.
I20250814 13:57:28.351451 13179 raft_consensus.cc:2802] T 957ede6934bf4f58a343031ac21099b5 P 23ef25c60f6648b99641850f79708d68 [term 1 FOLLOWER]: Leader election won for term 1
I20250814 13:57:28.353330 13179 raft_consensus.cc:695] T 957ede6934bf4f58a343031ac21099b5 P 23ef25c60f6648b99641850f79708d68 [term 1 LEADER]: Becoming Leader. State: Replica: 23ef25c60f6648b99641850f79708d68, State: Running, Role: LEADER
I20250814 13:57:28.354236 13179 consensus_queue.cc:237] T 957ede6934bf4f58a343031ac21099b5 P 23ef25c60f6648b99641850f79708d68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cf2c78e87e48e58984f181c88c5f2c" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 40389 } } peers { permanent_uuid: "11ced0d3c04e40099f16f2fa321c0f3b" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 44889 } } peers { permanent_uuid: "23ef25c60f6648b99641850f79708d68" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 39197 } }
I20250814 13:57:28.364768 12698 catalog_manager.cc:5582] T 957ede6934bf4f58a343031ac21099b5 P 23ef25c60f6648b99641850f79708d68 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23ef25c60f6648b99641850f79708d68 (127.11.132.129). New cstate: current_term: 1 leader_uuid: "23ef25c60f6648b99641850f79708d68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cf2c78e87e48e58984f181c88c5f2c" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 40389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11ced0d3c04e40099f16f2fa321c0f3b" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 44889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23ef25c60f6648b99641850f79708d68" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 39197 } health_report { overall_health: HEALTHY } } }
I20250814 13:57:28.443862 11794 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
W20250814 13:57:28.448053 13021 tablet.cc:2378] T 957ede6934bf4f58a343031ac21099b5 P 11ced0d3c04e40099f16f2fa321c0f3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250814 13:57:28.648826 11794 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250814 13:57:28.867515 13187 consensus_queue.cc:1035] T 957ede6934bf4f58a343031ac21099b5 P 23ef25c60f6648b99641850f79708d68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11ced0d3c04e40099f16f2fa321c0f3b" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 44889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250814 13:57:28.884593 13187 consensus_queue.cc:1035] T 957ede6934bf4f58a343031ac21099b5 P 23ef25c60f6648b99641850f79708d68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44cf2c78e87e48e58984f181c88c5f2c" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 40389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250814 13:57:30.595481 12842 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "957ede6934bf4f58a343031ac21099b5"
dest_uuid: "23ef25c60f6648b99641850f79708d68"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54686
I20250814 13:57:30.596145 12842 raft_consensus.cc:604] T 957ede6934bf4f58a343031ac21099b5 P 23ef25c60f6648b99641850f79708d68 [term 1 LEADER]: Received request to transfer leadership
I20250814 13:57:30.612974 13222 raft_consensus.cc:991] T 957ede6934bf4f58a343031ac21099b5 P 23ef25c60f6648b99641850f79708d68: : Instructing follower 44cf2c78e87e48e58984f181c88c5f2c to start an election
I20250814 13:57:30.613406 13222 raft_consensus.cc:1079] T 957ede6934bf4f58a343031ac21099b5 P 23ef25c60f6648b99641850f79708d68 [term 1 LEADER]: Signalling peer 44cf2c78e87e48e58984f181c88c5f2c to start an election
I20250814 13:57:30.614843 13108 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "957ede6934bf4f58a343031ac21099b5"
dest_uuid: "44cf2c78e87e48e58984f181c88c5f2c"
from {username='slave'} at 127.11.132.129:35267
I20250814 13:57:30.615499 13108 raft_consensus.cc:491] T 957ede6934bf4f58a343031ac21099b5 P 44cf2c78e87e48e58984f181c88c5f2c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250814 13:57:30.615897 13108 raft_consensus.cc:3058] T 957ede6934bf4f58a343031ac21099b5 P 44cf2c78e87e48e58984f181c88c5f2c [term 1 FOLLOWER]: Advancing to term 2
I20250814 13:57:30.622936 13108 raft_consensus.cc:513] T 957ede6934bf4f58a343031ac21099b5 P 44cf2c78e87e48e58984f181c88c5f2c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cf2c78e87e48e58984f181c88c5f2c" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 40389 } } peers { permanent_uuid: "11ced0d3c04e40099f16f2fa321c0f3b" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 44889 } } peers { permanent_uuid: "23ef25c60f6648b99641850f79708d68" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 39197 } }
I20250814 13:57:30.625865 13108 leader_election.cc:290] T 957ede6934bf4f58a343031ac21099b5 P 44cf2c78e87e48e58984f181c88c5f2c [CANDIDATE]: Term 2 election: Requested vote from peers 11ced0d3c04e40099f16f2fa321c0f3b (127.11.132.130:44889), 23ef25c60f6648b99641850f79708d68 (127.11.132.129:39197)
I20250814 13:57:30.637931 12975 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "957ede6934bf4f58a343031ac21099b5" candidate_uuid: "44cf2c78e87e48e58984f181c88c5f2c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "11ced0d3c04e40099f16f2fa321c0f3b"
I20250814 13:57:30.638506 12975 raft_consensus.cc:3058] T 957ede6934bf4f58a343031ac21099b5 P 11ced0d3c04e40099f16f2fa321c0f3b [term 1 FOLLOWER]: Advancing to term 2
I20250814 13:57:30.639690 12842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "957ede6934bf4f58a343031ac21099b5" candidate_uuid: "44cf2c78e87e48e58984f181c88c5f2c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "23ef25c60f6648b99641850f79708d68"
I20250814 13:57:30.640280 12842 raft_consensus.cc:3053] T 957ede6934bf4f58a343031ac21099b5 P 23ef25c60f6648b99641850f79708d68 [term 1 LEADER]: Stepping down as leader of term 1
I20250814 13:57:30.640563 12842 raft_consensus.cc:738] T 957ede6934bf4f58a343031ac21099b5 P 23ef25c60f6648b99641850f79708d68 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 23ef25c60f6648b99641850f79708d68, State: Running, Role: LEADER
I20250814 13:57:30.641263 12842 consensus_queue.cc:260] T 957ede6934bf4f58a343031ac21099b5 P 23ef25c60f6648b99641850f79708d68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cf2c78e87e48e58984f181c88c5f2c" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 40389 } } peers { permanent_uuid: "11ced0d3c04e40099f16f2fa321c0f3b" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 44889 } } peers { permanent_uuid: "23ef25c60f6648b99641850f79708d68" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 39197 } }
I20250814 13:57:30.642463 12842 raft_consensus.cc:3058] T 957ede6934bf4f58a343031ac21099b5 P 23ef25c60f6648b99641850f79708d68 [term 1 FOLLOWER]: Advancing to term 2
I20250814 13:57:30.644467 12975 raft_consensus.cc:2466] T 957ede6934bf4f58a343031ac21099b5 P 11ced0d3c04e40099f16f2fa321c0f3b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44cf2c78e87e48e58984f181c88c5f2c in term 2.
I20250814 13:57:30.645602 13044 leader_election.cc:304] T 957ede6934bf4f58a343031ac21099b5 P 44cf2c78e87e48e58984f181c88c5f2c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11ced0d3c04e40099f16f2fa321c0f3b, 44cf2c78e87e48e58984f181c88c5f2c; no voters:
I20250814 13:57:30.647607 13226 raft_consensus.cc:2802] T 957ede6934bf4f58a343031ac21099b5 P 44cf2c78e87e48e58984f181c88c5f2c [term 2 FOLLOWER]: Leader election won for term 2
I20250814 13:57:30.648371 12842 raft_consensus.cc:2466] T 957ede6934bf4f58a343031ac21099b5 P 23ef25c60f6648b99641850f79708d68 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44cf2c78e87e48e58984f181c88c5f2c in term 2.
I20250814 13:57:30.649149 13226 raft_consensus.cc:695] T 957ede6934bf4f58a343031ac21099b5 P 44cf2c78e87e48e58984f181c88c5f2c [term 2 LEADER]: Becoming Leader. State: Replica: 44cf2c78e87e48e58984f181c88c5f2c, State: Running, Role: LEADER
I20250814 13:57:30.649984 13226 consensus_queue.cc:237] T 957ede6934bf4f58a343031ac21099b5 P 44cf2c78e87e48e58984f181c88c5f2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cf2c78e87e48e58984f181c88c5f2c" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 40389 } } peers { permanent_uuid: "11ced0d3c04e40099f16f2fa321c0f3b" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 44889 } } peers { permanent_uuid: "23ef25c60f6648b99641850f79708d68" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 39197 } }
I20250814 13:57:30.656984 12699 catalog_manager.cc:5582] T 957ede6934bf4f58a343031ac21099b5 P 44cf2c78e87e48e58984f181c88c5f2c reported cstate change: term changed from 1 to 2, leader changed from 23ef25c60f6648b99641850f79708d68 (127.11.132.129) to 44cf2c78e87e48e58984f181c88c5f2c (127.11.132.131). New cstate: current_term: 2 leader_uuid: "44cf2c78e87e48e58984f181c88c5f2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cf2c78e87e48e58984f181c88c5f2c" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 40389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11ced0d3c04e40099f16f2fa321c0f3b" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 44889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23ef25c60f6648b99641850f79708d68" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 39197 } health_report { overall_health: UNKNOWN } } }
I20250814 13:57:31.081985 12842 raft_consensus.cc:1273] T 957ede6934bf4f58a343031ac21099b5 P 23ef25c60f6648b99641850f79708d68 [term 2 FOLLOWER]: Refusing update from remote peer 44cf2c78e87e48e58984f181c88c5f2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250814 13:57:31.083101 13226 consensus_queue.cc:1035] T 957ede6934bf4f58a343031ac21099b5 P 44cf2c78e87e48e58984f181c88c5f2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ef25c60f6648b99641850f79708d68" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 39197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250814 13:57:31.092417 12975 raft_consensus.cc:1273] T 957ede6934bf4f58a343031ac21099b5 P 11ced0d3c04e40099f16f2fa321c0f3b [term 2 FOLLOWER]: Refusing update from remote peer 44cf2c78e87e48e58984f181c88c5f2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250814 13:57:31.094053 13227 consensus_queue.cc:1035] T 957ede6934bf4f58a343031ac21099b5 P 44cf2c78e87e48e58984f181c88c5f2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "11ced0d3c04e40099f16f2fa321c0f3b" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 44889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250814 13:57:33.263257 12842 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "957ede6934bf4f58a343031ac21099b5"
dest_uuid: "23ef25c60f6648b99641850f79708d68"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54698
I20250814 13:57:33.263890 12842 raft_consensus.cc:604] T 957ede6934bf4f58a343031ac21099b5 P 23ef25c60f6648b99641850f79708d68 [term 2 FOLLOWER]: Received request to transfer leadership
I20250814 13:57:33.264190 12842 raft_consensus.cc:612] T 957ede6934bf4f58a343031ac21099b5 P 23ef25c60f6648b99641850f79708d68 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250814 13:57:34.303027 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 12758
I20250814 13:57:34.336452 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 12891
I20250814 13:57:34.360123 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 13024
I20250814 13:57:34.386677 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 12666
2025-08-14T13:57:34Z chronyd exiting
[ OK ] AdminCliTest.TestGracefulSpecificLeaderStepDown (14407 ms)
[ RUN ] AdminCliTest.TestDescribeTableColumnFlags
I20250814 13:57:34.439601 11794 test_util.cc:276] Using random seed: -1480743785
I20250814 13:57:34.443394 11794 ts_itest-base.cc:115] Starting cluster with:
I20250814 13:57:34.443557 11794 ts_itest-base.cc:116] --------------
I20250814 13:57:34.443706 11794 ts_itest-base.cc:117] 3 tablet servers
I20250814 13:57:34.443873 11794 ts_itest-base.cc:118] 3 replicas per TS
I20250814 13:57:34.444016 11794 ts_itest-base.cc:119] --------------
2025-08-14T13:57:34Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-14T13:57:34Z Disabled control of system clock
I20250814 13:57:34.478008 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.190:43725
--webserver_interface=127.11.132.190
--webserver_port=0
--builtin_ntp_servers=127.11.132.148:43231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.132.190:43725 with env {}
W20250814 13:57:34.758499 13270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:57:34.759097 13270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:57:34.759553 13270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:57:34.789871 13270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250814 13:57:34.790177 13270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:57:34.790443 13270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250814 13:57:34.790685 13270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250814 13:57:34.823369 13270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:43231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.132.190:43725
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.190:43725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.11.132.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:57:34.824712 13270 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:57:34.826241 13270 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:57:34.835345 13276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:57:34.836740 13277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:57:35.967171 13279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:57:35.969551 13278 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1128 milliseconds
I20250814 13:57:35.969640 13270 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 13:57:35.970847 13270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:57:35.973250 13270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:57:35.974566 13270 hybrid_clock.cc:648] HybridClock initialized: now 1755179855974526 us; error 43 us; skew 500 ppm
I20250814 13:57:35.975327 13270 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:57:35.980943 13270 webserver.cc:480] Webserver started at http://127.11.132.190:40597/ using document root <none> and password file <none>
I20250814 13:57:35.981871 13270 fs_manager.cc:362] Metadata directory not provided
I20250814 13:57:35.982096 13270 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:57:35.982515 13270 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 13:57:35.986748 13270 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "03ed323512ab4168ad3b8972e3997170"
format_stamp: "Formatted at 2025-08-14 13:57:35 on dist-test-slave-q1tc"
I20250814 13:57:35.987766 13270 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "03ed323512ab4168ad3b8972e3997170"
format_stamp: "Formatted at 2025-08-14 13:57:35 on dist-test-slave-q1tc"
I20250814 13:57:35.994514 13270 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20250814 13:57:35.999636 13286 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:57:36.000666 13270 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250814 13:57:36.000982 13270 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
uuid: "03ed323512ab4168ad3b8972e3997170"
format_stamp: "Formatted at 2025-08-14 13:57:35 on dist-test-slave-q1tc"
I20250814 13:57:36.001354 13270 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:57:36.068527 13270 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:57:36.069973 13270 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:57:36.070389 13270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:57:36.138278 13270 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.190:43725
I20250814 13:57:36.138353 13337 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.190:43725 every 8 connection(s)
I20250814 13:57:36.140815 13270 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250814 13:57:36.145612 13338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:57:36.150952 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 13270
I20250814 13:57:36.151500 11794 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250814 13:57:36.170559 13338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03ed323512ab4168ad3b8972e3997170: Bootstrap starting.
I20250814 13:57:36.177100 13338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03ed323512ab4168ad3b8972e3997170: Neither blocks nor log segments found. Creating new log.
I20250814 13:57:36.178732 13338 log.cc:826] T 00000000000000000000000000000000 P 03ed323512ab4168ad3b8972e3997170: Log is configured to *not* fsync() on all Append() calls
I20250814 13:57:36.183027 13338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03ed323512ab4168ad3b8972e3997170: No bootstrap required, opened a new log
I20250814 13:57:36.199927 13338 raft_consensus.cc:357] T 00000000000000000000000000000000 P 03ed323512ab4168ad3b8972e3997170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03ed323512ab4168ad3b8972e3997170" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 43725 } }
I20250814 13:57:36.200542 13338 raft_consensus.cc:383] T 00000000000000000000000000000000 P 03ed323512ab4168ad3b8972e3997170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 13:57:36.200845 13338 raft_consensus.cc:738] T 00000000000000000000000000000000 P 03ed323512ab4168ad3b8972e3997170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ed323512ab4168ad3b8972e3997170, State: Initialized, Role: FOLLOWER
I20250814 13:57:36.201574 13338 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03ed323512ab4168ad3b8972e3997170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03ed323512ab4168ad3b8972e3997170" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 43725 } }
I20250814 13:57:36.202102 13338 raft_consensus.cc:397] T 00000000000000000000000000000000 P 03ed323512ab4168ad3b8972e3997170 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 13:57:36.202351 13338 raft_consensus.cc:491] T 00000000000000000000000000000000 P 03ed323512ab4168ad3b8972e3997170 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 13:57:36.202656 13338 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 03ed323512ab4168ad3b8972e3997170 [term 0 FOLLOWER]: Advancing to term 1
I20250814 13:57:36.206552 13338 raft_consensus.cc:513] T 00000000000000000000000000000000 P 03ed323512ab4168ad3b8972e3997170 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03ed323512ab4168ad3b8972e3997170" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 43725 } }
I20250814 13:57:36.207193 13338 leader_election.cc:304] T 00000000000000000000000000000000 P 03ed323512ab4168ad3b8972e3997170 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03ed323512ab4168ad3b8972e3997170; no voters:
I20250814 13:57:36.208732 13338 leader_election.cc:290] T 00000000000000000000000000000000 P 03ed323512ab4168ad3b8972e3997170 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250814 13:57:36.209619 13343 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 03ed323512ab4168ad3b8972e3997170 [term 1 FOLLOWER]: Leader election won for term 1
I20250814 13:57:36.212577 13338 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03ed323512ab4168ad3b8972e3997170 [sys.catalog]: configured and running, proceeding with master startup.
I20250814 13:57:36.212522 13343 raft_consensus.cc:695] T 00000000000000000000000000000000 P 03ed323512ab4168ad3b8972e3997170 [term 1 LEADER]: Becoming Leader. State: Replica: 03ed323512ab4168ad3b8972e3997170, State: Running, Role: LEADER
I20250814 13:57:36.213326 13343 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03ed323512ab4168ad3b8972e3997170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03ed323512ab4168ad3b8972e3997170" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 43725 } }
I20250814 13:57:36.227294 13344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03ed323512ab4168ad3b8972e3997170 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03ed323512ab4168ad3b8972e3997170" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03ed323512ab4168ad3b8972e3997170" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 43725 } } }
I20250814 13:57:36.227957 13344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03ed323512ab4168ad3b8972e3997170 [sys.catalog]: This master's current role is: LEADER
I20250814 13:57:36.228770 13345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03ed323512ab4168ad3b8972e3997170 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03ed323512ab4168ad3b8972e3997170. Latest consensus state: current_term: 1 leader_uuid: "03ed323512ab4168ad3b8972e3997170" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03ed323512ab4168ad3b8972e3997170" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 43725 } } }
I20250814 13:57:36.229398 13345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03ed323512ab4168ad3b8972e3997170 [sys.catalog]: This master's current role is: LEADER
I20250814 13:57:36.233644 13353 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250814 13:57:36.245546 13353 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250814 13:57:36.263715 13353 catalog_manager.cc:1349] Generated new cluster ID: 750e5e50818347ae9a193e59fd6661fc
I20250814 13:57:36.264051 13353 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250814 13:57:36.286181 13353 catalog_manager.cc:1372] Generated new certificate authority record
I20250814 13:57:36.287920 13353 catalog_manager.cc:1506] Loading token signing keys...
I20250814 13:57:36.303516 13353 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 03ed323512ab4168ad3b8972e3997170: Generated new TSK 0
I20250814 13:57:36.304533 13353 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250814 13:57:36.325279 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.132.129:0
--local_ip_for_outbound_sockets=127.11.132.129
--webserver_interface=127.11.132.129
--webserver_port=0
--tserver_master_addrs=127.11.132.190:43725
--builtin_ntp_servers=127.11.132.148:43231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250814 13:57:36.614156 13362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:57:36.614663 13362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:57:36.615156 13362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:57:36.645901 13362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:57:36.646721 13362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.132.129
I20250814 13:57:36.680222 13362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:43231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.132.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.11.132.129
--webserver_port=0
--tserver_master_addrs=127.11.132.190:43725
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.132.129
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:57:36.681563 13362 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:57:36.683142 13362 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:57:36.695039 13368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:57:38.098156 13367 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 13362
W20250814 13:57:38.138414 13367 kernel_stack_watchdog.cc:198] Thread 13362 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 399ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250814 13:57:38.139443 13362 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.443s user 0.481s sys 0.919s
W20250814 13:57:38.139942 13362 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.443s user 0.482s sys 0.919s
W20250814 13:57:38.141479 13371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:57:36.696489 13369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:57:38.142308 13370 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1445 milliseconds
I20250814 13:57:38.142360 13362 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 13:57:38.147040 13362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:57:38.149637 13362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:57:38.151099 13362 hybrid_clock.cc:648] HybridClock initialized: now 1755179858151050 us; error 48 us; skew 500 ppm
I20250814 13:57:38.152119 13362 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:57:38.159173 13362 webserver.cc:480] Webserver started at http://127.11.132.129:45773/ using document root <none> and password file <none>
I20250814 13:57:38.160352 13362 fs_manager.cc:362] Metadata directory not provided
I20250814 13:57:38.160607 13362 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:57:38.161186 13362 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 13:57:38.167191 13362 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "18ed8d3c4a054b10a8d95e6f07d51216"
format_stamp: "Formatted at 2025-08-14 13:57:38 on dist-test-slave-q1tc"
I20250814 13:57:38.168561 13362 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "18ed8d3c4a054b10a8d95e6f07d51216"
format_stamp: "Formatted at 2025-08-14 13:57:38 on dist-test-slave-q1tc"
I20250814 13:57:38.177338 13362 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.002s
I20250814 13:57:38.184481 13378 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:57:38.185634 13362 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250814 13:57:38.186007 13362 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "18ed8d3c4a054b10a8d95e6f07d51216"
format_stamp: "Formatted at 2025-08-14 13:57:38 on dist-test-slave-q1tc"
I20250814 13:57:38.186435 13362 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:57:38.244805 13362 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:57:38.246191 13362 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:57:38.246587 13362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:57:38.249461 13362 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 13:57:38.253720 13362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 13:57:38.253893 13362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 13:57:38.254144 13362 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 13:57:38.254282 13362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 13:57:38.425468 13362 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.129:43095
I20250814 13:57:38.425587 13490 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.129:43095 every 8 connection(s)
I20250814 13:57:38.427968 13362 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250814 13:57:38.433235 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 13362
I20250814 13:57:38.433739 11794 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250814 13:57:38.441546 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.132.130:0
--local_ip_for_outbound_sockets=127.11.132.130
--webserver_interface=127.11.132.130
--webserver_port=0
--tserver_master_addrs=127.11.132.190:43725
--builtin_ntp_servers=127.11.132.148:43231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250814 13:57:38.468349 13491 heartbeater.cc:344] Connected to a master server at 127.11.132.190:43725
I20250814 13:57:38.468811 13491 heartbeater.cc:461] Registering TS with master...
I20250814 13:57:38.469983 13491 heartbeater.cc:507] Master 127.11.132.190:43725 requested a full tablet report, sending...
I20250814 13:57:38.472738 13303 ts_manager.cc:194] Registered new tserver with Master: 18ed8d3c4a054b10a8d95e6f07d51216 (127.11.132.129:43095)
I20250814 13:57:38.475299 13303 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.132.129:43365
W20250814 13:57:38.734040 13495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:57:38.734508 13495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:57:38.734982 13495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:57:38.765314 13495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:57:38.766142 13495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.132.130
I20250814 13:57:38.798635 13495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:43231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.132.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.11.132.130
--webserver_port=0
--tserver_master_addrs=127.11.132.190:43725
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.132.130
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:57:38.799903 13495 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:57:38.801391 13495 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:57:38.811774 13501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 13:57:39.479051 13491 heartbeater.cc:499] Master 127.11.132.190:43725 was elected leader, sending a full tablet report...
W20250814 13:57:38.813728 13502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:57:38.817291 13504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:57:39.979128 13503 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1162 milliseconds
I20250814 13:57:39.979241 13495 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 13:57:39.980358 13495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:57:39.982913 13495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:57:39.984326 13495 hybrid_clock.cc:648] HybridClock initialized: now 1755179859984293 us; error 54 us; skew 500 ppm
I20250814 13:57:39.985123 13495 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:57:39.991194 13495 webserver.cc:480] Webserver started at http://127.11.132.130:39009/ using document root <none> and password file <none>
I20250814 13:57:39.992027 13495 fs_manager.cc:362] Metadata directory not provided
I20250814 13:57:39.992242 13495 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:57:39.992671 13495 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 13:57:39.996758 13495 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "435806ef34b349968a8732e7662792ce"
format_stamp: "Formatted at 2025-08-14 13:57:39 on dist-test-slave-q1tc"
I20250814 13:57:39.997862 13495 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "435806ef34b349968a8732e7662792ce"
format_stamp: "Formatted at 2025-08-14 13:57:39 on dist-test-slave-q1tc"
I20250814 13:57:40.004281 13495 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250814 13:57:40.009572 13512 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:57:40.010490 13495 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20250814 13:57:40.010775 13495 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "435806ef34b349968a8732e7662792ce"
format_stamp: "Formatted at 2025-08-14 13:57:39 on dist-test-slave-q1tc"
I20250814 13:57:40.011086 13495 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:57:40.064122 13495 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:57:40.065763 13495 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:57:40.066192 13495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:57:40.068615 13495 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 13:57:40.072527 13495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 13:57:40.072741 13495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 13:57:40.072981 13495 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 13:57:40.073159 13495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
W20250814 13:57:40.201047 13487 debug-util.cc:398] Leaking SignalData structure 0x7b08000ac060 after lost signal to thread 13363
W20250814 13:57:40.202332 13487 debug-util.cc:398] Leaking SignalData structure 0x7b08000acf40 after lost signal to thread 13490
I20250814 13:57:40.202390 13495 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.130:40861
I20250814 13:57:40.202450 13624 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.130:40861 every 8 connection(s)
I20250814 13:57:40.206137 13495 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250814 13:57:40.216583 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 13495
I20250814 13:57:40.217307 11794 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250814 13:57:40.224797 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.132.131:0
--local_ip_for_outbound_sockets=127.11.132.131
--webserver_interface=127.11.132.131
--webserver_port=0
--tserver_master_addrs=127.11.132.190:43725
--builtin_ntp_servers=127.11.132.148:43231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250814 13:57:40.229909 13625 heartbeater.cc:344] Connected to a master server at 127.11.132.190:43725
I20250814 13:57:40.230387 13625 heartbeater.cc:461] Registering TS with master...
I20250814 13:57:40.231621 13625 heartbeater.cc:507] Master 127.11.132.190:43725 requested a full tablet report, sending...
I20250814 13:57:40.234163 13303 ts_manager.cc:194] Registered new tserver with Master: 435806ef34b349968a8732e7662792ce (127.11.132.130:40861)
I20250814 13:57:40.235910 13303 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.132.130:37227
W20250814 13:57:40.520714 13629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:57:40.521204 13629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:57:40.521695 13629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:57:40.553088 13629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:57:40.553941 13629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.132.131
I20250814 13:57:40.587466 13629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:43231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.132.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.11.132.131
--webserver_port=0
--tserver_master_addrs=127.11.132.190:43725
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.132.131
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:57:40.588846 13629 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:57:40.590435 13629 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:57:40.602149 13635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 13:57:41.239552 13625 heartbeater.cc:499] Master 127.11.132.190:43725 was elected leader, sending a full tablet report...
W20250814 13:57:42.005316 13634 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 13629
W20250814 13:57:40.604272 13636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:57:42.062211 13634 kernel_stack_watchdog.cc:198] Thread 13629 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 399ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250814 13:57:42.062578 13637 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250814 13:57:42.062917 13629 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.460s user 0.001s sys 0.000s
W20250814 13:57:42.063247 13629 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.460s user 0.001s sys 0.000s
I20250814 13:57:42.064554 13629 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250814 13:57:42.064644 13638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 13:57:42.067058 13629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:57:42.068905 13629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:57:42.070250 13629 hybrid_clock.cc:648] HybridClock initialized: now 1755179862070245 us; error 52 us; skew 500 ppm
I20250814 13:57:42.070967 13629 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:57:42.076586 13629 webserver.cc:480] Webserver started at http://127.11.132.131:43937/ using document root <none> and password file <none>
I20250814 13:57:42.077483 13629 fs_manager.cc:362] Metadata directory not provided
I20250814 13:57:42.077668 13629 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:57:42.078066 13629 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 13:57:42.082288 13629 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "d4bc58bf426f4f44ade6ed663e227552"
format_stamp: "Formatted at 2025-08-14 13:57:42 on dist-test-slave-q1tc"
I20250814 13:57:42.083420 13629 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "d4bc58bf426f4f44ade6ed663e227552"
format_stamp: "Formatted at 2025-08-14 13:57:42 on dist-test-slave-q1tc"
I20250814 13:57:42.090636 13629 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20250814 13:57:42.095772 13645 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:57:42.096665 13629 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250814 13:57:42.096976 13629 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "d4bc58bf426f4f44ade6ed663e227552"
format_stamp: "Formatted at 2025-08-14 13:57:42 on dist-test-slave-q1tc"
I20250814 13:57:42.097308 13629 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:57:42.155342 13629 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:57:42.156729 13629 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:57:42.157173 13629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:57:42.159580 13629 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 13:57:42.163519 13629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 13:57:42.163718 13629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 13:57:42.163980 13629 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 13:57:42.164144 13629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 13:57:42.300222 13629 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.131:37621
I20250814 13:57:42.300326 13757 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.131:37621 every 8 connection(s)
I20250814 13:57:42.302727 13629 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250814 13:57:42.312949 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 13629
I20250814 13:57:42.313400 11794 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250814 13:57:42.323488 13758 heartbeater.cc:344] Connected to a master server at 127.11.132.190:43725
I20250814 13:57:42.323917 13758 heartbeater.cc:461] Registering TS with master...
I20250814 13:57:42.324821 13758 heartbeater.cc:507] Master 127.11.132.190:43725 requested a full tablet report, sending...
I20250814 13:57:42.326858 13303 ts_manager.cc:194] Registered new tserver with Master: d4bc58bf426f4f44ade6ed663e227552 (127.11.132.131:37621)
I20250814 13:57:42.328020 13303 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.132.131:48029
I20250814 13:57:42.332671 11794 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250814 13:57:42.364354 13303 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59568:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250814 13:57:42.382477 13303 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250814 13:57:42.440148 13560 tablet_service.cc:1468] Processing CreateTablet for tablet d0f31a93d114413a84f457479b5bf438 (DEFAULT_TABLE table=TestTable [id=2cb776fc24e144a991ace15df098d244]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 13:57:42.442037 13560 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0f31a93d114413a84f457479b5bf438. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:57:42.442184 13693 tablet_service.cc:1468] Processing CreateTablet for tablet d0f31a93d114413a84f457479b5bf438 (DEFAULT_TABLE table=TestTable [id=2cb776fc24e144a991ace15df098d244]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 13:57:42.444092 13693 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0f31a93d114413a84f457479b5bf438. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:57:42.445075 13426 tablet_service.cc:1468] Processing CreateTablet for tablet d0f31a93d114413a84f457479b5bf438 (DEFAULT_TABLE table=TestTable [id=2cb776fc24e144a991ace15df098d244]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 13:57:42.447158 13426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0f31a93d114413a84f457479b5bf438. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:57:42.463057 13777 tablet_bootstrap.cc:492] T d0f31a93d114413a84f457479b5bf438 P 435806ef34b349968a8732e7662792ce: Bootstrap starting.
I20250814 13:57:42.470417 13777 tablet_bootstrap.cc:654] T d0f31a93d114413a84f457479b5bf438 P 435806ef34b349968a8732e7662792ce: Neither blocks nor log segments found. Creating new log.
I20250814 13:57:42.471661 13778 tablet_bootstrap.cc:492] T d0f31a93d114413a84f457479b5bf438 P d4bc58bf426f4f44ade6ed663e227552: Bootstrap starting.
I20250814 13:57:42.472635 13777 log.cc:826] T d0f31a93d114413a84f457479b5bf438 P 435806ef34b349968a8732e7662792ce: Log is configured to *not* fsync() on all Append() calls
I20250814 13:57:42.474152 13779 tablet_bootstrap.cc:492] T d0f31a93d114413a84f457479b5bf438 P 18ed8d3c4a054b10a8d95e6f07d51216: Bootstrap starting.
I20250814 13:57:42.479596 13777 tablet_bootstrap.cc:492] T d0f31a93d114413a84f457479b5bf438 P 435806ef34b349968a8732e7662792ce: No bootstrap required, opened a new log
I20250814 13:57:42.479980 13777 ts_tablet_manager.cc:1397] T d0f31a93d114413a84f457479b5bf438 P 435806ef34b349968a8732e7662792ce: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.005s
I20250814 13:57:42.480923 13778 tablet_bootstrap.cc:654] T d0f31a93d114413a84f457479b5bf438 P d4bc58bf426f4f44ade6ed663e227552: Neither blocks nor log segments found. Creating new log.
I20250814 13:57:42.481611 13779 tablet_bootstrap.cc:654] T d0f31a93d114413a84f457479b5bf438 P 18ed8d3c4a054b10a8d95e6f07d51216: Neither blocks nor log segments found. Creating new log.
I20250814 13:57:42.483163 13778 log.cc:826] T d0f31a93d114413a84f457479b5bf438 P d4bc58bf426f4f44ade6ed663e227552: Log is configured to *not* fsync() on all Append() calls
I20250814 13:57:42.483783 13779 log.cc:826] T d0f31a93d114413a84f457479b5bf438 P 18ed8d3c4a054b10a8d95e6f07d51216: Log is configured to *not* fsync() on all Append() calls
I20250814 13:57:42.489364 13779 tablet_bootstrap.cc:492] T d0f31a93d114413a84f457479b5bf438 P 18ed8d3c4a054b10a8d95e6f07d51216: No bootstrap required, opened a new log
I20250814 13:57:42.489467 13778 tablet_bootstrap.cc:492] T d0f31a93d114413a84f457479b5bf438 P d4bc58bf426f4f44ade6ed663e227552: No bootstrap required, opened a new log
I20250814 13:57:42.489791 13779 ts_tablet_manager.cc:1397] T d0f31a93d114413a84f457479b5bf438 P 18ed8d3c4a054b10a8d95e6f07d51216: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.013s
I20250814 13:57:42.489969 13778 ts_tablet_manager.cc:1397] T d0f31a93d114413a84f457479b5bf438 P d4bc58bf426f4f44ade6ed663e227552: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.008s
I20250814 13:57:42.503425 13777 raft_consensus.cc:357] T d0f31a93d114413a84f457479b5bf438 P 435806ef34b349968a8732e7662792ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4bc58bf426f4f44ade6ed663e227552" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 37621 } } peers { permanent_uuid: "435806ef34b349968a8732e7662792ce" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 40861 } } peers { permanent_uuid: "18ed8d3c4a054b10a8d95e6f07d51216" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 43095 } }
I20250814 13:57:42.504386 13777 raft_consensus.cc:383] T d0f31a93d114413a84f457479b5bf438 P 435806ef34b349968a8732e7662792ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 13:57:42.504688 13777 raft_consensus.cc:738] T d0f31a93d114413a84f457479b5bf438 P 435806ef34b349968a8732e7662792ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 435806ef34b349968a8732e7662792ce, State: Initialized, Role: FOLLOWER
I20250814 13:57:42.505641 13777 consensus_queue.cc:260] T d0f31a93d114413a84f457479b5bf438 P 435806ef34b349968a8732e7662792ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4bc58bf426f4f44ade6ed663e227552" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 37621 } } peers { permanent_uuid: "435806ef34b349968a8732e7662792ce" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 40861 } } peers { permanent_uuid: "18ed8d3c4a054b10a8d95e6f07d51216" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 43095 } }
I20250814 13:57:42.509675 13777 ts_tablet_manager.cc:1428] T d0f31a93d114413a84f457479b5bf438 P 435806ef34b349968a8732e7662792ce: Time spent starting tablet: real 0.029s user 0.026s sys 0.005s
I20250814 13:57:42.509814 13779 raft_consensus.cc:357] T d0f31a93d114413a84f457479b5bf438 P 18ed8d3c4a054b10a8d95e6f07d51216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4bc58bf426f4f44ade6ed663e227552" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 37621 } } peers { permanent_uuid: "435806ef34b349968a8732e7662792ce" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 40861 } } peers { permanent_uuid: "18ed8d3c4a054b10a8d95e6f07d51216" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 43095 } }
I20250814 13:57:42.510749 13779 raft_consensus.cc:383] T d0f31a93d114413a84f457479b5bf438 P 18ed8d3c4a054b10a8d95e6f07d51216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 13:57:42.510975 13779 raft_consensus.cc:738] T d0f31a93d114413a84f457479b5bf438 P 18ed8d3c4a054b10a8d95e6f07d51216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18ed8d3c4a054b10a8d95e6f07d51216, State: Initialized, Role: FOLLOWER
I20250814 13:57:42.511662 13779 consensus_queue.cc:260] T d0f31a93d114413a84f457479b5bf438 P 18ed8d3c4a054b10a8d95e6f07d51216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4bc58bf426f4f44ade6ed663e227552" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 37621 } } peers { permanent_uuid: "435806ef34b349968a8732e7662792ce" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 40861 } } peers { permanent_uuid: "18ed8d3c4a054b10a8d95e6f07d51216" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 43095 } }
I20250814 13:57:42.515627 13779 ts_tablet_manager.cc:1428] T d0f31a93d114413a84f457479b5bf438 P 18ed8d3c4a054b10a8d95e6f07d51216: Time spent starting tablet: real 0.026s user 0.017s sys 0.008s
I20250814 13:57:42.515443 13778 raft_consensus.cc:357] T d0f31a93d114413a84f457479b5bf438 P d4bc58bf426f4f44ade6ed663e227552 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4bc58bf426f4f44ade6ed663e227552" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 37621 } } peers { permanent_uuid: "435806ef34b349968a8732e7662792ce" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 40861 } } peers { permanent_uuid: "18ed8d3c4a054b10a8d95e6f07d51216" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 43095 } }
I20250814 13:57:42.516369 13778 raft_consensus.cc:383] T d0f31a93d114413a84f457479b5bf438 P d4bc58bf426f4f44ade6ed663e227552 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 13:57:42.516671 13778 raft_consensus.cc:738] T d0f31a93d114413a84f457479b5bf438 P d4bc58bf426f4f44ade6ed663e227552 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4bc58bf426f4f44ade6ed663e227552, State: Initialized, Role: FOLLOWER
I20250814 13:57:42.517381 13778 consensus_queue.cc:260] T d0f31a93d114413a84f457479b5bf438 P d4bc58bf426f4f44ade6ed663e227552 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4bc58bf426f4f44ade6ed663e227552" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 37621 } } peers { permanent_uuid: "435806ef34b349968a8732e7662792ce" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 40861 } } peers { permanent_uuid: "18ed8d3c4a054b10a8d95e6f07d51216" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 43095 } }
I20250814 13:57:42.521021 13758 heartbeater.cc:499] Master 127.11.132.190:43725 was elected leader, sending a full tablet report...
I20250814 13:57:42.521644 13778 ts_tablet_manager.cc:1428] T d0f31a93d114413a84f457479b5bf438 P d4bc58bf426f4f44ade6ed663e227552: Time spent starting tablet: real 0.031s user 0.021s sys 0.011s
W20250814 13:57:42.557406 13759 tablet.cc:2378] T d0f31a93d114413a84f457479b5bf438 P d4bc58bf426f4f44ade6ed663e227552: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250814 13:57:42.686957 13492 tablet.cc:2378] T d0f31a93d114413a84f457479b5bf438 P 18ed8d3c4a054b10a8d95e6f07d51216: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250814 13:57:42.714133 13626 tablet.cc:2378] T d0f31a93d114413a84f457479b5bf438 P 435806ef34b349968a8732e7662792ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250814 13:57:42.755669 13784 raft_consensus.cc:491] T d0f31a93d114413a84f457479b5bf438 P 18ed8d3c4a054b10a8d95e6f07d51216 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250814 13:57:42.756146 13784 raft_consensus.cc:513] T d0f31a93d114413a84f457479b5bf438 P 18ed8d3c4a054b10a8d95e6f07d51216 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4bc58bf426f4f44ade6ed663e227552" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 37621 } } peers { permanent_uuid: "435806ef34b349968a8732e7662792ce" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 40861 } } peers { permanent_uuid: "18ed8d3c4a054b10a8d95e6f07d51216" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 43095 } }
I20250814 13:57:42.758534 13784 leader_election.cc:290] T d0f31a93d114413a84f457479b5bf438 P 18ed8d3c4a054b10a8d95e6f07d51216 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4bc58bf426f4f44ade6ed663e227552 (127.11.132.131:37621), 435806ef34b349968a8732e7662792ce (127.11.132.130:40861)
I20250814 13:57:42.769295 13713 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0f31a93d114413a84f457479b5bf438" candidate_uuid: "18ed8d3c4a054b10a8d95e6f07d51216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4bc58bf426f4f44ade6ed663e227552" is_pre_election: true
I20250814 13:57:42.769311 13580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0f31a93d114413a84f457479b5bf438" candidate_uuid: "18ed8d3c4a054b10a8d95e6f07d51216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "435806ef34b349968a8732e7662792ce" is_pre_election: true
I20250814 13:57:42.769976 13580 raft_consensus.cc:2466] T d0f31a93d114413a84f457479b5bf438 P 435806ef34b349968a8732e7662792ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18ed8d3c4a054b10a8d95e6f07d51216 in term 0.
I20250814 13:57:42.769974 13713 raft_consensus.cc:2466] T d0f31a93d114413a84f457479b5bf438 P d4bc58bf426f4f44ade6ed663e227552 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18ed8d3c4a054b10a8d95e6f07d51216 in term 0.
I20250814 13:57:42.771015 13381 leader_election.cc:304] T d0f31a93d114413a84f457479b5bf438 P 18ed8d3c4a054b10a8d95e6f07d51216 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18ed8d3c4a054b10a8d95e6f07d51216, d4bc58bf426f4f44ade6ed663e227552; no voters:
I20250814 13:57:42.771677 13784 raft_consensus.cc:2802] T d0f31a93d114413a84f457479b5bf438 P 18ed8d3c4a054b10a8d95e6f07d51216 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250814 13:57:42.771910 13784 raft_consensus.cc:491] T d0f31a93d114413a84f457479b5bf438 P 18ed8d3c4a054b10a8d95e6f07d51216 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250814 13:57:42.772171 13784 raft_consensus.cc:3058] T d0f31a93d114413a84f457479b5bf438 P 18ed8d3c4a054b10a8d95e6f07d51216 [term 0 FOLLOWER]: Advancing to term 1
I20250814 13:57:42.776336 13784 raft_consensus.cc:513] T d0f31a93d114413a84f457479b5bf438 P 18ed8d3c4a054b10a8d95e6f07d51216 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4bc58bf426f4f44ade6ed663e227552" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 37621 } } peers { permanent_uuid: "435806ef34b349968a8732e7662792ce" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 40861 } } peers { permanent_uuid: "18ed8d3c4a054b10a8d95e6f07d51216" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 43095 } }
I20250814 13:57:42.777726 13784 leader_election.cc:290] T d0f31a93d114413a84f457479b5bf438 P 18ed8d3c4a054b10a8d95e6f07d51216 [CANDIDATE]: Term 1 election: Requested vote from peers d4bc58bf426f4f44ade6ed663e227552 (127.11.132.131:37621), 435806ef34b349968a8732e7662792ce (127.11.132.130:40861)
I20250814 13:57:42.778462 13713 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0f31a93d114413a84f457479b5bf438" candidate_uuid: "18ed8d3c4a054b10a8d95e6f07d51216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4bc58bf426f4f44ade6ed663e227552"
I20250814 13:57:42.778685 13580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0f31a93d114413a84f457479b5bf438" candidate_uuid: "18ed8d3c4a054b10a8d95e6f07d51216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "435806ef34b349968a8732e7662792ce"
I20250814 13:57:42.778944 13713 raft_consensus.cc:3058] T d0f31a93d114413a84f457479b5bf438 P d4bc58bf426f4f44ade6ed663e227552 [term 0 FOLLOWER]: Advancing to term 1
I20250814 13:57:42.779165 13580 raft_consensus.cc:3058] T d0f31a93d114413a84f457479b5bf438 P 435806ef34b349968a8732e7662792ce [term 0 FOLLOWER]: Advancing to term 1
I20250814 13:57:42.785864 13580 raft_consensus.cc:2466] T d0f31a93d114413a84f457479b5bf438 P 435806ef34b349968a8732e7662792ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18ed8d3c4a054b10a8d95e6f07d51216 in term 1.
I20250814 13:57:42.785864 13713 raft_consensus.cc:2466] T d0f31a93d114413a84f457479b5bf438 P d4bc58bf426f4f44ade6ed663e227552 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18ed8d3c4a054b10a8d95e6f07d51216 in term 1.
I20250814 13:57:42.786901 13381 leader_election.cc:304] T d0f31a93d114413a84f457479b5bf438 P 18ed8d3c4a054b10a8d95e6f07d51216 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18ed8d3c4a054b10a8d95e6f07d51216, d4bc58bf426f4f44ade6ed663e227552; no voters:
I20250814 13:57:42.787560 13784 raft_consensus.cc:2802] T d0f31a93d114413a84f457479b5bf438 P 18ed8d3c4a054b10a8d95e6f07d51216 [term 1 FOLLOWER]: Leader election won for term 1
I20250814 13:57:42.789165 13784 raft_consensus.cc:695] T d0f31a93d114413a84f457479b5bf438 P 18ed8d3c4a054b10a8d95e6f07d51216 [term 1 LEADER]: Becoming Leader. State: Replica: 18ed8d3c4a054b10a8d95e6f07d51216, State: Running, Role: LEADER
I20250814 13:57:42.790074 13784 consensus_queue.cc:237] T d0f31a93d114413a84f457479b5bf438 P 18ed8d3c4a054b10a8d95e6f07d51216 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4bc58bf426f4f44ade6ed663e227552" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 37621 } } peers { permanent_uuid: "435806ef34b349968a8732e7662792ce" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 40861 } } peers { permanent_uuid: "18ed8d3c4a054b10a8d95e6f07d51216" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 43095 } }
I20250814 13:57:42.799917 13303 catalog_manager.cc:5582] T d0f31a93d114413a84f457479b5bf438 P 18ed8d3c4a054b10a8d95e6f07d51216 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18ed8d3c4a054b10a8d95e6f07d51216 (127.11.132.129). New cstate: current_term: 1 leader_uuid: "18ed8d3c4a054b10a8d95e6f07d51216" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4bc58bf426f4f44ade6ed663e227552" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 37621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "435806ef34b349968a8732e7662792ce" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 40861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18ed8d3c4a054b10a8d95e6f07d51216" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 43095 } health_report { overall_health: HEALTHY } } }
I20250814 13:57:42.820338 11794 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250814 13:57:42.823450 11794 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 18ed8d3c4a054b10a8d95e6f07d51216 to finish bootstrapping
I20250814 13:57:42.836426 11794 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 435806ef34b349968a8732e7662792ce to finish bootstrapping
I20250814 13:57:42.846410 11794 ts_itest-base.cc:246] Waiting for 1 tablets on tserver d4bc58bf426f4f44ade6ed663e227552 to finish bootstrapping
I20250814 13:57:42.858719 13303 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59568:
name: "TestAnotherTable"
schema {
columns {
name: "foo"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "bar"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
comment: "comment for bar"
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "foo"
}
}
}
W20250814 13:57:42.860230 13303 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAnotherTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250814 13:57:42.875641 13426 tablet_service.cc:1468] Processing CreateTablet for tablet f4dee5e8ebe74a6d907659eabd76e057 (DEFAULT_TABLE table=TestAnotherTable [id=8ce8fc995b6f4cf3a58f1fabdf382fc0]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250814 13:57:42.876281 13560 tablet_service.cc:1468] Processing CreateTablet for tablet f4dee5e8ebe74a6d907659eabd76e057 (DEFAULT_TABLE table=TestAnotherTable [id=8ce8fc995b6f4cf3a58f1fabdf382fc0]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250814 13:57:42.876741 13426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4dee5e8ebe74a6d907659eabd76e057. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:57:42.876928 13693 tablet_service.cc:1468] Processing CreateTablet for tablet f4dee5e8ebe74a6d907659eabd76e057 (DEFAULT_TABLE table=TestAnotherTable [id=8ce8fc995b6f4cf3a58f1fabdf382fc0]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250814 13:57:42.877337 13560 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4dee5e8ebe74a6d907659eabd76e057. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:57:42.877794 13693 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4dee5e8ebe74a6d907659eabd76e057. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:57:42.888619 13778 tablet_bootstrap.cc:492] T f4dee5e8ebe74a6d907659eabd76e057 P d4bc58bf426f4f44ade6ed663e227552: Bootstrap starting.
I20250814 13:57:42.890522 13779 tablet_bootstrap.cc:492] T f4dee5e8ebe74a6d907659eabd76e057 P 18ed8d3c4a054b10a8d95e6f07d51216: Bootstrap starting.
I20250814 13:57:42.892093 13777 tablet_bootstrap.cc:492] T f4dee5e8ebe74a6d907659eabd76e057 P 435806ef34b349968a8732e7662792ce: Bootstrap starting.
I20250814 13:57:42.894234 13778 tablet_bootstrap.cc:654] T f4dee5e8ebe74a6d907659eabd76e057 P d4bc58bf426f4f44ade6ed663e227552: Neither blocks nor log segments found. Creating new log.
I20250814 13:57:42.895110 13779 tablet_bootstrap.cc:654] T f4dee5e8ebe74a6d907659eabd76e057 P 18ed8d3c4a054b10a8d95e6f07d51216: Neither blocks nor log segments found. Creating new log.
I20250814 13:57:42.898336 13777 tablet_bootstrap.cc:654] T f4dee5e8ebe74a6d907659eabd76e057 P 435806ef34b349968a8732e7662792ce: Neither blocks nor log segments found. Creating new log.
I20250814 13:57:42.900175 13778 tablet_bootstrap.cc:492] T f4dee5e8ebe74a6d907659eabd76e057 P d4bc58bf426f4f44ade6ed663e227552: No bootstrap required, opened a new log
I20250814 13:57:42.900556 13778 ts_tablet_manager.cc:1397] T f4dee5e8ebe74a6d907659eabd76e057 P d4bc58bf426f4f44ade6ed663e227552: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20250814 13:57:42.900790 13779 tablet_bootstrap.cc:492] T f4dee5e8ebe74a6d907659eabd76e057 P 18ed8d3c4a054b10a8d95e6f07d51216: No bootstrap required, opened a new log
I20250814 13:57:42.901254 13779 ts_tablet_manager.cc:1397] T f4dee5e8ebe74a6d907659eabd76e057 P 18ed8d3c4a054b10a8d95e6f07d51216: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20250814 13:57:42.903195 13778 raft_consensus.cc:357] T f4dee5e8ebe74a6d907659eabd76e057 P d4bc58bf426f4f44ade6ed663e227552 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18ed8d3c4a054b10a8d95e6f07d51216" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 43095 } } peers { permanent_uuid: "435806ef34b349968a8732e7662792ce" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 40861 } } peers { permanent_uuid: "d4bc58bf426f4f44ade6ed663e227552" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 37621 } }
I20250814 13:57:42.903973 13778 raft_consensus.cc:383] T f4dee5e8ebe74a6d907659eabd76e057 P d4bc58bf426f4f44ade6ed663e227552 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 13:57:42.904057 13777 tablet_bootstrap.cc:492] T f4dee5e8ebe74a6d907659eabd76e057 P 435806ef34b349968a8732e7662792ce: No bootstrap required, opened a new log
I20250814 13:57:42.903695 13779 raft_consensus.cc:357] T f4dee5e8ebe74a6d907659eabd76e057 P 18ed8d3c4a054b10a8d95e6f07d51216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18ed8d3c4a054b10a8d95e6f07d51216" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 43095 } } peers { permanent_uuid: "435806ef34b349968a8732e7662792ce" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 40861 } } peers { permanent_uuid: "d4bc58bf426f4f44ade6ed663e227552" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 37621 } }
I20250814 13:57:42.904414 13779 raft_consensus.cc:383] T f4dee5e8ebe74a6d907659eabd76e057 P 18ed8d3c4a054b10a8d95e6f07d51216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 13:57:42.904328 13778 raft_consensus.cc:738] T f4dee5e8ebe74a6d907659eabd76e057 P d4bc58bf426f4f44ade6ed663e227552 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4bc58bf426f4f44ade6ed663e227552, State: Initialized, Role: FOLLOWER
I20250814 13:57:42.904508 13777 ts_tablet_manager.cc:1397] T f4dee5e8ebe74a6d907659eabd76e057 P 435806ef34b349968a8732e7662792ce: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20250814 13:57:42.904750 13779 raft_consensus.cc:738] T f4dee5e8ebe74a6d907659eabd76e057 P 18ed8d3c4a054b10a8d95e6f07d51216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18ed8d3c4a054b10a8d95e6f07d51216, State: Initialized, Role: FOLLOWER
I20250814 13:57:42.905228 13778 consensus_queue.cc:260] T f4dee5e8ebe74a6d907659eabd76e057 P d4bc58bf426f4f44ade6ed663e227552 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18ed8d3c4a054b10a8d95e6f07d51216" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 43095 } } peers { permanent_uuid: "435806ef34b349968a8732e7662792ce" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 40861 } } peers { permanent_uuid: "d4bc58bf426f4f44ade6ed663e227552" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 37621 } }
I20250814 13:57:42.905429 13779 consensus_queue.cc:260] T f4dee5e8ebe74a6d907659eabd76e057 P 18ed8d3c4a054b10a8d95e6f07d51216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18ed8d3c4a054b10a8d95e6f07d51216" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 43095 } } peers { permanent_uuid: "435806ef34b349968a8732e7662792ce" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 40861 } } peers { permanent_uuid: "d4bc58bf426f4f44ade6ed663e227552" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 37621 } }
I20250814 13:57:42.907343 13778 ts_tablet_manager.cc:1428] T f4dee5e8ebe74a6d907659eabd76e057 P d4bc58bf426f4f44ade6ed663e227552: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250814 13:57:42.907943 13777 raft_consensus.cc:357] T f4dee5e8ebe74a6d907659eabd76e057 P 435806ef34b349968a8732e7662792ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18ed8d3c4a054b10a8d95e6f07d51216" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 43095 } } peers { permanent_uuid: "435806ef34b349968a8732e7662792ce" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 40861 } } peers { permanent_uuid: "d4bc58bf426f4f44ade6ed663e227552" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 37621 } }
I20250814 13:57:42.908658 13777 raft_consensus.cc:383] T f4dee5e8ebe74a6d907659eabd76e057 P 435806ef34b349968a8732e7662792ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 13:57:42.908926 13777 raft_consensus.cc:738] T f4dee5e8ebe74a6d907659eabd76e057 P 435806ef34b349968a8732e7662792ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 435806ef34b349968a8732e7662792ce, State: Initialized, Role: FOLLOWER
I20250814 13:57:42.909840 13779 ts_tablet_manager.cc:1428] T f4dee5e8ebe74a6d907659eabd76e057 P 18ed8d3c4a054b10a8d95e6f07d51216: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20250814 13:57:42.909614 13777 consensus_queue.cc:260] T f4dee5e8ebe74a6d907659eabd76e057 P 435806ef34b349968a8732e7662792ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18ed8d3c4a054b10a8d95e6f07d51216" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 43095 } } peers { permanent_uuid: "435806ef34b349968a8732e7662792ce" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 40861 } } peers { permanent_uuid: "d4bc58bf426f4f44ade6ed663e227552" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 37621 } }
I20250814 13:57:42.911412 13777 ts_tablet_manager.cc:1428] T f4dee5e8ebe74a6d907659eabd76e057 P 435806ef34b349968a8732e7662792ce: Time spent starting tablet: real 0.007s user 0.001s sys 0.003s
I20250814 13:57:43.020829 13784 raft_consensus.cc:491] T f4dee5e8ebe74a6d907659eabd76e057 P 18ed8d3c4a054b10a8d95e6f07d51216 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250814 13:57:43.021257 13784 raft_consensus.cc:513] T f4dee5e8ebe74a6d907659eabd76e057 P 18ed8d3c4a054b10a8d95e6f07d51216 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18ed8d3c4a054b10a8d95e6f07d51216" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 43095 } } peers { permanent_uuid: "435806ef34b349968a8732e7662792ce" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 40861 } } peers { permanent_uuid: "d4bc58bf426f4f44ade6ed663e227552" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 37621 } }
I20250814 13:57:43.022603 13784 leader_election.cc:290] T f4dee5e8ebe74a6d907659eabd76e057 P 18ed8d3c4a054b10a8d95e6f07d51216 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 435806ef34b349968a8732e7662792ce (127.11.132.130:40861), d4bc58bf426f4f44ade6ed663e227552 (127.11.132.131:37621)
I20250814 13:57:43.023351 13580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4dee5e8ebe74a6d907659eabd76e057" candidate_uuid: "18ed8d3c4a054b10a8d95e6f07d51216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "435806ef34b349968a8732e7662792ce" is_pre_election: true
I20250814 13:57:43.023525 13713 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4dee5e8ebe74a6d907659eabd76e057" candidate_uuid: "18ed8d3c4a054b10a8d95e6f07d51216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4bc58bf426f4f44ade6ed663e227552" is_pre_election: true
I20250814 13:57:43.023785 13580 raft_consensus.cc:2466] T f4dee5e8ebe74a6d907659eabd76e057 P 435806ef34b349968a8732e7662792ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18ed8d3c4a054b10a8d95e6f07d51216 in term 0.
I20250814 13:57:43.024063 13713 raft_consensus.cc:2466] T f4dee5e8ebe74a6d907659eabd76e057 P d4bc58bf426f4f44ade6ed663e227552 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18ed8d3c4a054b10a8d95e6f07d51216 in term 0.
I20250814 13:57:43.024780 13380 leader_election.cc:304] T f4dee5e8ebe74a6d907659eabd76e057 P 18ed8d3c4a054b10a8d95e6f07d51216 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18ed8d3c4a054b10a8d95e6f07d51216, 435806ef34b349968a8732e7662792ce; no voters:
I20250814 13:57:43.025451 13784 raft_consensus.cc:2802] T f4dee5e8ebe74a6d907659eabd76e057 P 18ed8d3c4a054b10a8d95e6f07d51216 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250814 13:57:43.025697 13784 raft_consensus.cc:491] T f4dee5e8ebe74a6d907659eabd76e057 P 18ed8d3c4a054b10a8d95e6f07d51216 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250814 13:57:43.025951 13784 raft_consensus.cc:3058] T f4dee5e8ebe74a6d907659eabd76e057 P 18ed8d3c4a054b10a8d95e6f07d51216 [term 0 FOLLOWER]: Advancing to term 1
I20250814 13:57:43.029676 13784 raft_consensus.cc:513] T f4dee5e8ebe74a6d907659eabd76e057 P 18ed8d3c4a054b10a8d95e6f07d51216 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18ed8d3c4a054b10a8d95e6f07d51216" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 43095 } } peers { permanent_uuid: "435806ef34b349968a8732e7662792ce" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 40861 } } peers { permanent_uuid: "d4bc58bf426f4f44ade6ed663e227552" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 37621 } }
I20250814 13:57:43.030973 13784 leader_election.cc:290] T f4dee5e8ebe74a6d907659eabd76e057 P 18ed8d3c4a054b10a8d95e6f07d51216 [CANDIDATE]: Term 1 election: Requested vote from peers 435806ef34b349968a8732e7662792ce (127.11.132.130:40861), d4bc58bf426f4f44ade6ed663e227552 (127.11.132.131:37621)
I20250814 13:57:43.031639 13580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4dee5e8ebe74a6d907659eabd76e057" candidate_uuid: "18ed8d3c4a054b10a8d95e6f07d51216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "435806ef34b349968a8732e7662792ce"
I20250814 13:57:43.031778 13713 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4dee5e8ebe74a6d907659eabd76e057" candidate_uuid: "18ed8d3c4a054b10a8d95e6f07d51216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4bc58bf426f4f44ade6ed663e227552"
I20250814 13:57:43.032112 13580 raft_consensus.cc:3058] T f4dee5e8ebe74a6d907659eabd76e057 P 435806ef34b349968a8732e7662792ce [term 0 FOLLOWER]: Advancing to term 1
I20250814 13:57:43.032213 13713 raft_consensus.cc:3058] T f4dee5e8ebe74a6d907659eabd76e057 P d4bc58bf426f4f44ade6ed663e227552 [term 0 FOLLOWER]: Advancing to term 1
I20250814 13:57:43.038182 13713 raft_consensus.cc:2466] T f4dee5e8ebe74a6d907659eabd76e057 P d4bc58bf426f4f44ade6ed663e227552 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18ed8d3c4a054b10a8d95e6f07d51216 in term 1.
I20250814 13:57:43.038179 13580 raft_consensus.cc:2466] T f4dee5e8ebe74a6d907659eabd76e057 P 435806ef34b349968a8732e7662792ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18ed8d3c4a054b10a8d95e6f07d51216 in term 1.
I20250814 13:57:43.039017 13380 leader_election.cc:304] T f4dee5e8ebe74a6d907659eabd76e057 P 18ed8d3c4a054b10a8d95e6f07d51216 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18ed8d3c4a054b10a8d95e6f07d51216, 435806ef34b349968a8732e7662792ce; no voters:
I20250814 13:57:43.039534 13784 raft_consensus.cc:2802] T f4dee5e8ebe74a6d907659eabd76e057 P 18ed8d3c4a054b10a8d95e6f07d51216 [term 1 FOLLOWER]: Leader election won for term 1
I20250814 13:57:43.039889 13784 raft_consensus.cc:695] T f4dee5e8ebe74a6d907659eabd76e057 P 18ed8d3c4a054b10a8d95e6f07d51216 [term 1 LEADER]: Becoming Leader. State: Replica: 18ed8d3c4a054b10a8d95e6f07d51216, State: Running, Role: LEADER
I20250814 13:57:43.040547 13784 consensus_queue.cc:237] T f4dee5e8ebe74a6d907659eabd76e057 P 18ed8d3c4a054b10a8d95e6f07d51216 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18ed8d3c4a054b10a8d95e6f07d51216" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 43095 } } peers { permanent_uuid: "435806ef34b349968a8732e7662792ce" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 40861 } } peers { permanent_uuid: "d4bc58bf426f4f44ade6ed663e227552" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 37621 } }
I20250814 13:57:43.046823 13302 catalog_manager.cc:5582] T f4dee5e8ebe74a6d907659eabd76e057 P 18ed8d3c4a054b10a8d95e6f07d51216 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18ed8d3c4a054b10a8d95e6f07d51216 (127.11.132.129). New cstate: current_term: 1 leader_uuid: "18ed8d3c4a054b10a8d95e6f07d51216" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18ed8d3c4a054b10a8d95e6f07d51216" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 43095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "435806ef34b349968a8732e7662792ce" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 40861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4bc58bf426f4f44ade6ed663e227552" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 37621 } health_report { overall_health: UNKNOWN } } }
I20250814 13:57:43.322265 13784 consensus_queue.cc:1035] T d0f31a93d114413a84f457479b5bf438 P 18ed8d3c4a054b10a8d95e6f07d51216 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4bc58bf426f4f44ade6ed663e227552" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 37621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250814 13:57:43.341025 13784 consensus_queue.cc:1035] T d0f31a93d114413a84f457479b5bf438 P 18ed8d3c4a054b10a8d95e6f07d51216 [LEADER]: Connected to new peer: Peer: permanent_uuid: "435806ef34b349968a8732e7662792ce" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 40861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250814 13:57:43.409523 13795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:57:43.410125 13795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:57:43.439894 13795 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250814 13:57:43.459748 13801 consensus_queue.cc:1035] T f4dee5e8ebe74a6d907659eabd76e057 P 18ed8d3c4a054b10a8d95e6f07d51216 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4bc58bf426f4f44ade6ed663e227552" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 37621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250814 13:57:43.468883 13788 consensus_queue.cc:1035] T f4dee5e8ebe74a6d907659eabd76e057 P 18ed8d3c4a054b10a8d95e6f07d51216 [LEADER]: Connected to new peer: Peer: permanent_uuid: "435806ef34b349968a8732e7662792ce" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 40861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250814 13:57:44.709738 13795 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.226s user 0.479s sys 0.746s
W20250814 13:57:44.710137 13795 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.227s user 0.479s sys 0.746s
W20250814 13:57:46.083817 13817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:57:46.084442 13817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:57:46.113299 13817 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250814 13:57:47.340430 13817 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.187s user 0.522s sys 0.661s
W20250814 13:57:47.340754 13817 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.188s user 0.522s sys 0.661s
W20250814 13:57:48.699895 13831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:57:48.700446 13831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:57:48.738798 13831 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250814 13:57:49.954804 13831 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.178s user 0.462s sys 0.714s
W20250814 13:57:49.955072 13831 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.179s user 0.462s sys 0.714s
W20250814 13:57:51.311024 13848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:57:51.311580 13848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:57:51.341955 13848 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250814 13:57:52.588362 13848 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.210s user 0.402s sys 0.805s
W20250814 13:57:52.588655 13848 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.210s user 0.402s sys 0.805s
I20250814 13:57:53.658026 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 13362
I20250814 13:57:53.685716 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 13495
I20250814 13:57:53.710602 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 13629
I20250814 13:57:53.735767 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 13270
2025-08-14T13:57:53Z chronyd exiting
[ OK ] AdminCliTest.TestDescribeTableColumnFlags (19352 ms)
[ RUN ] AdminCliTest.TestAuthzResetCacheNotAuthorized
I20250814 13:57:53.792304 11794 test_util.cc:276] Using random seed: -1461391082
I20250814 13:57:53.796411 11794 ts_itest-base.cc:115] Starting cluster with:
I20250814 13:57:53.796571 11794 ts_itest-base.cc:116] --------------
I20250814 13:57:53.796741 11794 ts_itest-base.cc:117] 3 tablet servers
I20250814 13:57:53.796876 11794 ts_itest-base.cc:118] 3 replicas per TS
I20250814 13:57:53.797011 11794 ts_itest-base.cc:119] --------------
2025-08-14T13:57:53Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-14T13:57:53Z Disabled control of system clock
I20250814 13:57:53.831024 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.190:43071
--webserver_interface=127.11.132.190
--webserver_port=0
--builtin_ntp_servers=127.11.132.148:39715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.132.190:43071
--superuser_acl=no-such-user with env {}
W20250814 13:57:54.123134 13869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:57:54.123660 13869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:57:54.124118 13869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:57:54.153594 13869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250814 13:57:54.153847 13869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:57:54.154021 13869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250814 13:57:54.154196 13869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250814 13:57:54.186614 13869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:39715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.132.190:43071
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.190:43071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.11.132.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:57:54.187824 13869 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:57:54.189349 13869 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:57:54.199595 13875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:57:55.603574 13874 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ec0 after lost signal to thread 13869
W20250814 13:57:54.199693 13876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:57:55.924966 13869 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.725s user 0.577s sys 1.148s
W20250814 13:57:55.926398 13869 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.727s user 0.577s sys 1.149s
W20250814 13:57:55.927763 13878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:57:55.930574 13877 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1724 milliseconds
I20250814 13:57:55.930591 13869 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 13:57:55.931793 13869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:57:55.934101 13869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:57:55.935420 13869 hybrid_clock.cc:648] HybridClock initialized: now 1755179875935388 us; error 56 us; skew 500 ppm
I20250814 13:57:55.936161 13869 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:57:55.941816 13869 webserver.cc:480] Webserver started at http://127.11.132.190:38619/ using document root <none> and password file <none>
I20250814 13:57:55.942667 13869 fs_manager.cc:362] Metadata directory not provided
I20250814 13:57:55.942874 13869 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:57:55.943302 13869 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 13:57:55.947508 13869 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "69d50aae7bca473bb72e191d963a9205"
format_stamp: "Formatted at 2025-08-14 13:57:55 on dist-test-slave-q1tc"
I20250814 13:57:55.948552 13869 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "69d50aae7bca473bb72e191d963a9205"
format_stamp: "Formatted at 2025-08-14 13:57:55 on dist-test-slave-q1tc"
I20250814 13:57:55.955230 13869 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.000s sys 0.005s
I20250814 13:57:55.960350 13885 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:57:55.961267 13869 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250814 13:57:55.961555 13869 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
uuid: "69d50aae7bca473bb72e191d963a9205"
format_stamp: "Formatted at 2025-08-14 13:57:55 on dist-test-slave-q1tc"
I20250814 13:57:55.961860 13869 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:57:56.013049 13869 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:57:56.014390 13869 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:57:56.014791 13869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:57:56.081512 13869 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.190:43071
I20250814 13:57:56.081571 13936 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.190:43071 every 8 connection(s)
I20250814 13:57:56.083962 13869 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250814 13:57:56.086124 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 13869
I20250814 13:57:56.086596 11794 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250814 13:57:56.089861 13937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:57:56.112143 13937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69d50aae7bca473bb72e191d963a9205: Bootstrap starting.
I20250814 13:57:56.117843 13937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69d50aae7bca473bb72e191d963a9205: Neither blocks nor log segments found. Creating new log.
I20250814 13:57:56.119416 13937 log.cc:826] T 00000000000000000000000000000000 P 69d50aae7bca473bb72e191d963a9205: Log is configured to *not* fsync() on all Append() calls
I20250814 13:57:56.123551 13937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69d50aae7bca473bb72e191d963a9205: No bootstrap required, opened a new log
I20250814 13:57:56.139600 13937 raft_consensus.cc:357] T 00000000000000000000000000000000 P 69d50aae7bca473bb72e191d963a9205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d50aae7bca473bb72e191d963a9205" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 43071 } }
I20250814 13:57:56.140189 13937 raft_consensus.cc:383] T 00000000000000000000000000000000 P 69d50aae7bca473bb72e191d963a9205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 13:57:56.140429 13937 raft_consensus.cc:738] T 00000000000000000000000000000000 P 69d50aae7bca473bb72e191d963a9205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69d50aae7bca473bb72e191d963a9205, State: Initialized, Role: FOLLOWER
I20250814 13:57:56.141108 13937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69d50aae7bca473bb72e191d963a9205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d50aae7bca473bb72e191d963a9205" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 43071 } }
I20250814 13:57:56.141608 13937 raft_consensus.cc:397] T 00000000000000000000000000000000 P 69d50aae7bca473bb72e191d963a9205 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 13:57:56.141857 13937 raft_consensus.cc:491] T 00000000000000000000000000000000 P 69d50aae7bca473bb72e191d963a9205 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 13:57:56.142184 13937 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 69d50aae7bca473bb72e191d963a9205 [term 0 FOLLOWER]: Advancing to term 1
I20250814 13:57:56.146423 13937 raft_consensus.cc:513] T 00000000000000000000000000000000 P 69d50aae7bca473bb72e191d963a9205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d50aae7bca473bb72e191d963a9205" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 43071 } }
I20250814 13:57:56.147066 13937 leader_election.cc:304] T 00000000000000000000000000000000 P 69d50aae7bca473bb72e191d963a9205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69d50aae7bca473bb72e191d963a9205; no voters:
I20250814 13:57:56.148655 13937 leader_election.cc:290] T 00000000000000000000000000000000 P 69d50aae7bca473bb72e191d963a9205 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250814 13:57:56.149381 13942 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 69d50aae7bca473bb72e191d963a9205 [term 1 FOLLOWER]: Leader election won for term 1
I20250814 13:57:56.151430 13942 raft_consensus.cc:695] T 00000000000000000000000000000000 P 69d50aae7bca473bb72e191d963a9205 [term 1 LEADER]: Becoming Leader. State: Replica: 69d50aae7bca473bb72e191d963a9205, State: Running, Role: LEADER
I20250814 13:57:56.152123 13942 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69d50aae7bca473bb72e191d963a9205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d50aae7bca473bb72e191d963a9205" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 43071 } }
I20250814 13:57:56.153131 13937 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69d50aae7bca473bb72e191d963a9205 [sys.catalog]: configured and running, proceeding with master startup.
I20250814 13:57:56.161103 13944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69d50aae7bca473bb72e191d963a9205 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69d50aae7bca473bb72e191d963a9205. Latest consensus state: current_term: 1 leader_uuid: "69d50aae7bca473bb72e191d963a9205" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d50aae7bca473bb72e191d963a9205" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 43071 } } }
I20250814 13:57:56.161310 13943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69d50aae7bca473bb72e191d963a9205 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69d50aae7bca473bb72e191d963a9205" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d50aae7bca473bb72e191d963a9205" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 43071 } } }
I20250814 13:57:56.161947 13944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69d50aae7bca473bb72e191d963a9205 [sys.catalog]: This master's current role is: LEADER
I20250814 13:57:56.162074 13943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69d50aae7bca473bb72e191d963a9205 [sys.catalog]: This master's current role is: LEADER
I20250814 13:57:56.166574 13951 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250814 13:57:56.176637 13951 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250814 13:57:56.190099 13951 catalog_manager.cc:1349] Generated new cluster ID: 9277888ea7ad4488b5381425162b5ef2
I20250814 13:57:56.190409 13951 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250814 13:57:56.213757 13951 catalog_manager.cc:1372] Generated new certificate authority record
I20250814 13:57:56.215049 13951 catalog_manager.cc:1506] Loading token signing keys...
I20250814 13:57:56.227757 13951 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 69d50aae7bca473bb72e191d963a9205: Generated new TSK 0
I20250814 13:57:56.228551 13951 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250814 13:57:56.243171 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.132.129:0
--local_ip_for_outbound_sockets=127.11.132.129
--webserver_interface=127.11.132.129
--webserver_port=0
--tserver_master_addrs=127.11.132.190:43071
--builtin_ntp_servers=127.11.132.148:39715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250814 13:57:56.541330 13961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:57:56.541775 13961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:57:56.542243 13961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:57:56.572060 13961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:57:56.572883 13961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.132.129
I20250814 13:57:56.606469 13961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:39715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.132.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.11.132.129
--webserver_port=0
--tserver_master_addrs=127.11.132.190:43071
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.132.129
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:57:56.607748 13961 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:57:56.609258 13961 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:57:56.621781 13967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:57:56.622267 13968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:57:57.984820 13961 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.364s user 0.519s sys 0.837s
W20250814 13:57:57.985224 13961 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.365s user 0.523s sys 0.837s
I20250814 13:57:57.991966 13961 server_base.cc:1047] running on GCE node
W20250814 13:57:57.993150 13972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 13:57:57.994649 13961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:57:57.997295 13961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:57:57.998800 13961 hybrid_clock.cc:648] HybridClock initialized: now 1755179877998725 us; error 68 us; skew 500 ppm
I20250814 13:57:57.999828 13961 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:57:58.008754 13961 webserver.cc:480] Webserver started at http://127.11.132.129:41341/ using document root <none> and password file <none>
I20250814 13:57:58.010002 13961 fs_manager.cc:362] Metadata directory not provided
I20250814 13:57:58.010272 13961 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:57:58.010824 13961 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 13:57:58.017673 13961 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "8c647b84fe56454e8ce009681a95b446"
format_stamp: "Formatted at 2025-08-14 13:57:58 on dist-test-slave-q1tc"
I20250814 13:57:58.019199 13961 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "8c647b84fe56454e8ce009681a95b446"
format_stamp: "Formatted at 2025-08-14 13:57:58 on dist-test-slave-q1tc"
I20250814 13:57:58.030889 13961 fs_manager.cc:696] Time spent creating directory manager: real 0.011s user 0.004s sys 0.008s
I20250814 13:57:58.036824 13977 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:57:58.037928 13961 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20250814 13:57:58.038244 13961 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "8c647b84fe56454e8ce009681a95b446"
format_stamp: "Formatted at 2025-08-14 13:57:58 on dist-test-slave-q1tc"
I20250814 13:57:58.038583 13961 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:57:58.126288 13961 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:57:58.127694 13961 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:57:58.128108 13961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:57:58.130882 13961 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 13:57:58.135699 13961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 13:57:58.135895 13961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 13:57:58.136137 13961 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 13:57:58.136291 13961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 13:57:58.299789 13961 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.129:41565
I20250814 13:57:58.299886 14089 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.129:41565 every 8 connection(s)
I20250814 13:57:58.302378 13961 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250814 13:57:58.309983 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 13961
I20250814 13:57:58.310439 11794 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250814 13:57:58.315922 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.132.130:0
--local_ip_for_outbound_sockets=127.11.132.130
--webserver_interface=127.11.132.130
--webserver_port=0
--tserver_master_addrs=127.11.132.190:43071
--builtin_ntp_servers=127.11.132.148:39715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250814 13:57:58.327435 14090 heartbeater.cc:344] Connected to a master server at 127.11.132.190:43071
I20250814 13:57:58.327909 14090 heartbeater.cc:461] Registering TS with master...
I20250814 13:57:58.329061 14090 heartbeater.cc:507] Master 127.11.132.190:43071 requested a full tablet report, sending...
I20250814 13:57:58.331806 13902 ts_manager.cc:194] Registered new tserver with Master: 8c647b84fe56454e8ce009681a95b446 (127.11.132.129:41565)
I20250814 13:57:58.334776 13902 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.132.129:57133
W20250814 13:57:58.607120 14094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:57:58.607652 14094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:57:58.608088 14094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:57:58.636902 14094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:57:58.637748 14094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.132.130
I20250814 13:57:58.671105 14094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:39715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.132.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.11.132.130
--webserver_port=0
--tserver_master_addrs=127.11.132.190:43071
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.132.130
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:57:58.672257 14094 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:57:58.673744 14094 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:57:58.685741 14100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 13:57:59.338707 14090 heartbeater.cc:499] Master 127.11.132.190:43071 was elected leader, sending a full tablet report...
W20250814 13:58:00.092363 14099 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 14094
W20250814 13:57:58.686195 14101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:58:00.183997 14099 kernel_stack_watchdog.cc:198] Thread 14094 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 403ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250814 13:58:00.188822 14094 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.503s user 0.498s sys 0.992s
W20250814 13:58:00.189178 14094 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.504s user 0.498s sys 0.993s
W20250814 13:58:00.189286 14102 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1498 milliseconds
W20250814 13:58:00.190513 14103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 13:58:00.190555 14094 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 13:58:00.191514 14094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:58:00.193634 14094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:58:00.194988 14094 hybrid_clock.cc:648] HybridClock initialized: now 1755179880194949 us; error 37 us; skew 500 ppm
I20250814 13:58:00.195720 14094 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:58:00.201155 14094 webserver.cc:480] Webserver started at http://127.11.132.130:43035/ using document root <none> and password file <none>
I20250814 13:58:00.201987 14094 fs_manager.cc:362] Metadata directory not provided
I20250814 13:58:00.202167 14094 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:58:00.202564 14094 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 13:58:00.206526 14094 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "6c9fdd7eebba45f58a9ada76ded89610"
format_stamp: "Formatted at 2025-08-14 13:58:00 on dist-test-slave-q1tc"
I20250814 13:58:00.207537 14094 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "6c9fdd7eebba45f58a9ada76ded89610"
format_stamp: "Formatted at 2025-08-14 13:58:00 on dist-test-slave-q1tc"
I20250814 13:58:00.213893 14094 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.006s
I20250814 13:58:00.218772 14111 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:58:00.219684 14094 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250814 13:58:00.219962 14094 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "6c9fdd7eebba45f58a9ada76ded89610"
format_stamp: "Formatted at 2025-08-14 13:58:00 on dist-test-slave-q1tc"
I20250814 13:58:00.220242 14094 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:58:00.264586 14094 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:58:00.265974 14094 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:58:00.266367 14094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:58:00.268747 14094 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 13:58:00.272348 14094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 13:58:00.272532 14094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 13:58:00.272786 14094 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 13:58:00.272948 14094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 13:58:00.402742 14094 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.130:46301
I20250814 13:58:00.402856 14223 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.130:46301 every 8 connection(s)
I20250814 13:58:00.405233 14094 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250814 13:58:00.412719 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 14094
I20250814 13:58:00.413297 11794 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250814 13:58:00.419663 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.132.131:0
--local_ip_for_outbound_sockets=127.11.132.131
--webserver_interface=127.11.132.131
--webserver_port=0
--tserver_master_addrs=127.11.132.190:43071
--builtin_ntp_servers=127.11.132.148:39715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250814 13:58:00.425305 14224 heartbeater.cc:344] Connected to a master server at 127.11.132.190:43071
I20250814 13:58:00.425827 14224 heartbeater.cc:461] Registering TS with master...
I20250814 13:58:00.426998 14224 heartbeater.cc:507] Master 127.11.132.190:43071 requested a full tablet report, sending...
I20250814 13:58:00.429145 13902 ts_manager.cc:194] Registered new tserver with Master: 6c9fdd7eebba45f58a9ada76ded89610 (127.11.132.130:46301)
I20250814 13:58:00.430912 13902 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.132.130:39037
W20250814 13:58:00.708668 14228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:58:00.709192 14228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:58:00.709684 14228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:58:00.739603 14228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:58:00.740456 14228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.132.131
I20250814 13:58:00.774062 14228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:39715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.132.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.11.132.131
--webserver_port=0
--tserver_master_addrs=127.11.132.190:43071
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.132.131
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:58:00.775293 14228 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:58:00.776741 14228 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:58:00.787315 14234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 13:58:01.434129 14224 heartbeater.cc:499] Master 127.11.132.190:43071 was elected leader, sending a full tablet report...
W20250814 13:58:00.789276 14235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:58:02.095511 14236 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1304 milliseconds
W20250814 13:58:02.097290 14237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:58:02.100642 14228 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.312s user 0.003s sys 0.004s
W20250814 13:58:02.100996 14228 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.312s user 0.003s sys 0.004s
I20250814 13:58:02.101301 14228 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 13:58:02.102743 14228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:58:02.105612 14228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:58:02.107121 14228 hybrid_clock.cc:648] HybridClock initialized: now 1755179882107046 us; error 70 us; skew 500 ppm
I20250814 13:58:02.108299 14228 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:58:02.116946 14228 webserver.cc:480] Webserver started at http://127.11.132.131:43597/ using document root <none> and password file <none>
I20250814 13:58:02.118373 14228 fs_manager.cc:362] Metadata directory not provided
I20250814 13:58:02.118690 14228 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:58:02.119311 14228 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 13:58:02.126564 14228 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "a670c674d6284d8ea16a304dd4f2a04f"
format_stamp: "Formatted at 2025-08-14 13:58:02 on dist-test-slave-q1tc"
I20250814 13:58:02.128154 14228 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "a670c674d6284d8ea16a304dd4f2a04f"
format_stamp: "Formatted at 2025-08-14 13:58:02 on dist-test-slave-q1tc"
I20250814 13:58:02.137997 14228 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.011s sys 0.001s
I20250814 13:58:02.145272 14244 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:58:02.146345 14228 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250814 13:58:02.146699 14228 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "a670c674d6284d8ea16a304dd4f2a04f"
format_stamp: "Formatted at 2025-08-14 13:58:02 on dist-test-slave-q1tc"
I20250814 13:58:02.147089 14228 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:58:02.237609 14228 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:58:02.238993 14228 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:58:02.239396 14228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:58:02.241760 14228 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 13:58:02.245744 14228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 13:58:02.245954 14228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 13:58:02.246148 14228 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 13:58:02.246275 14228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 13:58:02.383229 14228 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.131:42011
I20250814 13:58:02.383315 14356 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.131:42011 every 8 connection(s)
I20250814 13:58:02.385746 14228 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250814 13:58:02.395999 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 14228
I20250814 13:58:02.396482 11794 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250814 13:58:02.412710 14357 heartbeater.cc:344] Connected to a master server at 127.11.132.190:43071
I20250814 13:58:02.413131 14357 heartbeater.cc:461] Registering TS with master...
I20250814 13:58:02.414009 14357 heartbeater.cc:507] Master 127.11.132.190:43071 requested a full tablet report, sending...
I20250814 13:58:02.416072 13902 ts_manager.cc:194] Registered new tserver with Master: a670c674d6284d8ea16a304dd4f2a04f (127.11.132.131:42011)
I20250814 13:58:02.417438 11794 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250814 13:58:02.417523 13902 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.132.131:54265
I20250814 13:58:02.448365 13902 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58598:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250814 13:58:02.467036 13902 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250814 13:58:02.519977 14159 tablet_service.cc:1468] Processing CreateTablet for tablet d2036f06f6f741628dc9eb456f1c33ac (DEFAULT_TABLE table=TestTable [id=11a632dad9fb4a63b3629fa5174d2508]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 13:58:02.521951 14159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2036f06f6f741628dc9eb456f1c33ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:58:02.525790 14025 tablet_service.cc:1468] Processing CreateTablet for tablet d2036f06f6f741628dc9eb456f1c33ac (DEFAULT_TABLE table=TestTable [id=11a632dad9fb4a63b3629fa5174d2508]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 13:58:02.528172 14025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2036f06f6f741628dc9eb456f1c33ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:58:02.529153 14292 tablet_service.cc:1468] Processing CreateTablet for tablet d2036f06f6f741628dc9eb456f1c33ac (DEFAULT_TABLE table=TestTable [id=11a632dad9fb4a63b3629fa5174d2508]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 13:58:02.530814 14292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2036f06f6f741628dc9eb456f1c33ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:58:02.542011 14376 tablet_bootstrap.cc:492] T d2036f06f6f741628dc9eb456f1c33ac P 6c9fdd7eebba45f58a9ada76ded89610: Bootstrap starting.
I20250814 13:58:02.548362 14376 tablet_bootstrap.cc:654] T d2036f06f6f741628dc9eb456f1c33ac P 6c9fdd7eebba45f58a9ada76ded89610: Neither blocks nor log segments found. Creating new log.
I20250814 13:58:02.550052 14376 log.cc:826] T d2036f06f6f741628dc9eb456f1c33ac P 6c9fdd7eebba45f58a9ada76ded89610: Log is configured to *not* fsync() on all Append() calls
I20250814 13:58:02.557500 14378 tablet_bootstrap.cc:492] T d2036f06f6f741628dc9eb456f1c33ac P 8c647b84fe56454e8ce009681a95b446: Bootstrap starting.
I20250814 13:58:02.558321 14376 tablet_bootstrap.cc:492] T d2036f06f6f741628dc9eb456f1c33ac P 6c9fdd7eebba45f58a9ada76ded89610: No bootstrap required, opened a new log
I20250814 13:58:02.558802 14376 ts_tablet_manager.cc:1397] T d2036f06f6f741628dc9eb456f1c33ac P 6c9fdd7eebba45f58a9ada76ded89610: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.009s
I20250814 13:58:02.560046 14379 tablet_bootstrap.cc:492] T d2036f06f6f741628dc9eb456f1c33ac P a670c674d6284d8ea16a304dd4f2a04f: Bootstrap starting.
I20250814 13:58:02.564895 14378 tablet_bootstrap.cc:654] T d2036f06f6f741628dc9eb456f1c33ac P 8c647b84fe56454e8ce009681a95b446: Neither blocks nor log segments found. Creating new log.
I20250814 13:58:02.565886 14379 tablet_bootstrap.cc:654] T d2036f06f6f741628dc9eb456f1c33ac P a670c674d6284d8ea16a304dd4f2a04f: Neither blocks nor log segments found. Creating new log.
I20250814 13:58:02.568125 14379 log.cc:826] T d2036f06f6f741628dc9eb456f1c33ac P a670c674d6284d8ea16a304dd4f2a04f: Log is configured to *not* fsync() on all Append() calls
I20250814 13:58:02.568126 14378 log.cc:826] T d2036f06f6f741628dc9eb456f1c33ac P 8c647b84fe56454e8ce009681a95b446: Log is configured to *not* fsync() on all Append() calls
I20250814 13:58:02.573050 14379 tablet_bootstrap.cc:492] T d2036f06f6f741628dc9eb456f1c33ac P a670c674d6284d8ea16a304dd4f2a04f: No bootstrap required, opened a new log
I20250814 13:58:02.573465 14379 ts_tablet_manager.cc:1397] T d2036f06f6f741628dc9eb456f1c33ac P a670c674d6284d8ea16a304dd4f2a04f: Time spent bootstrapping tablet: real 0.014s user 0.013s sys 0.000s
I20250814 13:58:02.573968 14378 tablet_bootstrap.cc:492] T d2036f06f6f741628dc9eb456f1c33ac P 8c647b84fe56454e8ce009681a95b446: No bootstrap required, opened a new log
I20250814 13:58:02.574440 14378 ts_tablet_manager.cc:1397] T d2036f06f6f741628dc9eb456f1c33ac P 8c647b84fe56454e8ce009681a95b446: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.000s
I20250814 13:58:02.582721 14376 raft_consensus.cc:357] T d2036f06f6f741628dc9eb456f1c33ac P 6c9fdd7eebba45f58a9ada76ded89610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a670c674d6284d8ea16a304dd4f2a04f" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 42011 } } peers { permanent_uuid: "6c9fdd7eebba45f58a9ada76ded89610" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46301 } } peers { permanent_uuid: "8c647b84fe56454e8ce009681a95b446" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41565 } }
I20250814 13:58:02.583534 14376 raft_consensus.cc:383] T d2036f06f6f741628dc9eb456f1c33ac P 6c9fdd7eebba45f58a9ada76ded89610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 13:58:02.583813 14376 raft_consensus.cc:738] T d2036f06f6f741628dc9eb456f1c33ac P 6c9fdd7eebba45f58a9ada76ded89610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c9fdd7eebba45f58a9ada76ded89610, State: Initialized, Role: FOLLOWER
I20250814 13:58:02.584688 14376 consensus_queue.cc:260] T d2036f06f6f741628dc9eb456f1c33ac P 6c9fdd7eebba45f58a9ada76ded89610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a670c674d6284d8ea16a304dd4f2a04f" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 42011 } } peers { permanent_uuid: "6c9fdd7eebba45f58a9ada76ded89610" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46301 } } peers { permanent_uuid: "8c647b84fe56454e8ce009681a95b446" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41565 } }
I20250814 13:58:02.588629 14376 ts_tablet_manager.cc:1428] T d2036f06f6f741628dc9eb456f1c33ac P 6c9fdd7eebba45f58a9ada76ded89610: Time spent starting tablet: real 0.030s user 0.024s sys 0.004s
I20250814 13:58:02.592801 14379 raft_consensus.cc:357] T d2036f06f6f741628dc9eb456f1c33ac P a670c674d6284d8ea16a304dd4f2a04f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a670c674d6284d8ea16a304dd4f2a04f" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 42011 } } peers { permanent_uuid: "6c9fdd7eebba45f58a9ada76ded89610" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46301 } } peers { permanent_uuid: "8c647b84fe56454e8ce009681a95b446" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41565 } }
I20250814 13:58:02.593959 14379 raft_consensus.cc:383] T d2036f06f6f741628dc9eb456f1c33ac P a670c674d6284d8ea16a304dd4f2a04f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 13:58:02.594283 14379 raft_consensus.cc:738] T d2036f06f6f741628dc9eb456f1c33ac P a670c674d6284d8ea16a304dd4f2a04f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a670c674d6284d8ea16a304dd4f2a04f, State: Initialized, Role: FOLLOWER
I20250814 13:58:02.595108 14379 consensus_queue.cc:260] T d2036f06f6f741628dc9eb456f1c33ac P a670c674d6284d8ea16a304dd4f2a04f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a670c674d6284d8ea16a304dd4f2a04f" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 42011 } } peers { permanent_uuid: "6c9fdd7eebba45f58a9ada76ded89610" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46301 } } peers { permanent_uuid: "8c647b84fe56454e8ce009681a95b446" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41565 } }
I20250814 13:58:02.598954 14357 heartbeater.cc:499] Master 127.11.132.190:43071 was elected leader, sending a full tablet report...
I20250814 13:58:02.600342 14379 ts_tablet_manager.cc:1428] T d2036f06f6f741628dc9eb456f1c33ac P a670c674d6284d8ea16a304dd4f2a04f: Time spent starting tablet: real 0.027s user 0.022s sys 0.003s
I20250814 13:58:02.600826 14378 raft_consensus.cc:357] T d2036f06f6f741628dc9eb456f1c33ac P 8c647b84fe56454e8ce009681a95b446 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a670c674d6284d8ea16a304dd4f2a04f" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 42011 } } peers { permanent_uuid: "6c9fdd7eebba45f58a9ada76ded89610" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46301 } } peers { permanent_uuid: "8c647b84fe56454e8ce009681a95b446" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41565 } }
I20250814 13:58:02.601495 14378 raft_consensus.cc:383] T d2036f06f6f741628dc9eb456f1c33ac P 8c647b84fe56454e8ce009681a95b446 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 13:58:02.601747 14378 raft_consensus.cc:738] T d2036f06f6f741628dc9eb456f1c33ac P 8c647b84fe56454e8ce009681a95b446 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c647b84fe56454e8ce009681a95b446, State: Initialized, Role: FOLLOWER
I20250814 13:58:02.602404 14378 consensus_queue.cc:260] T d2036f06f6f741628dc9eb456f1c33ac P 8c647b84fe56454e8ce009681a95b446 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a670c674d6284d8ea16a304dd4f2a04f" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 42011 } } peers { permanent_uuid: "6c9fdd7eebba45f58a9ada76ded89610" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46301 } } peers { permanent_uuid: "8c647b84fe56454e8ce009681a95b446" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41565 } }
I20250814 13:58:02.606232 14378 ts_tablet_manager.cc:1428] T d2036f06f6f741628dc9eb456f1c33ac P 8c647b84fe56454e8ce009681a95b446: Time spent starting tablet: real 0.031s user 0.024s sys 0.008s
W20250814 13:58:02.639904 14358 tablet.cc:2378] T d2036f06f6f741628dc9eb456f1c33ac P a670c674d6284d8ea16a304dd4f2a04f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250814 13:58:02.662974 14225 tablet.cc:2378] T d2036f06f6f741628dc9eb456f1c33ac P 6c9fdd7eebba45f58a9ada76ded89610: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250814 13:58:02.693404 14382 raft_consensus.cc:491] T d2036f06f6f741628dc9eb456f1c33ac P 6c9fdd7eebba45f58a9ada76ded89610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250814 13:58:02.693920 14382 raft_consensus.cc:513] T d2036f06f6f741628dc9eb456f1c33ac P 6c9fdd7eebba45f58a9ada76ded89610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a670c674d6284d8ea16a304dd4f2a04f" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 42011 } } peers { permanent_uuid: "6c9fdd7eebba45f58a9ada76ded89610" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46301 } } peers { permanent_uuid: "8c647b84fe56454e8ce009681a95b446" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41565 } }
I20250814 13:58:02.696062 14382 leader_election.cc:290] T d2036f06f6f741628dc9eb456f1c33ac P 6c9fdd7eebba45f58a9ada76ded89610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a670c674d6284d8ea16a304dd4f2a04f (127.11.132.131:42011), 8c647b84fe56454e8ce009681a95b446 (127.11.132.129:41565)
I20250814 13:58:02.704818 14383 raft_consensus.cc:491] T d2036f06f6f741628dc9eb456f1c33ac P a670c674d6284d8ea16a304dd4f2a04f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250814 13:58:02.705509 14383 raft_consensus.cc:513] T d2036f06f6f741628dc9eb456f1c33ac P a670c674d6284d8ea16a304dd4f2a04f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a670c674d6284d8ea16a304dd4f2a04f" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 42011 } } peers { permanent_uuid: "6c9fdd7eebba45f58a9ada76ded89610" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46301 } } peers { permanent_uuid: "8c647b84fe56454e8ce009681a95b446" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41565 } }
I20250814 13:58:02.707198 14312 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2036f06f6f741628dc9eb456f1c33ac" candidate_uuid: "6c9fdd7eebba45f58a9ada76ded89610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a670c674d6284d8ea16a304dd4f2a04f" is_pre_election: true
I20250814 13:58:02.708209 14312 raft_consensus.cc:2466] T d2036f06f6f741628dc9eb456f1c33ac P a670c674d6284d8ea16a304dd4f2a04f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c9fdd7eebba45f58a9ada76ded89610 in term 0.
I20250814 13:58:02.708267 14045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2036f06f6f741628dc9eb456f1c33ac" candidate_uuid: "6c9fdd7eebba45f58a9ada76ded89610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c647b84fe56454e8ce009681a95b446" is_pre_election: true
I20250814 13:58:02.709144 14045 raft_consensus.cc:2466] T d2036f06f6f741628dc9eb456f1c33ac P 8c647b84fe56454e8ce009681a95b446 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c9fdd7eebba45f58a9ada76ded89610 in term 0.
I20250814 13:58:02.709658 14114 leader_election.cc:304] T d2036f06f6f741628dc9eb456f1c33ac P 6c9fdd7eebba45f58a9ada76ded89610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c9fdd7eebba45f58a9ada76ded89610, a670c674d6284d8ea16a304dd4f2a04f; no voters:
I20250814 13:58:02.710598 14382 raft_consensus.cc:2802] T d2036f06f6f741628dc9eb456f1c33ac P 6c9fdd7eebba45f58a9ada76ded89610 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250814 13:58:02.710937 14382 raft_consensus.cc:491] T d2036f06f6f741628dc9eb456f1c33ac P 6c9fdd7eebba45f58a9ada76ded89610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250814 13:58:02.711230 14382 raft_consensus.cc:3058] T d2036f06f6f741628dc9eb456f1c33ac P 6c9fdd7eebba45f58a9ada76ded89610 [term 0 FOLLOWER]: Advancing to term 1
I20250814 13:58:02.712515 14383 leader_election.cc:290] T d2036f06f6f741628dc9eb456f1c33ac P a670c674d6284d8ea16a304dd4f2a04f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c9fdd7eebba45f58a9ada76ded89610 (127.11.132.130:46301), 8c647b84fe56454e8ce009681a95b446 (127.11.132.129:41565)
I20250814 13:58:02.718015 14382 raft_consensus.cc:513] T d2036f06f6f741628dc9eb456f1c33ac P 6c9fdd7eebba45f58a9ada76ded89610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a670c674d6284d8ea16a304dd4f2a04f" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 42011 } } peers { permanent_uuid: "6c9fdd7eebba45f58a9ada76ded89610" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46301 } } peers { permanent_uuid: "8c647b84fe56454e8ce009681a95b446" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41565 } }
I20250814 13:58:02.719826 14179 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2036f06f6f741628dc9eb456f1c33ac" candidate_uuid: "a670c674d6284d8ea16a304dd4f2a04f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c9fdd7eebba45f58a9ada76ded89610" is_pre_election: true
I20250814 13:58:02.720233 14312 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2036f06f6f741628dc9eb456f1c33ac" candidate_uuid: "6c9fdd7eebba45f58a9ada76ded89610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a670c674d6284d8ea16a304dd4f2a04f"
I20250814 13:58:02.720507 14045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2036f06f6f741628dc9eb456f1c33ac" candidate_uuid: "6c9fdd7eebba45f58a9ada76ded89610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c647b84fe56454e8ce009681a95b446"
I20250814 13:58:02.720713 14179 raft_consensus.cc:2391] T d2036f06f6f741628dc9eb456f1c33ac P 6c9fdd7eebba45f58a9ada76ded89610 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a670c674d6284d8ea16a304dd4f2a04f in current term 1: Already voted for candidate 6c9fdd7eebba45f58a9ada76ded89610 in this term.
I20250814 13:58:02.720731 14312 raft_consensus.cc:3058] T d2036f06f6f741628dc9eb456f1c33ac P a670c674d6284d8ea16a304dd4f2a04f [term 0 FOLLOWER]: Advancing to term 1
I20250814 13:58:02.720997 14045 raft_consensus.cc:3058] T d2036f06f6f741628dc9eb456f1c33ac P 8c647b84fe56454e8ce009681a95b446 [term 0 FOLLOWER]: Advancing to term 1
I20250814 13:58:02.723541 14382 leader_election.cc:290] T d2036f06f6f741628dc9eb456f1c33ac P 6c9fdd7eebba45f58a9ada76ded89610 [CANDIDATE]: Term 1 election: Requested vote from peers a670c674d6284d8ea16a304dd4f2a04f (127.11.132.131:42011), 8c647b84fe56454e8ce009681a95b446 (127.11.132.129:41565)
I20250814 13:58:02.723713 14044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2036f06f6f741628dc9eb456f1c33ac" candidate_uuid: "a670c674d6284d8ea16a304dd4f2a04f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c647b84fe56454e8ce009681a95b446" is_pre_election: true
I20250814 13:58:02.725373 14246 leader_election.cc:304] T d2036f06f6f741628dc9eb456f1c33ac P a670c674d6284d8ea16a304dd4f2a04f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a670c674d6284d8ea16a304dd4f2a04f; no voters: 6c9fdd7eebba45f58a9ada76ded89610, 8c647b84fe56454e8ce009681a95b446
I20250814 13:58:02.727730 14045 raft_consensus.cc:2466] T d2036f06f6f741628dc9eb456f1c33ac P 8c647b84fe56454e8ce009681a95b446 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c9fdd7eebba45f58a9ada76ded89610 in term 1.
I20250814 13:58:02.728122 14312 raft_consensus.cc:2466] T d2036f06f6f741628dc9eb456f1c33ac P a670c674d6284d8ea16a304dd4f2a04f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c9fdd7eebba45f58a9ada76ded89610 in term 1.
I20250814 13:58:02.728567 14383 raft_consensus.cc:2747] T d2036f06f6f741628dc9eb456f1c33ac P a670c674d6284d8ea16a304dd4f2a04f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250814 13:58:02.728838 14113 leader_election.cc:304] T d2036f06f6f741628dc9eb456f1c33ac P 6c9fdd7eebba45f58a9ada76ded89610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c9fdd7eebba45f58a9ada76ded89610, 8c647b84fe56454e8ce009681a95b446; no voters:
I20250814 13:58:02.729612 14382 raft_consensus.cc:2802] T d2036f06f6f741628dc9eb456f1c33ac P 6c9fdd7eebba45f58a9ada76ded89610 [term 1 FOLLOWER]: Leader election won for term 1
I20250814 13:58:02.731474 14382 raft_consensus.cc:695] T d2036f06f6f741628dc9eb456f1c33ac P 6c9fdd7eebba45f58a9ada76ded89610 [term 1 LEADER]: Becoming Leader. State: Replica: 6c9fdd7eebba45f58a9ada76ded89610, State: Running, Role: LEADER
I20250814 13:58:02.732214 14382 consensus_queue.cc:237] T d2036f06f6f741628dc9eb456f1c33ac P 6c9fdd7eebba45f58a9ada76ded89610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a670c674d6284d8ea16a304dd4f2a04f" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 42011 } } peers { permanent_uuid: "6c9fdd7eebba45f58a9ada76ded89610" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46301 } } peers { permanent_uuid: "8c647b84fe56454e8ce009681a95b446" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41565 } }
I20250814 13:58:02.742496 13902 catalog_manager.cc:5582] T d2036f06f6f741628dc9eb456f1c33ac P 6c9fdd7eebba45f58a9ada76ded89610 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c9fdd7eebba45f58a9ada76ded89610 (127.11.132.130). New cstate: current_term: 1 leader_uuid: "6c9fdd7eebba45f58a9ada76ded89610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a670c674d6284d8ea16a304dd4f2a04f" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 42011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c9fdd7eebba45f58a9ada76ded89610" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c647b84fe56454e8ce009681a95b446" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41565 } health_report { overall_health: UNKNOWN } } }
I20250814 13:58:02.765772 11794 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250814 13:58:02.768934 11794 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 8c647b84fe56454e8ce009681a95b446 to finish bootstrapping
I20250814 13:58:02.780330 11794 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 6c9fdd7eebba45f58a9ada76ded89610 to finish bootstrapping
I20250814 13:58:02.789860 11794 ts_itest-base.cc:246] Waiting for 1 tablets on tserver a670c674d6284d8ea16a304dd4f2a04f to finish bootstrapping
W20250814 13:58:02.817615 14091 tablet.cc:2378] T d2036f06f6f741628dc9eb456f1c33ac P 8c647b84fe56454e8ce009681a95b446: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250814 13:58:03.149830 14382 consensus_queue.cc:1035] T d2036f06f6f741628dc9eb456f1c33ac P 6c9fdd7eebba45f58a9ada76ded89610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a670c674d6284d8ea16a304dd4f2a04f" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 42011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250814 13:58:03.170899 14382 consensus_queue.cc:1035] T d2036f06f6f741628dc9eb456f1c33ac P 6c9fdd7eebba45f58a9ada76ded89610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c647b84fe56454e8ce009681a95b446" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 41565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250814 13:58:04.440773 13902 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.RefreshAuthzCache from {username='slave'} at 127.0.0.1:58614
I20250814 13:58:05.469411 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 13961
I20250814 13:58:05.493584 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 14094
I20250814 13:58:05.517289 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 14228
I20250814 13:58:05.545364 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 13869
2025-08-14T13:58:05Z chronyd exiting
[ OK ] AdminCliTest.TestAuthzResetCacheNotAuthorized (11801 ms)
[ RUN ] AdminCliTest.TestRebuildTables
I20250814 13:58:05.594422 11794 test_util.cc:276] Using random seed: -1449588964
I20250814 13:58:05.598064 11794 ts_itest-base.cc:115] Starting cluster with:
I20250814 13:58:05.598222 11794 ts_itest-base.cc:116] --------------
I20250814 13:58:05.598390 11794 ts_itest-base.cc:117] 3 tablet servers
I20250814 13:58:05.598531 11794 ts_itest-base.cc:118] 3 replicas per TS
I20250814 13:58:05.598673 11794 ts_itest-base.cc:119] --------------
2025-08-14T13:58:05Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-14T13:58:05Z Disabled control of system clock
I20250814 13:58:05.632359 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.190:39441
--webserver_interface=127.11.132.190
--webserver_port=0
--builtin_ntp_servers=127.11.132.148:37521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.132.190:39441 with env {}
W20250814 13:58:05.913955 14425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:58:05.914430 14425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:58:05.914798 14425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:58:05.943502 14425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250814 13:58:05.943763 14425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:58:05.943943 14425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250814 13:58:05.944118 14425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250814 13:58:05.976472 14425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:37521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.132.190:39441
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.190:39441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.11.132.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:58:05.977692 14425 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:58:05.979152 14425 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:58:05.989849 14431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:58:07.392948 14430 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 14425
W20250814 13:58:05.990396 14432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:58:07.723387 14430 kernel_stack_watchdog.cc:198] Thread 14425 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250814 13:58:07.723966 14425 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.734s user 0.000s sys 0.002s
W20250814 13:58:07.724040 14433 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1733 milliseconds
W20250814 13:58:07.724293 14425 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.735s user 0.000s sys 0.002s
W20250814 13:58:07.725394 14434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 13:58:07.725335 14425 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 13:58:07.728168 14425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:58:07.730548 14425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:58:07.731873 14425 hybrid_clock.cc:648] HybridClock initialized: now 1755179887731826 us; error 47 us; skew 500 ppm
I20250814 13:58:07.732601 14425 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:58:07.738045 14425 webserver.cc:480] Webserver started at http://127.11.132.190:45161/ using document root <none> and password file <none>
I20250814 13:58:07.738955 14425 fs_manager.cc:362] Metadata directory not provided
I20250814 13:58:07.739166 14425 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:58:07.739655 14425 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 13:58:07.743968 14425 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "2ff0f6a4351645cd82f1dc14d3ef065d"
format_stamp: "Formatted at 2025-08-14 13:58:07 on dist-test-slave-q1tc"
I20250814 13:58:07.745046 14425 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "2ff0f6a4351645cd82f1dc14d3ef065d"
format_stamp: "Formatted at 2025-08-14 13:58:07 on dist-test-slave-q1tc"
I20250814 13:58:07.752108 14425 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250814 13:58:07.757136 14441 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:58:07.758036 14425 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250814 13:58:07.758320 14425 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
uuid: "2ff0f6a4351645cd82f1dc14d3ef065d"
format_stamp: "Formatted at 2025-08-14 13:58:07 on dist-test-slave-q1tc"
I20250814 13:58:07.758644 14425 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:58:07.808648 14425 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:58:07.810094 14425 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:58:07.810523 14425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:58:07.876113 14425 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.190:39441
I20250814 13:58:07.876194 14492 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.190:39441 every 8 connection(s)
I20250814 13:58:07.878623 14425 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250814 13:58:07.882751 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 14425
I20250814 13:58:07.883237 11794 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250814 13:58:07.883984 14493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:58:07.905547 14493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d: Bootstrap starting.
I20250814 13:58:07.911226 14493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d: Neither blocks nor log segments found. Creating new log.
I20250814 13:58:07.912709 14493 log.cc:826] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d: Log is configured to *not* fsync() on all Append() calls
I20250814 13:58:07.917192 14493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d: No bootstrap required, opened a new log
I20250814 13:58:07.933796 14493 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff0f6a4351645cd82f1dc14d3ef065d" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 39441 } }
I20250814 13:58:07.934298 14493 raft_consensus.cc:383] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 13:58:07.934504 14493 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ff0f6a4351645cd82f1dc14d3ef065d, State: Initialized, Role: FOLLOWER
I20250814 13:58:07.935050 14493 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff0f6a4351645cd82f1dc14d3ef065d" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 39441 } }
I20250814 13:58:07.935468 14493 raft_consensus.cc:397] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 13:58:07.935698 14493 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 13:58:07.935957 14493 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [term 0 FOLLOWER]: Advancing to term 1
I20250814 13:58:07.940238 14493 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff0f6a4351645cd82f1dc14d3ef065d" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 39441 } }
I20250814 13:58:07.940865 14493 leader_election.cc:304] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ff0f6a4351645cd82f1dc14d3ef065d; no voters:
I20250814 13:58:07.942485 14493 leader_election.cc:290] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250814 13:58:07.943091 14498 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [term 1 FOLLOWER]: Leader election won for term 1
I20250814 13:58:07.945406 14498 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [term 1 LEADER]: Becoming Leader. State: Replica: 2ff0f6a4351645cd82f1dc14d3ef065d, State: Running, Role: LEADER
I20250814 13:58:07.946298 14493 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [sys.catalog]: configured and running, proceeding with master startup.
I20250814 13:58:07.946161 14498 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff0f6a4351645cd82f1dc14d3ef065d" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 39441 } }
I20250814 13:58:07.955092 14499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ff0f6a4351645cd82f1dc14d3ef065d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff0f6a4351645cd82f1dc14d3ef065d" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 39441 } } }
I20250814 13:58:07.955848 14499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [sys.catalog]: This master's current role is: LEADER
I20250814 13:58:07.958096 14500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ff0f6a4351645cd82f1dc14d3ef065d. Latest consensus state: current_term: 1 leader_uuid: "2ff0f6a4351645cd82f1dc14d3ef065d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff0f6a4351645cd82f1dc14d3ef065d" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 39441 } } }
I20250814 13:58:07.958876 14500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [sys.catalog]: This master's current role is: LEADER
I20250814 13:58:07.961483 14507 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250814 13:58:07.971961 14507 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250814 13:58:07.980252 14515 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250814 13:58:07.987483 14507 catalog_manager.cc:1349] Generated new cluster ID: da09c53bd0d74cc9a334120407ff837f
I20250814 13:58:07.987717 14507 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250814 13:58:08.001843 14507 catalog_manager.cc:1372] Generated new certificate authority record
I20250814 13:58:08.003082 14507 catalog_manager.cc:1506] Loading token signing keys...
I20250814 13:58:08.015690 14507 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d: Generated new TSK 0
I20250814 13:58:08.016400 14507 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250814 13:58:08.027498 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.132.129:0
--local_ip_for_outbound_sockets=127.11.132.129
--webserver_interface=127.11.132.129
--webserver_port=0
--tserver_master_addrs=127.11.132.190:39441
--builtin_ntp_servers=127.11.132.148:37521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250814 13:58:08.341645 14517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:58:08.342110 14517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:58:08.342725 14517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:58:08.371232 14517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:58:08.372079 14517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.132.129
I20250814 13:58:08.405121 14517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:37521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.132.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.11.132.129
--webserver_port=0
--tserver_master_addrs=127.11.132.190:39441
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.132.129
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:58:08.406385 14517 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:58:08.407876 14517 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:58:08.419287 14523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:58:08.420843 14524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:58:09.822367 14522 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 14517
W20250814 13:58:09.843250 14517 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.422s user 0.433s sys 0.958s
W20250814 13:58:09.844674 14517 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.424s user 0.433s sys 0.959s
W20250814 13:58:09.848564 14525 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1422 milliseconds
W20250814 13:58:09.849668 14526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 13:58:09.849742 14517 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 13:58:09.850872 14517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:58:09.853253 14517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:58:09.854713 14517 hybrid_clock.cc:648] HybridClock initialized: now 1755179889854674 us; error 62 us; skew 500 ppm
I20250814 13:58:09.855456 14517 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:58:09.862330 14517 webserver.cc:480] Webserver started at http://127.11.132.129:44059/ using document root <none> and password file <none>
I20250814 13:58:09.863219 14517 fs_manager.cc:362] Metadata directory not provided
I20250814 13:58:09.863420 14517 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:58:09.863812 14517 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 13:58:09.867887 14517 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "eff1057ae98849458f656363167a5337"
format_stamp: "Formatted at 2025-08-14 13:58:09 on dist-test-slave-q1tc"
I20250814 13:58:09.868914 14517 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "eff1057ae98849458f656363167a5337"
format_stamp: "Formatted at 2025-08-14 13:58:09 on dist-test-slave-q1tc"
I20250814 13:58:09.876104 14517 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20250814 13:58:09.888870 14534 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:58:09.890501 14517 fs_manager.cc:730] Time spent opening block manager: real 0.010s user 0.005s sys 0.006s
I20250814 13:58:09.890854 14517 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "eff1057ae98849458f656363167a5337"
format_stamp: "Formatted at 2025-08-14 13:58:09 on dist-test-slave-q1tc"
I20250814 13:58:09.891251 14517 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:58:09.961970 14517 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:58:09.963397 14517 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:58:09.963785 14517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:58:09.966745 14517 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 13:58:09.970616 14517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 13:58:09.970795 14517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 13:58:09.971072 14517 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 13:58:09.971225 14517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 13:58:10.140003 14517 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.129:42009
I20250814 13:58:10.140107 14646 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.129:42009 every 8 connection(s)
I20250814 13:58:10.142496 14517 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250814 13:58:10.149868 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 14517
I20250814 13:58:10.150597 11794 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250814 13:58:10.158044 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.132.130:0
--local_ip_for_outbound_sockets=127.11.132.130
--webserver_interface=127.11.132.130
--webserver_port=0
--tserver_master_addrs=127.11.132.190:39441
--builtin_ntp_servers=127.11.132.148:37521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250814 13:58:10.167726 14647 heartbeater.cc:344] Connected to a master server at 127.11.132.190:39441
I20250814 13:58:10.168185 14647 heartbeater.cc:461] Registering TS with master...
I20250814 13:58:10.169458 14647 heartbeater.cc:507] Master 127.11.132.190:39441 requested a full tablet report, sending...
I20250814 13:58:10.172269 14458 ts_manager.cc:194] Registered new tserver with Master: eff1057ae98849458f656363167a5337 (127.11.132.129:42009)
I20250814 13:58:10.174983 14458 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.132.129:46995
W20250814 13:58:10.435773 14651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:58:10.436273 14651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:58:10.436882 14651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:58:10.466609 14651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:58:10.467358 14651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.132.130
I20250814 13:58:10.498705 14651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:37521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.132.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.11.132.130
--webserver_port=0
--tserver_master_addrs=127.11.132.190:39441
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.132.130
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:58:10.499991 14651 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:58:10.501453 14651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:58:10.511858 14657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 13:58:11.178828 14647 heartbeater.cc:499] Master 127.11.132.190:39441 was elected leader, sending a full tablet report...
W20250814 13:58:10.513461 14658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:58:10.516387 14660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:58:11.618270 14659 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250814 13:58:11.618377 14651 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 13:58:11.621874 14651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:58:11.623870 14651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:58:11.625149 14651 hybrid_clock.cc:648] HybridClock initialized: now 1755179891625115 us; error 44 us; skew 500 ppm
I20250814 13:58:11.625871 14651 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:58:11.631857 14651 webserver.cc:480] Webserver started at http://127.11.132.130:44439/ using document root <none> and password file <none>
I20250814 13:58:11.632653 14651 fs_manager.cc:362] Metadata directory not provided
I20250814 13:58:11.632844 14651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:58:11.633328 14651 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 13:58:11.637450 14651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "513c7d5f4e3f41e5b9f8918182af4955"
format_stamp: "Formatted at 2025-08-14 13:58:11 on dist-test-slave-q1tc"
I20250814 13:58:11.638478 14651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "513c7d5f4e3f41e5b9f8918182af4955"
format_stamp: "Formatted at 2025-08-14 13:58:11 on dist-test-slave-q1tc"
I20250814 13:58:11.645005 14651 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250814 13:58:11.650313 14667 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:58:11.651274 14651 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250814 13:58:11.651558 14651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "513c7d5f4e3f41e5b9f8918182af4955"
format_stamp: "Formatted at 2025-08-14 13:58:11 on dist-test-slave-q1tc"
I20250814 13:58:11.651862 14651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:58:11.714759 14651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:58:11.716168 14651 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:58:11.716565 14651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:58:11.718959 14651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 13:58:11.722818 14651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 13:58:11.723018 14651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 13:58:11.723232 14651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 13:58:11.723372 14651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 13:58:11.849071 14651 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.130:42415
I20250814 13:58:11.849187 14779 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.130:42415 every 8 connection(s)
I20250814 13:58:11.851382 14651 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250814 13:58:11.854518 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 14651
I20250814 13:58:11.855000 11794 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250814 13:58:11.861726 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.132.131:0
--local_ip_for_outbound_sockets=127.11.132.131
--webserver_interface=127.11.132.131
--webserver_port=0
--tserver_master_addrs=127.11.132.190:39441
--builtin_ntp_servers=127.11.132.148:37521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250814 13:58:11.873246 14780 heartbeater.cc:344] Connected to a master server at 127.11.132.190:39441
I20250814 13:58:11.873610 14780 heartbeater.cc:461] Registering TS with master...
I20250814 13:58:11.874493 14780 heartbeater.cc:507] Master 127.11.132.190:39441 requested a full tablet report, sending...
I20250814 13:58:11.876334 14458 ts_manager.cc:194] Registered new tserver with Master: 513c7d5f4e3f41e5b9f8918182af4955 (127.11.132.130:42415)
I20250814 13:58:11.877460 14458 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.132.130:33561
W20250814 13:58:12.153857 14784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:58:12.154305 14784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:58:12.154780 14784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:58:12.186184 14784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:58:12.186978 14784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.132.131
I20250814 13:58:12.221211 14784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:37521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.132.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.11.132.131
--webserver_port=0
--tserver_master_addrs=127.11.132.190:39441
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.132.131
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:58:12.222458 14784 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:58:12.223982 14784 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:58:12.235802 14790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 13:58:12.880384 14780 heartbeater.cc:499] Master 127.11.132.190:39441 was elected leader, sending a full tablet report...
W20250814 13:58:12.237380 14791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:58:12.240432 14793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:58:13.356156 14792 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250814 13:58:13.356230 14784 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 13:58:13.359774 14784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:58:13.361850 14784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:58:13.363149 14784 hybrid_clock.cc:648] HybridClock initialized: now 1755179893363126 us; error 42 us; skew 500 ppm
I20250814 13:58:13.363854 14784 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:58:13.369908 14784 webserver.cc:480] Webserver started at http://127.11.132.131:39163/ using document root <none> and password file <none>
I20250814 13:58:13.370775 14784 fs_manager.cc:362] Metadata directory not provided
I20250814 13:58:13.370971 14784 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:58:13.371405 14784 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 13:58:13.375303 14784 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "ac4ac7370c8c4b0c967fa990af77a067"
format_stamp: "Formatted at 2025-08-14 13:58:13 on dist-test-slave-q1tc"
I20250814 13:58:13.376266 14784 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "ac4ac7370c8c4b0c967fa990af77a067"
format_stamp: "Formatted at 2025-08-14 13:58:13 on dist-test-slave-q1tc"
I20250814 13:58:13.382900 14784 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20250814 13:58:13.388293 14800 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:58:13.389333 14784 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250814 13:58:13.389617 14784 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ac4ac7370c8c4b0c967fa990af77a067"
format_stamp: "Formatted at 2025-08-14 13:58:13 on dist-test-slave-q1tc"
I20250814 13:58:13.389940 14784 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:58:13.440765 14784 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:58:13.442112 14784 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:58:13.442521 14784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:58:13.445149 14784 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 13:58:13.449290 14784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 13:58:13.449492 14784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 13:58:13.449720 14784 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 13:58:13.449883 14784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 13:58:13.573367 14784 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.131:44563
I20250814 13:58:13.573441 14912 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.131:44563 every 8 connection(s)
I20250814 13:58:13.575743 14784 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250814 13:58:13.584133 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 14784
I20250814 13:58:13.584621 11794 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250814 13:58:13.599674 14913 heartbeater.cc:344] Connected to a master server at 127.11.132.190:39441
I20250814 13:58:13.600070 14913 heartbeater.cc:461] Registering TS with master...
I20250814 13:58:13.601235 14913 heartbeater.cc:507] Master 127.11.132.190:39441 requested a full tablet report, sending...
I20250814 13:58:13.603305 14457 ts_manager.cc:194] Registered new tserver with Master: ac4ac7370c8c4b0c967fa990af77a067 (127.11.132.131:44563)
I20250814 13:58:13.604694 14457 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.132.131:47411
I20250814 13:58:13.617153 11794 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250814 13:58:13.649915 11794 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250814 13:58:13.650225 11794 test_util.cc:276] Using random seed: -1441533156
I20250814 13:58:13.687186 14457 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55978:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250814 13:58:13.724714 14582 tablet_service.cc:1468] Processing CreateTablet for tablet 24df6cb14940420ea85c32a197b4fbc6 (DEFAULT_TABLE table=TestTable [id=28a385c4010b4ee2a25b28115a35fcae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 13:58:13.726166 14582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24df6cb14940420ea85c32a197b4fbc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:58:13.743222 14933 tablet_bootstrap.cc:492] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337: Bootstrap starting.
I20250814 13:58:13.748502 14933 tablet_bootstrap.cc:654] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337: Neither blocks nor log segments found. Creating new log.
I20250814 13:58:13.750257 14933 log.cc:826] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337: Log is configured to *not* fsync() on all Append() calls
I20250814 13:58:13.754875 14933 tablet_bootstrap.cc:492] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337: No bootstrap required, opened a new log
I20250814 13:58:13.755249 14933 ts_tablet_manager.cc:1397] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20250814 13:58:13.771168 14933 raft_consensus.cc:357] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } }
I20250814 13:58:13.771661 14933 raft_consensus.cc:383] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 13:58:13.771915 14933 raft_consensus.cc:738] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff1057ae98849458f656363167a5337, State: Initialized, Role: FOLLOWER
I20250814 13:58:13.772516 14933 consensus_queue.cc:260] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } }
I20250814 13:58:13.773015 14933 raft_consensus.cc:397] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 13:58:13.773324 14933 raft_consensus.cc:491] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 13:58:13.773622 14933 raft_consensus.cc:3058] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [term 0 FOLLOWER]: Advancing to term 1
I20250814 13:58:13.777546 14933 raft_consensus.cc:513] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } }
I20250814 13:58:13.778213 14933 leader_election.cc:304] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eff1057ae98849458f656363167a5337; no voters:
I20250814 13:58:13.779819 14933 leader_election.cc:290] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250814 13:58:13.780189 14935 raft_consensus.cc:2802] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [term 1 FOLLOWER]: Leader election won for term 1
I20250814 13:58:13.782668 14935 raft_consensus.cc:695] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [term 1 LEADER]: Becoming Leader. State: Replica: eff1057ae98849458f656363167a5337, State: Running, Role: LEADER
I20250814 13:58:13.783459 14935 consensus_queue.cc:237] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } }
I20250814 13:58:13.784132 14933 ts_tablet_manager.cc:1428] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337: Time spent starting tablet: real 0.029s user 0.027s sys 0.003s
I20250814 13:58:13.795123 14457 catalog_manager.cc:5582] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 reported cstate change: term changed from 0 to 1, leader changed from <none> to eff1057ae98849458f656363167a5337 (127.11.132.129). New cstate: current_term: 1 leader_uuid: "eff1057ae98849458f656363167a5337" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } health_report { overall_health: HEALTHY } } }
I20250814 13:58:13.993367 11794 test_util.cc:276] Using random seed: -1441190025
I20250814 13:58:14.013593 14452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55982:
name: "TestTable1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250814 13:58:14.042352 14848 tablet_service.cc:1468] Processing CreateTablet for tablet e5e55ced4bfc4fb6b6991fc92964cbaa (DEFAULT_TABLE table=TestTable1 [id=00f83533c0714c49bdef143ab29ed1e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 13:58:14.043802 14848 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5e55ced4bfc4fb6b6991fc92964cbaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:58:14.062565 14954 tablet_bootstrap.cc:492] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067: Bootstrap starting.
I20250814 13:58:14.067951 14954 tablet_bootstrap.cc:654] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067: Neither blocks nor log segments found. Creating new log.
I20250814 13:58:14.069604 14954 log.cc:826] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067: Log is configured to *not* fsync() on all Append() calls
I20250814 13:58:14.073640 14954 tablet_bootstrap.cc:492] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067: No bootstrap required, opened a new log
I20250814 13:58:14.074067 14954 ts_tablet_manager.cc:1397] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20250814 13:58:14.090886 14954 raft_consensus.cc:357] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } }
I20250814 13:58:14.091401 14954 raft_consensus.cc:383] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 13:58:14.091601 14954 raft_consensus.cc:738] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac4ac7370c8c4b0c967fa990af77a067, State: Initialized, Role: FOLLOWER
I20250814 13:58:14.092244 14954 consensus_queue.cc:260] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } }
I20250814 13:58:14.092736 14954 raft_consensus.cc:397] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 13:58:14.092986 14954 raft_consensus.cc:491] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 13:58:14.093343 14954 raft_consensus.cc:3058] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [term 0 FOLLOWER]: Advancing to term 1
I20250814 13:58:14.097342 14954 raft_consensus.cc:513] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } }
I20250814 13:58:14.097929 14954 leader_election.cc:304] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac4ac7370c8c4b0c967fa990af77a067; no voters:
I20250814 13:58:14.099879 14954 leader_election.cc:290] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250814 13:58:14.100275 14956 raft_consensus.cc:2802] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [term 1 FOLLOWER]: Leader election won for term 1
I20250814 13:58:14.102658 14956 raft_consensus.cc:695] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [term 1 LEADER]: Becoming Leader. State: Replica: ac4ac7370c8c4b0c967fa990af77a067, State: Running, Role: LEADER
I20250814 13:58:14.103324 14913 heartbeater.cc:499] Master 127.11.132.190:39441 was elected leader, sending a full tablet report...
I20250814 13:58:14.103454 14956 consensus_queue.cc:237] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } }
I20250814 13:58:14.103827 14954 ts_tablet_manager.cc:1428] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067: Time spent starting tablet: real 0.030s user 0.023s sys 0.006s
I20250814 13:58:14.113803 14452 catalog_manager.cc:5582] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac4ac7370c8c4b0c967fa990af77a067 (127.11.132.131). New cstate: current_term: 1 leader_uuid: "ac4ac7370c8c4b0c967fa990af77a067" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } health_report { overall_health: HEALTHY } } }
I20250814 13:58:14.314368 11794 test_util.cc:276] Using random seed: -1440869015
I20250814 13:58:14.335433 14457 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55994:
name: "TestTable2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250814 13:58:14.362644 14715 tablet_service.cc:1468] Processing CreateTablet for tablet 1db867af4d4b43dca1cf4ddc26d3321b (DEFAULT_TABLE table=TestTable2 [id=730812e3c05f49c8980b7153f836960f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 13:58:14.364020 14715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1db867af4d4b43dca1cf4ddc26d3321b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:58:14.382498 14975 tablet_bootstrap.cc:492] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955: Bootstrap starting.
I20250814 13:58:14.387981 14975 tablet_bootstrap.cc:654] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955: Neither blocks nor log segments found. Creating new log.
I20250814 13:58:14.389659 14975 log.cc:826] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955: Log is configured to *not* fsync() on all Append() calls
I20250814 13:58:14.394098 14975 tablet_bootstrap.cc:492] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955: No bootstrap required, opened a new log
I20250814 13:58:14.394490 14975 ts_tablet_manager.cc:1397] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.010s
I20250814 13:58:14.411461 14975 raft_consensus.cc:357] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } }
I20250814 13:58:14.412025 14975 raft_consensus.cc:383] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 13:58:14.412225 14975 raft_consensus.cc:738] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 513c7d5f4e3f41e5b9f8918182af4955, State: Initialized, Role: FOLLOWER
I20250814 13:58:14.412820 14975 consensus_queue.cc:260] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } }
I20250814 13:58:14.413360 14975 raft_consensus.cc:397] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 13:58:14.413610 14975 raft_consensus.cc:491] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 13:58:14.414007 14975 raft_consensus.cc:3058] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955 [term 0 FOLLOWER]: Advancing to term 1
I20250814 13:58:14.418017 14975 raft_consensus.cc:513] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } }
I20250814 13:58:14.418653 14975 leader_election.cc:304] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 513c7d5f4e3f41e5b9f8918182af4955; no voters:
I20250814 13:58:14.420415 14975 leader_election.cc:290] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250814 13:58:14.420823 14977 raft_consensus.cc:2802] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955 [term 1 FOLLOWER]: Leader election won for term 1
I20250814 13:58:14.423261 14977 raft_consensus.cc:695] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955 [term 1 LEADER]: Becoming Leader. State: Replica: 513c7d5f4e3f41e5b9f8918182af4955, State: Running, Role: LEADER
I20250814 13:58:14.424156 14977 consensus_queue.cc:237] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } }
I20250814 13:58:14.424743 14975 ts_tablet_manager.cc:1428] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955: Time spent starting tablet: real 0.030s user 0.030s sys 0.000s
I20250814 13:58:14.435154 14457 catalog_manager.cc:5582] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955 reported cstate change: term changed from 0 to 1, leader changed from <none> to 513c7d5f4e3f41e5b9f8918182af4955 (127.11.132.130). New cstate: current_term: 1 leader_uuid: "513c7d5f4e3f41e5b9f8918182af4955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } health_report { overall_health: HEALTHY } } }
I20250814 13:58:14.632357 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 14425
W20250814 13:58:14.809477 14647 heartbeater.cc:646] Failed to heartbeat to 127.11.132.190:39441 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.132.190:39441: connect: Connection refused (error 111)
W20250814 13:58:15.130347 14913 heartbeater.cc:646] Failed to heartbeat to 127.11.132.190:39441 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.132.190:39441: connect: Connection refused (error 111)
W20250814 13:58:15.451606 14780 heartbeater.cc:646] Failed to heartbeat to 127.11.132.190:39441 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.132.190:39441: connect: Connection refused (error 111)
I20250814 13:58:19.371491 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 14517
I20250814 13:58:19.394505 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 14651
I20250814 13:58:19.421686 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 14784
I20250814 13:58:19.447021 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.190:39441
--webserver_interface=127.11.132.190
--webserver_port=45161
--builtin_ntp_servers=127.11.132.148:37521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.132.190:39441 with env {}
W20250814 13:58:19.747937 15052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:58:19.748718 15052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:58:19.749557 15052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:58:19.785724 15052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250814 13:58:19.785995 15052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:58:19.786207 15052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250814 13:58:19.786412 15052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250814 13:58:19.819906 15052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:37521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.132.190:39441
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.190:39441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.11.132.190
--webserver_port=45161
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:58:19.821190 15052 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:58:19.822763 15052 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:58:19.833622 15058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:58:21.235924 15057 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 15052
W20250814 13:58:19.834173 15059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:58:21.287436 15052 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.454s user 0.457s sys 0.992s
W20250814 13:58:21.288311 15052 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.455s user 0.457s sys 0.993s
W20250814 13:58:21.289369 15061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:58:21.291776 15060 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1455 milliseconds
I20250814 13:58:21.291831 15052 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 13:58:21.292999 15052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:58:21.295384 15052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:58:21.296697 15052 hybrid_clock.cc:648] HybridClock initialized: now 1755179901296660 us; error 48 us; skew 500 ppm
I20250814 13:58:21.297511 15052 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:58:21.303390 15052 webserver.cc:480] Webserver started at http://127.11.132.190:45161/ using document root <none> and password file <none>
I20250814 13:58:21.304193 15052 fs_manager.cc:362] Metadata directory not provided
I20250814 13:58:21.304409 15052 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:58:21.311620 15052 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.001s
I20250814 13:58:21.315905 15069 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:58:21.316897 15052 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250814 13:58:21.317217 15052 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
uuid: "2ff0f6a4351645cd82f1dc14d3ef065d"
format_stamp: "Formatted at 2025-08-14 13:58:07 on dist-test-slave-q1tc"
I20250814 13:58:21.318998 15052 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:58:21.365729 15052 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:58:21.367064 15052 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:58:21.367497 15052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:58:21.435060 15052 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.190:39441
I20250814 13:58:21.435135 15120 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.190:39441 every 8 connection(s)
I20250814 13:58:21.437695 15052 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250814 13:58:21.446532 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 15052
I20250814 13:58:21.448029 15121 sys_catalog.cc:263] Verifying existing consensus state
I20250814 13:58:21.448662 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.132.129:42009
--local_ip_for_outbound_sockets=127.11.132.129
--tserver_master_addrs=127.11.132.190:39441
--webserver_port=44059
--webserver_interface=127.11.132.129
--builtin_ntp_servers=127.11.132.148:37521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250814 13:58:21.455482 15121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d: Bootstrap starting.
I20250814 13:58:21.465231 15121 log.cc:826] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d: Log is configured to *not* fsync() on all Append() calls
I20250814 13:58:21.508329 15121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=13 ignored=0} mutations{seen=10 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 13:58:21.509133 15121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d: Bootstrap complete.
I20250814 13:58:21.527534 15121 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff0f6a4351645cd82f1dc14d3ef065d" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 39441 } }
I20250814 13:58:21.529464 15121 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ff0f6a4351645cd82f1dc14d3ef065d, State: Initialized, Role: FOLLOWER
I20250814 13:58:21.530133 15121 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff0f6a4351645cd82f1dc14d3ef065d" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 39441 } }
I20250814 13:58:21.530755 15121 raft_consensus.cc:397] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 13:58:21.531005 15121 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 13:58:21.531324 15121 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [term 2 FOLLOWER]: Advancing to term 3
I20250814 13:58:21.536267 15121 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff0f6a4351645cd82f1dc14d3ef065d" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 39441 } }
I20250814 13:58:21.536782 15121 leader_election.cc:304] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ff0f6a4351645cd82f1dc14d3ef065d; no voters:
I20250814 13:58:21.538798 15121 leader_election.cc:290] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [CANDIDATE]: Term 3 election: Requested vote from peers
I20250814 13:58:21.539176 15125 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [term 3 FOLLOWER]: Leader election won for term 3
I20250814 13:58:21.542130 15125 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [term 3 LEADER]: Becoming Leader. State: Replica: 2ff0f6a4351645cd82f1dc14d3ef065d, State: Running, Role: LEADER
I20250814 13:58:21.542889 15125 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff0f6a4351645cd82f1dc14d3ef065d" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 39441 } }
I20250814 13:58:21.543401 15121 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [sys.catalog]: configured and running, proceeding with master startup.
I20250814 13:58:21.552459 15127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ff0f6a4351645cd82f1dc14d3ef065d. Latest consensus state: current_term: 3 leader_uuid: "2ff0f6a4351645cd82f1dc14d3ef065d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff0f6a4351645cd82f1dc14d3ef065d" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 39441 } } }
I20250814 13:58:21.553687 15126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2ff0f6a4351645cd82f1dc14d3ef065d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff0f6a4351645cd82f1dc14d3ef065d" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 39441 } } }
I20250814 13:58:21.555840 15126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [sys.catalog]: This master's current role is: LEADER
I20250814 13:58:21.553037 15127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [sys.catalog]: This master's current role is: LEADER
I20250814 13:58:21.559891 15133 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250814 13:58:21.574311 15133 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=730812e3c05f49c8980b7153f836960f]
I20250814 13:58:21.575888 15133 catalog_manager.cc:671] Loaded metadata for table TestTable [id=7f20d84458954c4db411207565551a70]
I20250814 13:58:21.577801 15133 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=dbc2e364d16e40cb809d6ebb7a2c8df6]
I20250814 13:58:21.586761 15133 tablet_loader.cc:96] loaded metadata for tablet 1db867af4d4b43dca1cf4ddc26d3321b (table TestTable2 [id=730812e3c05f49c8980b7153f836960f])
I20250814 13:58:21.588251 15133 tablet_loader.cc:96] loaded metadata for tablet 24df6cb14940420ea85c32a197b4fbc6 (table TestTable [id=7f20d84458954c4db411207565551a70])
I20250814 13:58:21.589725 15133 tablet_loader.cc:96] loaded metadata for tablet e5e55ced4bfc4fb6b6991fc92964cbaa (table TestTable1 [id=dbc2e364d16e40cb809d6ebb7a2c8df6])
I20250814 13:58:21.591223 15133 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250814 13:58:21.597301 15133 catalog_manager.cc:1261] Loaded cluster ID: da09c53bd0d74cc9a334120407ff837f
I20250814 13:58:21.597621 15133 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250814 13:58:21.606541 15133 catalog_manager.cc:1506] Loading token signing keys...
I20250814 13:58:21.612336 15133 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d: Loaded TSK: 0
I20250814 13:58:21.614084 15133 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250814 13:58:21.616873 15143 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20250814 13:58:21.765679 15123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:58:21.766140 15123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:58:21.766619 15123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:58:21.796674 15123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:58:21.797519 15123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.132.129
I20250814 13:58:21.830142 15123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:37521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.132.129:42009
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.11.132.129
--webserver_port=44059
--tserver_master_addrs=127.11.132.190:39441
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.132.129
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:58:21.831439 15123 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:58:21.832986 15123 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:58:21.844409 15149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:58:23.252264 15148 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 15123
W20250814 13:58:21.845942 15150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:58:23.590761 15123 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.745s user 0.550s sys 1.084s
W20250814 13:58:23.591140 15123 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.745s user 0.550s sys 1.084s
W20250814 13:58:23.593143 15152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:58:23.595256 15151 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1745 milliseconds
I20250814 13:58:23.595399 15123 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 13:58:23.596586 15123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:58:23.599015 15123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:58:23.600450 15123 hybrid_clock.cc:648] HybridClock initialized: now 1755179903600389 us; error 64 us; skew 500 ppm
I20250814 13:58:23.601240 15123 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:58:23.607280 15123 webserver.cc:480] Webserver started at http://127.11.132.129:44059/ using document root <none> and password file <none>
I20250814 13:58:23.608023 15123 fs_manager.cc:362] Metadata directory not provided
I20250814 13:58:23.608234 15123 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:58:23.615629 15123 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20250814 13:58:23.620374 15159 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:58:23.621392 15123 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250814 13:58:23.621683 15123 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "eff1057ae98849458f656363167a5337"
format_stamp: "Formatted at 2025-08-14 13:58:09 on dist-test-slave-q1tc"
I20250814 13:58:23.623462 15123 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:58:23.679901 15123 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:58:23.681305 15123 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:58:23.681720 15123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:58:23.684592 15123 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 13:58:23.690418 15166 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250814 13:58:23.697309 15123 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250814 13:58:23.697491 15123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.002s
I20250814 13:58:23.697752 15123 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250814 13:58:23.701962 15123 ts_tablet_manager.cc:610] Registered 1 tablets
I20250814 13:58:23.702193 15123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.003s sys 0.000s
I20250814 13:58:23.702517 15166 tablet_bootstrap.cc:492] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337: Bootstrap starting.
I20250814 13:58:23.778749 15166 log.cc:826] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337: Log is configured to *not* fsync() on all Append() calls
I20250814 13:58:23.888425 15166 tablet_bootstrap.cc:492] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 13:58:23.889460 15166 tablet_bootstrap.cc:492] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337: Bootstrap complete.
I20250814 13:58:23.890995 15123 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.129:42009
I20250814 13:58:23.891232 15166 ts_tablet_manager.cc:1397] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337: Time spent bootstrapping tablet: real 0.189s user 0.139s sys 0.043s
I20250814 13:58:23.891191 15273 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.129:42009 every 8 connection(s)
I20250814 13:58:23.893656 15123 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250814 13:58:23.903579 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 15123
I20250814 13:58:23.905130 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.132.130:42415
--local_ip_for_outbound_sockets=127.11.132.130
--tserver_master_addrs=127.11.132.190:39441
--webserver_port=44439
--webserver_interface=127.11.132.130
--builtin_ntp_servers=127.11.132.148:37521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250814 13:58:23.913137 15166 raft_consensus.cc:357] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } }
I20250814 13:58:23.916117 15166 raft_consensus.cc:738] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff1057ae98849458f656363167a5337, State: Initialized, Role: FOLLOWER
I20250814 13:58:23.916957 15166 consensus_queue.cc:260] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } }
I20250814 13:58:23.917605 15166 raft_consensus.cc:397] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 13:58:23.917944 15166 raft_consensus.cc:491] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 13:58:23.918354 15166 raft_consensus.cc:3058] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [term 1 FOLLOWER]: Advancing to term 2
I20250814 13:58:23.926148 15166 raft_consensus.cc:513] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } }
I20250814 13:58:23.927049 15166 leader_election.cc:304] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eff1057ae98849458f656363167a5337; no voters:
I20250814 13:58:23.937637 15166 leader_election.cc:290] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250814 13:58:23.938061 15279 raft_consensus.cc:2802] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [term 2 FOLLOWER]: Leader election won for term 2
I20250814 13:58:23.947908 15274 heartbeater.cc:344] Connected to a master server at 127.11.132.190:39441
I20250814 13:58:23.948470 15279 raft_consensus.cc:695] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [term 2 LEADER]: Becoming Leader. State: Replica: eff1057ae98849458f656363167a5337, State: Running, Role: LEADER
I20250814 13:58:23.948573 15274 heartbeater.cc:461] Registering TS with master...
I20250814 13:58:23.949740 15274 heartbeater.cc:507] Master 127.11.132.190:39441 requested a full tablet report, sending...
I20250814 13:58:23.948043 15166 ts_tablet_manager.cc:1428] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337: Time spent starting tablet: real 0.056s user 0.032s sys 0.020s
I20250814 13:58:23.951349 15279 consensus_queue.cc:237] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } }
I20250814 13:58:23.963048 15086 ts_manager.cc:194] Registered new tserver with Master: eff1057ae98849458f656363167a5337 (127.11.132.129:42009)
I20250814 13:58:23.966912 15086 catalog_manager.cc:5582] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 reported cstate change: term changed from 0 to 2, leader changed from <none> to eff1057ae98849458f656363167a5337 (127.11.132.129), VOTER eff1057ae98849458f656363167a5337 (127.11.132.129) added. New cstate: current_term: 2 leader_uuid: "eff1057ae98849458f656363167a5337" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } health_report { overall_health: HEALTHY } } }
W20250814 13:58:24.011561 15086 catalog_manager.cc:5260] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24df6cb14940420ea85c32a197b4fbc6 with cas_config_opid_index -1: no extra replica candidate found for tablet 24df6cb14940420ea85c32a197b4fbc6 (table TestTable [id=7f20d84458954c4db411207565551a70]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (1) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
I20250814 13:58:24.013687 15086 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.132.129:36961
I20250814 13:58:24.017640 15274 heartbeater.cc:499] Master 127.11.132.190:39441 was elected leader, sending a full tablet report...
W20250814 13:58:24.218600 15278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:58:24.219086 15278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:58:24.219528 15278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:58:24.249821 15278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:58:24.250639 15278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.132.130
I20250814 13:58:24.283749 15278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:37521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.132.130:42415
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.11.132.130
--webserver_port=44439
--tserver_master_addrs=127.11.132.190:39441
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.132.130
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:58:24.285115 15278 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:58:24.286679 15278 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:58:24.299013 15292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:58:24.302248 15293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 13:58:24.305514 15278 server_base.cc:1047] running on GCE node
W20250814 13:58:24.304186 15295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 13:58:25.395391 15278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:58:25.397682 15278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:58:25.399140 15278 hybrid_clock.cc:648] HybridClock initialized: now 1755179905399102 us; error 50 us; skew 500 ppm
I20250814 13:58:25.399881 15278 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:58:25.405686 15278 webserver.cc:480] Webserver started at http://127.11.132.130:44439/ using document root <none> and password file <none>
I20250814 13:58:25.406574 15278 fs_manager.cc:362] Metadata directory not provided
I20250814 13:58:25.406774 15278 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:58:25.414319 15278 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20250814 13:58:25.420231 15302 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:58:25.421200 15278 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250814 13:58:25.421496 15278 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "513c7d5f4e3f41e5b9f8918182af4955"
format_stamp: "Formatted at 2025-08-14 13:58:11 on dist-test-slave-q1tc"
I20250814 13:58:25.423313 15278 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:58:25.486351 15278 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:58:25.487665 15278 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:58:25.488063 15278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:58:25.490381 15278 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 13:58:25.495419 15309 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250814 13:58:25.502250 15278 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250814 13:58:25.502480 15278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.001s sys 0.001s
I20250814 13:58:25.502723 15278 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250814 13:58:25.507025 15278 ts_tablet_manager.cc:610] Registered 1 tablets
I20250814 13:58:25.507254 15278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.003s sys 0.000s
I20250814 13:58:25.507541 15309 tablet_bootstrap.cc:492] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955: Bootstrap starting.
I20250814 13:58:25.581655 15309 log.cc:826] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955: Log is configured to *not* fsync() on all Append() calls
I20250814 13:58:25.648243 15309 tablet_bootstrap.cc:492] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 13:58:25.649320 15309 tablet_bootstrap.cc:492] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955: Bootstrap complete.
I20250814 13:58:25.650679 15309 ts_tablet_manager.cc:1397] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955: Time spent bootstrapping tablet: real 0.143s user 0.093s sys 0.048s
I20250814 13:58:25.666214 15309 raft_consensus.cc:357] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } }
I20250814 13:58:25.668237 15309 raft_consensus.cc:738] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 513c7d5f4e3f41e5b9f8918182af4955, State: Initialized, Role: FOLLOWER
I20250814 13:58:25.668955 15309 consensus_queue.cc:260] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } }
I20250814 13:58:25.669509 15309 raft_consensus.cc:397] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 13:58:25.669734 15309 raft_consensus.cc:491] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 13:58:25.670022 15309 raft_consensus.cc:3058] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955 [term 1 FOLLOWER]: Advancing to term 2
I20250814 13:58:25.675204 15309 raft_consensus.cc:513] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } }
I20250814 13:58:25.675796 15309 leader_election.cc:304] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 513c7d5f4e3f41e5b9f8918182af4955; no voters:
I20250814 13:58:25.678637 15309 leader_election.cc:290] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250814 13:58:25.678959 15413 raft_consensus.cc:2802] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955 [term 2 FOLLOWER]: Leader election won for term 2
I20250814 13:58:25.682302 15413 raft_consensus.cc:695] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955 [term 2 LEADER]: Becoming Leader. State: Replica: 513c7d5f4e3f41e5b9f8918182af4955, State: Running, Role: LEADER
I20250814 13:58:25.683334 15413 consensus_queue.cc:237] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } }
I20250814 13:58:25.688099 15309 ts_tablet_manager.cc:1428] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955: Time spent starting tablet: real 0.037s user 0.030s sys 0.005s
I20250814 13:58:25.690384 15278 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.130:42415
I20250814 13:58:25.691180 15420 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.130:42415 every 8 connection(s)
I20250814 13:58:25.693481 15278 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250814 13:58:25.702730 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 15278
I20250814 13:58:25.704253 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.132.131:44563
--local_ip_for_outbound_sockets=127.11.132.131
--tserver_master_addrs=127.11.132.190:39441
--webserver_port=39163
--webserver_interface=127.11.132.131
--builtin_ntp_servers=127.11.132.148:37521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250814 13:58:25.714205 15422 heartbeater.cc:344] Connected to a master server at 127.11.132.190:39441
I20250814 13:58:25.714542 15422 heartbeater.cc:461] Registering TS with master...
I20250814 13:58:25.715404 15422 heartbeater.cc:507] Master 127.11.132.190:39441 requested a full tablet report, sending...
I20250814 13:58:25.718348 15086 ts_manager.cc:194] Registered new tserver with Master: 513c7d5f4e3f41e5b9f8918182af4955 (127.11.132.130:42415)
I20250814 13:58:25.719252 15086 catalog_manager.cc:5582] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "513c7d5f4e3f41e5b9f8918182af4955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } health_report { overall_health: HEALTHY } } }
I20250814 13:58:25.729759 15086 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.132.130:36999
I20250814 13:58:25.732803 15422 heartbeater.cc:499] Master 127.11.132.190:39441 was elected leader, sending a full tablet report...
W20250814 13:58:25.997035 15426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:58:25.997512 15426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:58:25.997977 15426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:58:26.032507 15426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:58:26.033349 15426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.132.131
I20250814 13:58:26.066155 15426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:37521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.132.131:44563
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.11.132.131
--webserver_port=39163
--tserver_master_addrs=127.11.132.190:39441
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.132.131
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:58:26.067392 15426 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:58:26.068825 15426 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:58:26.079331 15436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 13:58:26.360553 15229 consensus_queue.cc:237] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: NON_VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: true } }
I20250814 13:58:26.369629 15446 raft_consensus.cc:2953] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index -1 to 8, NON_VOTER 513c7d5f4e3f41e5b9f8918182af4955 (127.11.132.130) added. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: NON_VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: true } } }
W20250814 13:58:26.391489 15163 consensus_peers.cc:489] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 -> Peer 513c7d5f4e3f41e5b9f8918182af4955 (127.11.132.130:42415): Couldn't send request to peer 513c7d5f4e3f41e5b9f8918182af4955. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 24df6cb14940420ea85c32a197b4fbc6. This is attempt 1: this message will repeat every 5th retry.
I20250814 13:58:26.394490 15073 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24df6cb14940420ea85c32a197b4fbc6 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 8)
I20250814 13:58:26.401504 15086 catalog_manager.cc:5582] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 reported cstate change: config changed from index -1 to 8, NON_VOTER 513c7d5f4e3f41e5b9f8918182af4955 (127.11.132.130) added. New cstate: current_term: 2 leader_uuid: "eff1057ae98849458f656363167a5337" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: NON_VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250814 13:58:26.871230 15452 ts_tablet_manager.cc:927] T 24df6cb14940420ea85c32a197b4fbc6 P 513c7d5f4e3f41e5b9f8918182af4955: Initiating tablet copy from peer eff1057ae98849458f656363167a5337 (127.11.132.129:42009)
I20250814 13:58:26.881458 15452 tablet_copy_client.cc:323] T 24df6cb14940420ea85c32a197b4fbc6 P 513c7d5f4e3f41e5b9f8918182af4955: tablet copy: Beginning tablet copy session from remote peer at address 127.11.132.129:42009
I20250814 13:58:26.929649 15249 tablet_copy_service.cc:140] P eff1057ae98849458f656363167a5337: Received BeginTabletCopySession request for tablet 24df6cb14940420ea85c32a197b4fbc6 from peer 513c7d5f4e3f41e5b9f8918182af4955 ({username='slave'} at 127.11.132.130:38749)
I20250814 13:58:26.930159 15249 tablet_copy_service.cc:161] P eff1057ae98849458f656363167a5337: Beginning new tablet copy session on tablet 24df6cb14940420ea85c32a197b4fbc6 from peer 513c7d5f4e3f41e5b9f8918182af4955 at {username='slave'} at 127.11.132.130:38749: session id = 513c7d5f4e3f41e5b9f8918182af4955-24df6cb14940420ea85c32a197b4fbc6
I20250814 13:58:26.936977 15249 tablet_copy_source_session.cc:215] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337: Tablet Copy: opened 0 blocks and 1 log segments
I20250814 13:58:26.941848 15452 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24df6cb14940420ea85c32a197b4fbc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:58:26.995158 15452 tablet_copy_client.cc:806] T 24df6cb14940420ea85c32a197b4fbc6 P 513c7d5f4e3f41e5b9f8918182af4955: tablet copy: Starting download of 0 data blocks...
I20250814 13:58:26.996218 15452 tablet_copy_client.cc:670] T 24df6cb14940420ea85c32a197b4fbc6 P 513c7d5f4e3f41e5b9f8918182af4955: tablet copy: Starting download of 1 WAL segments...
I20250814 13:58:27.001681 15452 tablet_copy_client.cc:538] T 24df6cb14940420ea85c32a197b4fbc6 P 513c7d5f4e3f41e5b9f8918182af4955: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250814 13:58:27.012727 15452 tablet_bootstrap.cc:492] T 24df6cb14940420ea85c32a197b4fbc6 P 513c7d5f4e3f41e5b9f8918182af4955: Bootstrap starting.
I20250814 13:58:27.170328 15452 tablet_bootstrap.cc:492] T 24df6cb14940420ea85c32a197b4fbc6 P 513c7d5f4e3f41e5b9f8918182af4955: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 13:58:27.171710 15452 tablet_bootstrap.cc:492] T 24df6cb14940420ea85c32a197b4fbc6 P 513c7d5f4e3f41e5b9f8918182af4955: Bootstrap complete.
I20250814 13:58:27.172539 15452 ts_tablet_manager.cc:1397] T 24df6cb14940420ea85c32a197b4fbc6 P 513c7d5f4e3f41e5b9f8918182af4955: Time spent bootstrapping tablet: real 0.160s user 0.131s sys 0.025s
I20250814 13:58:27.175678 15452 raft_consensus.cc:357] T 24df6cb14940420ea85c32a197b4fbc6 P 513c7d5f4e3f41e5b9f8918182af4955 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: NON_VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: true } }
I20250814 13:58:27.176558 15452 raft_consensus.cc:738] T 24df6cb14940420ea85c32a197b4fbc6 P 513c7d5f4e3f41e5b9f8918182af4955 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 513c7d5f4e3f41e5b9f8918182af4955, State: Initialized, Role: LEARNER
I20250814 13:58:27.177349 15452 consensus_queue.cc:260] T 24df6cb14940420ea85c32a197b4fbc6 P 513c7d5f4e3f41e5b9f8918182af4955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: NON_VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: true } }
I20250814 13:58:27.187464 15452 ts_tablet_manager.cc:1428] T 24df6cb14940420ea85c32a197b4fbc6 P 513c7d5f4e3f41e5b9f8918182af4955: Time spent starting tablet: real 0.015s user 0.008s sys 0.004s
I20250814 13:58:27.202750 15249 tablet_copy_service.cc:342] P eff1057ae98849458f656363167a5337: Request end of tablet copy session 513c7d5f4e3f41e5b9f8918182af4955-24df6cb14940420ea85c32a197b4fbc6 received from {username='slave'} at 127.11.132.130:38749
I20250814 13:58:27.203193 15249 tablet_copy_service.cc:434] P eff1057ae98849458f656363167a5337: ending tablet copy session 513c7d5f4e3f41e5b9f8918182af4955-24df6cb14940420ea85c32a197b4fbc6 on tablet 24df6cb14940420ea85c32a197b4fbc6 with peer 513c7d5f4e3f41e5b9f8918182af4955
I20250814 13:58:27.321532 15372 raft_consensus.cc:1215] T 24df6cb14940420ea85c32a197b4fbc6 P 513c7d5f4e3f41e5b9f8918182af4955 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
W20250814 13:58:26.081094 15437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:58:27.461887 15439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:58:27.464958 15426 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.385s user 0.415s sys 0.893s
W20250814 13:58:27.465282 15426 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.385s user 0.419s sys 0.893s
W20250814 13:58:27.470760 15438 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1379 milliseconds
I20250814 13:58:27.470816 15426 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 13:58:27.472148 15426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:58:27.476727 15426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:58:27.478096 15426 hybrid_clock.cc:648] HybridClock initialized: now 1755179907478045 us; error 46 us; skew 500 ppm
I20250814 13:58:27.478843 15426 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:58:27.485333 15426 webserver.cc:480] Webserver started at http://127.11.132.131:39163/ using document root <none> and password file <none>
I20250814 13:58:27.486187 15426 fs_manager.cc:362] Metadata directory not provided
I20250814 13:58:27.486395 15426 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:58:27.493865 15426 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.003s
I20250814 13:58:27.498227 15463 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:58:27.499289 15426 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20250814 13:58:27.499573 15426 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ac4ac7370c8c4b0c967fa990af77a067"
format_stamp: "Formatted at 2025-08-14 13:58:13 on dist-test-slave-q1tc"
I20250814 13:58:27.501487 15426 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:58:27.574314 15426 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:58:27.575635 15426 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:58:27.576032 15426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:58:27.578294 15426 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 13:58:27.583284 15470 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250814 13:58:27.590284 15426 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250814 13:58:27.590518 15426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.001s sys 0.001s
I20250814 13:58:27.590778 15426 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250814 13:58:27.594961 15426 ts_tablet_manager.cc:610] Registered 1 tablets
I20250814 13:58:27.595142 15426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.003s sys 0.000s
I20250814 13:58:27.595438 15470 tablet_bootstrap.cc:492] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067: Bootstrap starting.
I20250814 13:58:27.646464 15470 log.cc:826] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067: Log is configured to *not* fsync() on all Append() calls
I20250814 13:58:27.718833 15470 tablet_bootstrap.cc:492] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 13:58:27.719736 15470 tablet_bootstrap.cc:492] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067: Bootstrap complete.
I20250814 13:58:27.721436 15470 ts_tablet_manager.cc:1397] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067: Time spent bootstrapping tablet: real 0.126s user 0.089s sys 0.035s
I20250814 13:58:27.739423 15446 raft_consensus.cc:1062] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337: attempting to promote NON_VOTER 513c7d5f4e3f41e5b9f8918182af4955 to VOTER
I20250814 13:58:27.741122 15446 consensus_queue.cc:237] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } }
I20250814 13:58:27.742437 15470 raft_consensus.cc:357] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } }
I20250814 13:58:27.746126 15470 raft_consensus.cc:738] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac4ac7370c8c4b0c967fa990af77a067, State: Initialized, Role: FOLLOWER
I20250814 13:58:27.747406 15470 consensus_queue.cc:260] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } }
I20250814 13:58:27.748231 15470 raft_consensus.cc:397] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 13:58:27.748649 15470 raft_consensus.cc:491] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 13:58:27.749148 15470 raft_consensus.cc:3058] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [term 1 FOLLOWER]: Advancing to term 2
I20250814 13:58:27.751911 15372 raft_consensus.cc:1273] T 24df6cb14940420ea85c32a197b4fbc6 P 513c7d5f4e3f41e5b9f8918182af4955 [term 2 LEARNER]: Refusing update from remote peer eff1057ae98849458f656363167a5337: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20250814 13:58:27.753415 15446 consensus_queue.cc:1035] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [LEADER]: Connected to new peer: Peer: permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250814 13:58:27.757236 15470 raft_consensus.cc:513] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } }
I20250814 13:58:27.758033 15470 leader_election.cc:304] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac4ac7370c8c4b0c967fa990af77a067; no voters:
I20250814 13:58:27.760296 15470 leader_election.cc:290] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250814 13:58:27.760565 15557 raft_consensus.cc:2802] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [term 2 FOLLOWER]: Leader election won for term 2
I20250814 13:58:27.761492 15446 raft_consensus.cc:2953] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 513c7d5f4e3f41e5b9f8918182af4955 (127.11.132.130) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } } }
I20250814 13:58:27.764040 15470 ts_tablet_manager.cc:1428] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067: Time spent starting tablet: real 0.042s user 0.029s sys 0.011s
I20250814 13:58:27.764276 15372 raft_consensus.cc:2953] T 24df6cb14940420ea85c32a197b4fbc6 P 513c7d5f4e3f41e5b9f8918182af4955 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 513c7d5f4e3f41e5b9f8918182af4955 (127.11.132.130) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } } }
I20250814 13:58:27.765652 15557 raft_consensus.cc:695] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [term 2 LEADER]: Becoming Leader. State: Replica: ac4ac7370c8c4b0c967fa990af77a067, State: Running, Role: LEADER
I20250814 13:58:27.766481 15557 consensus_queue.cc:237] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } }
I20250814 13:58:27.772178 15086 catalog_manager.cc:5582] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 reported cstate change: config changed from index 8 to 9, 513c7d5f4e3f41e5b9f8918182af4955 (127.11.132.130) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "eff1057ae98849458f656363167a5337" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250814 13:58:27.806080 15426 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.131:44563
I20250814 13:58:27.806599 15588 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.131:44563 every 8 connection(s)
I20250814 13:58:27.808230 15426 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250814 13:58:27.817999 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 15426
I20250814 13:58:27.824102 15589 heartbeater.cc:344] Connected to a master server at 127.11.132.190:39441
I20250814 13:58:27.824438 15589 heartbeater.cc:461] Registering TS with master...
I20250814 13:58:27.825582 15589 heartbeater.cc:507] Master 127.11.132.190:39441 requested a full tablet report, sending...
I20250814 13:58:27.828805 15086 ts_manager.cc:194] Registered new tserver with Master: ac4ac7370c8c4b0c967fa990af77a067 (127.11.132.131:44563)
I20250814 13:58:27.829869 15086 catalog_manager.cc:5582] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 reported cstate change: term changed from 0 to 2, leader changed from <none> to ac4ac7370c8c4b0c967fa990af77a067 (127.11.132.131), VOTER ac4ac7370c8c4b0c967fa990af77a067 (127.11.132.131) added. New cstate: current_term: 2 leader_uuid: "ac4ac7370c8c4b0c967fa990af77a067" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } health_report { overall_health: HEALTHY } } }
I20250814 13:58:27.833640 11794 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250814 13:58:27.837177 15229 consensus_queue.cc:237] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } } peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: NON_VOTER last_known_addr { host: "127.11.132.131" port: 44563 } attrs { promote: true } }
I20250814 13:58:27.838723 11794 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250814 13:58:27.838850 15086 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.132.131:50875
I20250814 13:58:27.842598 15589 heartbeater.cc:499] Master 127.11.132.190:39441 was elected leader, sending a full tablet report...
W20250814 13:58:27.842613 11794 ts_itest-base.cc:209] found only 2 out of 3 replicas of tablet 24df6cb14940420ea85c32a197b4fbc6: tablet_id: "24df6cb14940420ea85c32a197b4fbc6" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER } interned_replicas { ts_info_idx: 1 role: FOLLOWER }
I20250814 13:58:27.847239 15372 raft_consensus.cc:1273] T 24df6cb14940420ea85c32a197b4fbc6 P 513c7d5f4e3f41e5b9f8918182af4955 [term 2 FOLLOWER]: Refusing update from remote peer eff1057ae98849458f656363167a5337: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250814 13:58:27.849102 15446 consensus_queue.cc:1035] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [LEADER]: Connected to new peer: Peer: permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250814 13:58:27.855875 15458 raft_consensus.cc:2953] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, NON_VOTER ac4ac7370c8c4b0c967fa990af77a067 (127.11.132.131) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } } peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: NON_VOTER last_known_addr { host: "127.11.132.131" port: 44563 } attrs { promote: true } } }
I20250814 13:58:27.860217 15372 raft_consensus.cc:2953] T 24df6cb14940420ea85c32a197b4fbc6 P 513c7d5f4e3f41e5b9f8918182af4955 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, NON_VOTER ac4ac7370c8c4b0c967fa990af77a067 (127.11.132.131) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } } peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: NON_VOTER last_known_addr { host: "127.11.132.131" port: 44563 } attrs { promote: true } } }
I20250814 13:58:27.865379 15073 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24df6cb14940420ea85c32a197b4fbc6 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 2)
W20250814 13:58:27.868497 15163 consensus_peers.cc:489] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 -> Peer ac4ac7370c8c4b0c967fa990af77a067 (127.11.132.131:44563): Couldn't send request to peer ac4ac7370c8c4b0c967fa990af77a067. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 24df6cb14940420ea85c32a197b4fbc6. This is attempt 1: this message will repeat every 5th retry.
I20250814 13:58:27.868577 15533 consensus_queue.cc:237] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } } peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: NON_VOTER last_known_addr { host: "127.11.132.129" port: 42009 } attrs { promote: true } }
I20250814 13:58:27.870632 15085 catalog_manager.cc:5582] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 reported cstate change: config changed from index 9 to 10, NON_VOTER ac4ac7370c8c4b0c967fa990af77a067 (127.11.132.131) added. New cstate: current_term: 2 leader_uuid: "eff1057ae98849458f656363167a5337" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: NON_VOTER last_known_addr { host: "127.11.132.131" port: 44563 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250814 13:58:27.872206 15560 raft_consensus.cc:2953] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index -1 to 8, NON_VOTER eff1057ae98849458f656363167a5337 (127.11.132.129) added. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } } peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: NON_VOTER last_known_addr { host: "127.11.132.129" port: 42009 } attrs { promote: true } } }
I20250814 13:58:27.878724 15073 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet e5e55ced4bfc4fb6b6991fc92964cbaa with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250814 13:58:27.881867 15084 catalog_manager.cc:5582] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 reported cstate change: config changed from index -1 to 8, NON_VOTER eff1057ae98849458f656363167a5337 (127.11.132.129) added. New cstate: current_term: 2 leader_uuid: "ac4ac7370c8c4b0c967fa990af77a067" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: NON_VOTER last_known_addr { host: "127.11.132.129" port: 42009 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250814 13:58:27.887856 15467 consensus_peers.cc:489] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 -> Peer eff1057ae98849458f656363167a5337 (127.11.132.129:42009): Couldn't send request to peer eff1057ae98849458f656363167a5337. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5e55ced4bfc4fb6b6991fc92964cbaa. This is attempt 1: this message will repeat every 5th retry.
I20250814 13:58:27.892472 15533 consensus_queue.cc:237] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } } peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: NON_VOTER last_known_addr { host: "127.11.132.129" port: 42009 } attrs { promote: true } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: NON_VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: true } }
I20250814 13:58:27.895382 15557 raft_consensus.cc:2953] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 513c7d5f4e3f41e5b9f8918182af4955 (127.11.132.130) added. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } } peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: NON_VOTER last_known_addr { host: "127.11.132.129" port: 42009 } attrs { promote: true } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: NON_VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: true } } }
W20250814 13:58:27.897502 15467 consensus_peers.cc:489] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 -> Peer eff1057ae98849458f656363167a5337 (127.11.132.129:42009): Couldn't send request to peer eff1057ae98849458f656363167a5337. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5e55ced4bfc4fb6b6991fc92964cbaa. This is attempt 1: this message will repeat every 5th retry.
I20250814 13:58:27.902226 15073 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet e5e55ced4bfc4fb6b6991fc92964cbaa with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250814 13:58:27.906186 15084 catalog_manager.cc:5582] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 reported cstate change: config changed from index 8 to 9, NON_VOTER 513c7d5f4e3f41e5b9f8918182af4955 (127.11.132.130) added. New cstate: current_term: 2 leader_uuid: "ac4ac7370c8c4b0c967fa990af77a067" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: NON_VOTER last_known_addr { host: "127.11.132.129" port: 42009 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: NON_VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250814 13:58:27.911720 15467 consensus_peers.cc:489] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 -> Peer 513c7d5f4e3f41e5b9f8918182af4955 (127.11.132.130:42415): Couldn't send request to peer 513c7d5f4e3f41e5b9f8918182af4955. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5e55ced4bfc4fb6b6991fc92964cbaa. This is attempt 1: this message will repeat every 5th retry.
I20250814 13:58:28.275771 15601 ts_tablet_manager.cc:927] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337: Initiating tablet copy from peer ac4ac7370c8c4b0c967fa990af77a067 (127.11.132.131:44563)
I20250814 13:58:28.277840 15601 tablet_copy_client.cc:323] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337: tablet copy: Beginning tablet copy session from remote peer at address 127.11.132.131:44563
I20250814 13:58:28.279263 15554 tablet_copy_service.cc:140] P ac4ac7370c8c4b0c967fa990af77a067: Received BeginTabletCopySession request for tablet e5e55ced4bfc4fb6b6991fc92964cbaa from peer eff1057ae98849458f656363167a5337 ({username='slave'} at 127.11.132.129:59321)
I20250814 13:58:28.279651 15554 tablet_copy_service.cc:161] P ac4ac7370c8c4b0c967fa990af77a067: Beginning new tablet copy session on tablet e5e55ced4bfc4fb6b6991fc92964cbaa from peer eff1057ae98849458f656363167a5337 at {username='slave'} at 127.11.132.129:59321: session id = eff1057ae98849458f656363167a5337-e5e55ced4bfc4fb6b6991fc92964cbaa
I20250814 13:58:28.283246 15603 ts_tablet_manager.cc:927] T 24df6cb14940420ea85c32a197b4fbc6 P ac4ac7370c8c4b0c967fa990af77a067: Initiating tablet copy from peer eff1057ae98849458f656363167a5337 (127.11.132.129:42009)
I20250814 13:58:28.285195 15603 tablet_copy_client.cc:323] T 24df6cb14940420ea85c32a197b4fbc6 P ac4ac7370c8c4b0c967fa990af77a067: tablet copy: Beginning tablet copy session from remote peer at address 127.11.132.129:42009
I20250814 13:58:28.285323 15554 tablet_copy_source_session.cc:215] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067: Tablet Copy: opened 0 blocks and 1 log segments
I20250814 13:58:28.286546 15249 tablet_copy_service.cc:140] P eff1057ae98849458f656363167a5337: Received BeginTabletCopySession request for tablet 24df6cb14940420ea85c32a197b4fbc6 from peer ac4ac7370c8c4b0c967fa990af77a067 ({username='slave'} at 127.11.132.131:52407)
I20250814 13:58:28.286900 15249 tablet_copy_service.cc:161] P eff1057ae98849458f656363167a5337: Beginning new tablet copy session on tablet 24df6cb14940420ea85c32a197b4fbc6 from peer ac4ac7370c8c4b0c967fa990af77a067 at {username='slave'} at 127.11.132.131:52407: session id = ac4ac7370c8c4b0c967fa990af77a067-24df6cb14940420ea85c32a197b4fbc6
I20250814 13:58:28.288434 15601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5e55ced4bfc4fb6b6991fc92964cbaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:58:28.290956 15249 tablet_copy_source_session.cc:215] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337: Tablet Copy: opened 0 blocks and 1 log segments
I20250814 13:58:28.293339 15603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24df6cb14940420ea85c32a197b4fbc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:58:28.297871 15601 tablet_copy_client.cc:806] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337: tablet copy: Starting download of 0 data blocks...
I20250814 13:58:28.298311 15601 tablet_copy_client.cc:670] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337: tablet copy: Starting download of 1 WAL segments...
I20250814 13:58:28.301891 15601 tablet_copy_client.cc:538] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250814 13:58:28.305850 15603 tablet_copy_client.cc:806] T 24df6cb14940420ea85c32a197b4fbc6 P ac4ac7370c8c4b0c967fa990af77a067: tablet copy: Starting download of 0 data blocks...
I20250814 13:58:28.306289 15603 tablet_copy_client.cc:670] T 24df6cb14940420ea85c32a197b4fbc6 P ac4ac7370c8c4b0c967fa990af77a067: tablet copy: Starting download of 1 WAL segments...
I20250814 13:58:28.307125 15601 tablet_bootstrap.cc:492] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337: Bootstrap starting.
I20250814 13:58:28.309943 15603 tablet_copy_client.cc:538] T 24df6cb14940420ea85c32a197b4fbc6 P ac4ac7370c8c4b0c967fa990af77a067: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250814 13:58:28.315045 15603 tablet_bootstrap.cc:492] T 24df6cb14940420ea85c32a197b4fbc6 P ac4ac7370c8c4b0c967fa990af77a067: Bootstrap starting.
I20250814 13:58:28.379606 15601 tablet_bootstrap.cc:492] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 13:58:28.380204 15601 tablet_bootstrap.cc:492] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337: Bootstrap complete.
I20250814 13:58:28.380647 15601 ts_tablet_manager.cc:1397] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337: Time spent bootstrapping tablet: real 0.074s user 0.061s sys 0.009s
I20250814 13:58:28.382191 15601 raft_consensus.cc:357] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } } peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: NON_VOTER last_known_addr { host: "127.11.132.129" port: 42009 } attrs { promote: true } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: NON_VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: true } }
I20250814 13:58:28.382599 15601 raft_consensus.cc:738] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: eff1057ae98849458f656363167a5337, State: Initialized, Role: LEARNER
I20250814 13:58:28.382997 15601 consensus_queue.cc:260] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } } peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: NON_VOTER last_known_addr { host: "127.11.132.129" port: 42009 } attrs { promote: true } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: NON_VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: true } }
I20250814 13:58:28.384477 15601 ts_tablet_manager.cc:1428] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250814 13:58:28.386063 15554 tablet_copy_service.cc:342] P ac4ac7370c8c4b0c967fa990af77a067: Request end of tablet copy session eff1057ae98849458f656363167a5337-e5e55ced4bfc4fb6b6991fc92964cbaa received from {username='slave'} at 127.11.132.129:59321
I20250814 13:58:28.386467 15554 tablet_copy_service.cc:434] P ac4ac7370c8c4b0c967fa990af77a067: ending tablet copy session eff1057ae98849458f656363167a5337-e5e55ced4bfc4fb6b6991fc92964cbaa on tablet e5e55ced4bfc4fb6b6991fc92964cbaa with peer eff1057ae98849458f656363167a5337
I20250814 13:58:28.386950 15603 tablet_bootstrap.cc:492] T 24df6cb14940420ea85c32a197b4fbc6 P ac4ac7370c8c4b0c967fa990af77a067: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 13:58:28.387661 15603 tablet_bootstrap.cc:492] T 24df6cb14940420ea85c32a197b4fbc6 P ac4ac7370c8c4b0c967fa990af77a067: Bootstrap complete.
I20250814 13:58:28.388175 15603 ts_tablet_manager.cc:1397] T 24df6cb14940420ea85c32a197b4fbc6 P ac4ac7370c8c4b0c967fa990af77a067: Time spent bootstrapping tablet: real 0.073s user 0.057s sys 0.016s
I20250814 13:58:28.389912 15603 raft_consensus.cc:357] T 24df6cb14940420ea85c32a197b4fbc6 P ac4ac7370c8c4b0c967fa990af77a067 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } } peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: NON_VOTER last_known_addr { host: "127.11.132.131" port: 44563 } attrs { promote: true } }
I20250814 13:58:28.390323 15603 raft_consensus.cc:738] T 24df6cb14940420ea85c32a197b4fbc6 P ac4ac7370c8c4b0c967fa990af77a067 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: ac4ac7370c8c4b0c967fa990af77a067, State: Initialized, Role: LEARNER
I20250814 13:58:28.390771 15603 consensus_queue.cc:260] T 24df6cb14940420ea85c32a197b4fbc6 P ac4ac7370c8c4b0c967fa990af77a067 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } } peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: NON_VOTER last_known_addr { host: "127.11.132.131" port: 44563 } attrs { promote: true } }
I20250814 13:58:28.392149 15603 ts_tablet_manager.cc:1428] T 24df6cb14940420ea85c32a197b4fbc6 P ac4ac7370c8c4b0c967fa990af77a067: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250814 13:58:28.393417 15249 tablet_copy_service.cc:342] P eff1057ae98849458f656363167a5337: Request end of tablet copy session ac4ac7370c8c4b0c967fa990af77a067-24df6cb14940420ea85c32a197b4fbc6 received from {username='slave'} at 127.11.132.131:52407
I20250814 13:58:28.393760 15249 tablet_copy_service.cc:434] P eff1057ae98849458f656363167a5337: ending tablet copy session ac4ac7370c8c4b0c967fa990af77a067-24df6cb14940420ea85c32a197b4fbc6 on tablet 24df6cb14940420ea85c32a197b4fbc6 with peer ac4ac7370c8c4b0c967fa990af77a067
I20250814 13:58:28.517316 15607 ts_tablet_manager.cc:927] T e5e55ced4bfc4fb6b6991fc92964cbaa P 513c7d5f4e3f41e5b9f8918182af4955: Initiating tablet copy from peer ac4ac7370c8c4b0c967fa990af77a067 (127.11.132.131:44563)
I20250814 13:58:28.518308 15607 tablet_copy_client.cc:323] T e5e55ced4bfc4fb6b6991fc92964cbaa P 513c7d5f4e3f41e5b9f8918182af4955: tablet copy: Beginning tablet copy session from remote peer at address 127.11.132.131:44563
I20250814 13:58:28.526305 15554 tablet_copy_service.cc:140] P ac4ac7370c8c4b0c967fa990af77a067: Received BeginTabletCopySession request for tablet e5e55ced4bfc4fb6b6991fc92964cbaa from peer 513c7d5f4e3f41e5b9f8918182af4955 ({username='slave'} at 127.11.132.130:41073)
I20250814 13:58:28.526638 15554 tablet_copy_service.cc:161] P ac4ac7370c8c4b0c967fa990af77a067: Beginning new tablet copy session on tablet e5e55ced4bfc4fb6b6991fc92964cbaa from peer 513c7d5f4e3f41e5b9f8918182af4955 at {username='slave'} at 127.11.132.130:41073: session id = 513c7d5f4e3f41e5b9f8918182af4955-e5e55ced4bfc4fb6b6991fc92964cbaa
I20250814 13:58:28.530433 15554 tablet_copy_source_session.cc:215] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067: Tablet Copy: opened 0 blocks and 1 log segments
I20250814 13:58:28.532480 15607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5e55ced4bfc4fb6b6991fc92964cbaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:58:28.541142 15607 tablet_copy_client.cc:806] T e5e55ced4bfc4fb6b6991fc92964cbaa P 513c7d5f4e3f41e5b9f8918182af4955: tablet copy: Starting download of 0 data blocks...
I20250814 13:58:28.541579 15607 tablet_copy_client.cc:670] T e5e55ced4bfc4fb6b6991fc92964cbaa P 513c7d5f4e3f41e5b9f8918182af4955: tablet copy: Starting download of 1 WAL segments...
I20250814 13:58:28.544224 15607 tablet_copy_client.cc:538] T e5e55ced4bfc4fb6b6991fc92964cbaa P 513c7d5f4e3f41e5b9f8918182af4955: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250814 13:58:28.549154 15607 tablet_bootstrap.cc:492] T e5e55ced4bfc4fb6b6991fc92964cbaa P 513c7d5f4e3f41e5b9f8918182af4955: Bootstrap starting.
I20250814 13:58:28.614012 15607 tablet_bootstrap.cc:492] T e5e55ced4bfc4fb6b6991fc92964cbaa P 513c7d5f4e3f41e5b9f8918182af4955: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 13:58:28.614610 15607 tablet_bootstrap.cc:492] T e5e55ced4bfc4fb6b6991fc92964cbaa P 513c7d5f4e3f41e5b9f8918182af4955: Bootstrap complete.
I20250814 13:58:28.615021 15607 ts_tablet_manager.cc:1397] T e5e55ced4bfc4fb6b6991fc92964cbaa P 513c7d5f4e3f41e5b9f8918182af4955: Time spent bootstrapping tablet: real 0.066s user 0.056s sys 0.012s
I20250814 13:58:28.616456 15607 raft_consensus.cc:357] T e5e55ced4bfc4fb6b6991fc92964cbaa P 513c7d5f4e3f41e5b9f8918182af4955 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } } peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: NON_VOTER last_known_addr { host: "127.11.132.129" port: 42009 } attrs { promote: true } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: NON_VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: true } }
I20250814 13:58:28.616917 15607 raft_consensus.cc:738] T e5e55ced4bfc4fb6b6991fc92964cbaa P 513c7d5f4e3f41e5b9f8918182af4955 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 513c7d5f4e3f41e5b9f8918182af4955, State: Initialized, Role: LEARNER
I20250814 13:58:28.617354 15607 consensus_queue.cc:260] T e5e55ced4bfc4fb6b6991fc92964cbaa P 513c7d5f4e3f41e5b9f8918182af4955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } } peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: NON_VOTER last_known_addr { host: "127.11.132.129" port: 42009 } attrs { promote: true } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: NON_VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: true } }
I20250814 13:58:28.619855 15607 ts_tablet_manager.cc:1428] T e5e55ced4bfc4fb6b6991fc92964cbaa P 513c7d5f4e3f41e5b9f8918182af4955: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250814 13:58:28.621286 15554 tablet_copy_service.cc:342] P ac4ac7370c8c4b0c967fa990af77a067: Request end of tablet copy session 513c7d5f4e3f41e5b9f8918182af4955-e5e55ced4bfc4fb6b6991fc92964cbaa received from {username='slave'} at 127.11.132.130:41073
I20250814 13:58:28.621568 15554 tablet_copy_service.cc:434] P ac4ac7370c8c4b0c967fa990af77a067: ending tablet copy session 513c7d5f4e3f41e5b9f8918182af4955-e5e55ced4bfc4fb6b6991fc92964cbaa on tablet e5e55ced4bfc4fb6b6991fc92964cbaa with peer 513c7d5f4e3f41e5b9f8918182af4955
I20250814 13:58:28.677553 15072 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 24df6cb14940420ea85c32a197b4fbc6 with cas_config_opid_index 8: aborting the task: latest config opid_index 10; task opid_index 8
I20250814 13:58:28.727819 15229 raft_consensus.cc:1215] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20250814 13:58:28.847370 11794 ts_itest-base.cc:246] Waiting for 1 tablets on tserver eff1057ae98849458f656363167a5337 to finish bootstrapping
I20250814 13:58:28.860746 11794 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 513c7d5f4e3f41e5b9f8918182af4955 to finish bootstrapping
I20250814 13:58:28.872704 11794 ts_itest-base.cc:246] Waiting for 1 tablets on tserver ac4ac7370c8c4b0c967fa990af77a067 to finish bootstrapping
I20250814 13:58:28.882328 15533 raft_consensus.cc:1215] T 24df6cb14940420ea85c32a197b4fbc6 P ac4ac7370c8c4b0c967fa990af77a067 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20250814 13:58:28.972911 15372 raft_consensus.cc:1215] T e5e55ced4bfc4fb6b6991fc92964cbaa P 513c7d5f4e3f41e5b9f8918182af4955 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20250814 13:58:29.144739 15351 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250814 13:58:29.148366 15513 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250814 13:58:29.151675 15208 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250814 13:58:29.233690 15614 raft_consensus.cc:1062] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067: attempting to promote NON_VOTER eff1057ae98849458f656363167a5337 to VOTER
I20250814 13:58:29.235589 15614 consensus_queue.cc:237] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } } peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } attrs { promote: false } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: NON_VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: true } }
I20250814 13:58:29.251479 15372 raft_consensus.cc:1273] T e5e55ced4bfc4fb6b6991fc92964cbaa P 513c7d5f4e3f41e5b9f8918182af4955 [term 2 LEARNER]: Refusing update from remote peer ac4ac7370c8c4b0c967fa990af77a067: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250814 13:58:29.252364 15229 raft_consensus.cc:1273] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337 [term 2 LEARNER]: Refusing update from remote peer ac4ac7370c8c4b0c967fa990af77a067: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250814 13:58:29.253777 15557 consensus_queue.cc:1035] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250814 13:58:29.254981 15557 consensus_queue.cc:1035] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [LEADER]: Connected to new peer: Peer: permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: NON_VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250814 13:58:29.271749 15614 raft_consensus.cc:2953] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, eff1057ae98849458f656363167a5337 (127.11.132.129) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } } peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } attrs { promote: false } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: NON_VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: true } } }
I20250814 13:58:29.275807 15228 raft_consensus.cc:2953] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, eff1057ae98849458f656363167a5337 (127.11.132.129) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } } peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } attrs { promote: false } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: NON_VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: true } } }
I20250814 13:58:29.284766 15557 raft_consensus.cc:1062] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067: attempting to promote NON_VOTER 513c7d5f4e3f41e5b9f8918182af4955 to VOTER
I20250814 13:58:29.284703 15086 catalog_manager.cc:5582] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 reported cstate change: config changed from index 9 to 10, eff1057ae98849458f656363167a5337 (127.11.132.129) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "ac4ac7370c8c4b0c967fa990af77a067" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: NON_VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250814 13:58:29.286545 15557 consensus_queue.cc:237] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } } peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } attrs { promote: false } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } }
I20250814 13:58:29.290879 15372 raft_consensus.cc:1273] T e5e55ced4bfc4fb6b6991fc92964cbaa P 513c7d5f4e3f41e5b9f8918182af4955 [term 2 LEARNER]: Refusing update from remote peer ac4ac7370c8c4b0c967fa990af77a067: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250814 13:58:29.292234 15613 consensus_queue.cc:1035] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [LEADER]: Connected to new peer: Peer: permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20250814 13:58:29.294793 15372 raft_consensus.cc:2953] T e5e55ced4bfc4fb6b6991fc92964cbaa P 513c7d5f4e3f41e5b9f8918182af4955 [term 2 LEARNER]: Committing config change with OpId 2.10: config changed from index 9 to 10, eff1057ae98849458f656363167a5337 (127.11.132.129) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } } peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } attrs { promote: false } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: NON_VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: true } } }
I20250814 13:58:29.307307 15228 raft_consensus.cc:1273] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337 [term 2 FOLLOWER]: Refusing update from remote peer ac4ac7370c8c4b0c967fa990af77a067: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250814 13:58:29.308409 15614 consensus_queue.cc:1035] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250814 13:58:29.313452 15557 raft_consensus.cc:2953] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 513c7d5f4e3f41e5b9f8918182af4955 (127.11.132.130) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } } peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } attrs { promote: false } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } } }
I20250814 13:58:29.336802 15228 raft_consensus.cc:2953] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 513c7d5f4e3f41e5b9f8918182af4955 (127.11.132.130) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } } peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } attrs { promote: false } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } } }
I20250814 13:58:29.338536 15372 raft_consensus.cc:2953] T e5e55ced4bfc4fb6b6991fc92964cbaa P 513c7d5f4e3f41e5b9f8918182af4955 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 513c7d5f4e3f41e5b9f8918182af4955 (127.11.132.130) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } } peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } attrs { promote: false } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } } }
I20250814 13:58:29.356086 15085 catalog_manager.cc:5582] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 reported cstate change: config changed from index 10 to 11, 513c7d5f4e3f41e5b9f8918182af4955 (127.11.132.130) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "ac4ac7370c8c4b0c967fa990af77a067" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
2ff0f6a4351645cd82f1dc14d3ef065d | 127.11.132.190:39441 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.11.132.148:37521 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
513c7d5f4e3f41e5b9f8918182af4955 | 127.11.132.130:42415 | HEALTHY | <none> | 1 | 0
ac4ac7370c8c4b0c967fa990af77a067 | 127.11.132.131:44563 | HEALTHY | <none> | 1 | 0
eff1057ae98849458f656363167a5337 | 127.11.132.129:42009 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.11.132.129 | experimental | 127.11.132.129:42009
local_ip_for_outbound_sockets | 127.11.132.130 | experimental | 127.11.132.130:42415
local_ip_for_outbound_sockets | 127.11.132.131 | experimental | 127.11.132.131:44563
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.11.132.129:42009
server_dump_info_path | /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.11.132.130:42415
server_dump_info_path | /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.11.132.131:44563
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.11.132.148:37521 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable1 | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable2 | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 3
Tablets | 3
Replicas | 7
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250814 13:58:29.411855 11794 log_verifier.cc:126] Checking tablet 1db867af4d4b43dca1cf4ddc26d3321b
I20250814 13:58:29.440088 11794 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 1db867af4d4b43dca1cf4ddc26d3321b
I20250814 13:58:29.440326 11794 log_verifier.cc:126] Checking tablet 24df6cb14940420ea85c32a197b4fbc6
I20250814 13:58:29.498162 15619 raft_consensus.cc:1062] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337: attempting to promote NON_VOTER ac4ac7370c8c4b0c967fa990af77a067 to VOTER
I20250814 13:58:29.499341 15619 consensus_queue.cc:237] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } } peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } attrs { promote: false } }
I20250814 13:58:29.503768 15533 raft_consensus.cc:1273] T 24df6cb14940420ea85c32a197b4fbc6 P ac4ac7370c8c4b0c967fa990af77a067 [term 2 LEARNER]: Refusing update from remote peer eff1057ae98849458f656363167a5337: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250814 13:58:29.504884 15652 consensus_queue.cc:1035] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250814 13:58:29.505033 15372 raft_consensus.cc:1273] T 24df6cb14940420ea85c32a197b4fbc6 P 513c7d5f4e3f41e5b9f8918182af4955 [term 2 FOLLOWER]: Refusing update from remote peer eff1057ae98849458f656363167a5337: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250814 13:58:29.506356 15619 consensus_queue.cc:1035] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [LEADER]: Connected to new peer: Peer: permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250814 13:58:29.511747 15652 raft_consensus.cc:2953] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, ac4ac7370c8c4b0c967fa990af77a067 (127.11.132.131) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } } peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } attrs { promote: false } } }
I20250814 13:58:29.513222 15533 raft_consensus.cc:2953] T 24df6cb14940420ea85c32a197b4fbc6 P ac4ac7370c8c4b0c967fa990af77a067 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, ac4ac7370c8c4b0c967fa990af77a067 (127.11.132.131) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } } peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } attrs { promote: false } } }
I20250814 13:58:29.515309 15371 raft_consensus.cc:2953] T 24df6cb14940420ea85c32a197b4fbc6 P 513c7d5f4e3f41e5b9f8918182af4955 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, ac4ac7370c8c4b0c967fa990af77a067 (127.11.132.131) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } } peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } attrs { promote: false } } }
I20250814 13:58:29.523706 15084 catalog_manager.cc:5582] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 reported cstate change: config changed from index 10 to 11, ac4ac7370c8c4b0c967fa990af77a067 (127.11.132.131) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "eff1057ae98849458f656363167a5337" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250814 13:58:29.525035 11794 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 24df6cb14940420ea85c32a197b4fbc6
I20250814 13:58:29.525363 11794 log_verifier.cc:126] Checking tablet e5e55ced4bfc4fb6b6991fc92964cbaa
I20250814 13:58:29.602912 11794 log_verifier.cc:177] Verified matching terms for 11 ops in tablet e5e55ced4bfc4fb6b6991fc92964cbaa
I20250814 13:58:29.603327 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 15052
I20250814 13:58:29.632515 11794 minidump.cc:252] Setting minidump size limit to 20M
I20250814 13:58:29.633891 11794 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:58:29.635202 11794 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:58:29.647495 15658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:58:29.647750 15659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:58:29.648311 15661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 13:58:29.648730 11794 server_base.cc:1047] running on GCE node
I20250814 13:58:29.752624 11794 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250814 13:58:29.752844 11794 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250814 13:58:29.753002 11794 hybrid_clock.cc:648] HybridClock initialized: now 1755179909752985 us; error 0 us; skew 500 ppm
I20250814 13:58:29.753685 11794 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:58:29.756618 11794 webserver.cc:480] Webserver started at http://0.0.0.0:38987/ using document root <none> and password file <none>
I20250814 13:58:29.757411 11794 fs_manager.cc:362] Metadata directory not provided
I20250814 13:58:29.757612 11794 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:58:29.762415 11794 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20250814 13:58:29.765856 15666 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:58:29.766757 11794 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250814 13:58:29.767009 11794 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
uuid: "2ff0f6a4351645cd82f1dc14d3ef065d"
format_stamp: "Formatted at 2025-08-14 13:58:07 on dist-test-slave-q1tc"
I20250814 13:58:29.768590 11794 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:58:29.785913 11794 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:58:29.787693 11794 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:58:29.788095 11794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:58:29.800350 11794 sys_catalog.cc:263] Verifying existing consensus state
W20250814 13:58:29.806574 11794 sys_catalog.cc:243] For a single master config, on-disk Raft master: 127.11.132.190:39441 exists but no master address supplied!
I20250814 13:58:29.809113 11794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d: Bootstrap starting.
I20250814 13:58:29.853797 11794 log.cc:826] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d: Log is configured to *not* fsync() on all Append() calls
I20250814 13:58:29.918354 11794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=13 ignored=0} mutations{seen=21 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 13:58:29.919062 11794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d: Bootstrap complete.
I20250814 13:58:29.932133 11794 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff0f6a4351645cd82f1dc14d3ef065d" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 39441 } }
I20250814 13:58:29.932629 11794 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ff0f6a4351645cd82f1dc14d3ef065d, State: Initialized, Role: FOLLOWER
I20250814 13:58:29.933323 11794 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff0f6a4351645cd82f1dc14d3ef065d" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 39441 } }
I20250814 13:58:29.933743 11794 raft_consensus.cc:397] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [term 3 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 13:58:29.933991 11794 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [term 3 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 13:58:29.934304 11794 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [term 3 FOLLOWER]: Advancing to term 4
I20250814 13:58:29.939066 11794 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff0f6a4351645cd82f1dc14d3ef065d" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 39441 } }
I20250814 13:58:29.939651 11794 leader_election.cc:304] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ff0f6a4351645cd82f1dc14d3ef065d; no voters:
I20250814 13:58:29.940675 11794 leader_election.cc:290] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [CANDIDATE]: Term 4 election: Requested vote from peers
I20250814 13:58:29.940937 15673 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [term 4 FOLLOWER]: Leader election won for term 4
I20250814 13:58:29.942294 15673 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [term 4 LEADER]: Becoming Leader. State: Replica: 2ff0f6a4351645cd82f1dc14d3ef065d, State: Running, Role: LEADER
I20250814 13:58:29.943233 15673 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff0f6a4351645cd82f1dc14d3ef065d" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 39441 } }
I20250814 13:58:29.955813 15675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ff0f6a4351645cd82f1dc14d3ef065d. Latest consensus state: current_term: 4 leader_uuid: "2ff0f6a4351645cd82f1dc14d3ef065d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff0f6a4351645cd82f1dc14d3ef065d" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 39441 } } }
I20250814 13:58:29.955883 15674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 leader_uuid: "2ff0f6a4351645cd82f1dc14d3ef065d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff0f6a4351645cd82f1dc14d3ef065d" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 39441 } } }
I20250814 13:58:29.956651 15675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [sys.catalog]: This master's current role is: LEADER
I20250814 13:58:29.956597 15674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [sys.catalog]: This master's current role is: LEADER
I20250814 13:58:29.988595 11794 tablet_replica.cc:331] stopping tablet replica
I20250814 13:58:29.989171 11794 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [term 4 LEADER]: Raft consensus shutting down.
I20250814 13:58:29.989595 11794 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [term 4 FOLLOWER]: Raft consensus is shut down!
I20250814 13:58:29.992043 11794 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250814 13:58:29.992542 11794 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250814 13:58:30.016438 11794 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
W20250814 13:58:30.551273 15589 heartbeater.cc:646] Failed to heartbeat to 127.11.132.190:39441 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.132.190:39441: connect: Connection refused (error 111)
W20250814 13:58:30.561194 15274 heartbeater.cc:646] Failed to heartbeat to 127.11.132.190:39441 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.132.190:39441: connect: Connection refused (error 111)
W20250814 13:58:30.564771 15422 heartbeater.cc:646] Failed to heartbeat to 127.11.132.190:39441 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.132.190:39441: connect: Connection refused (error 111)
I20250814 13:58:34.950552 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 15123
I20250814 13:58:34.974589 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 15278
W20250814 13:58:34.998075 15467 connection.cc:537] client connection to 127.11.132.130:42415 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250814 13:58:34.998550 15467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250814 13:58:34.998565 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 15426
I20250814 13:58:35.021989 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.190:39441
--webserver_interface=127.11.132.190
--webserver_port=45161
--builtin_ntp_servers=127.11.132.148:37521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.132.190:39441 with env {}
W20250814 13:58:35.297169 15747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:58:35.297777 15747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:58:35.298218 15747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:58:35.326170 15747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250814 13:58:35.326400 15747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:58:35.326587 15747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250814 13:58:35.326759 15747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250814 13:58:35.357642 15747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:37521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.132.190:39441
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.190:39441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.11.132.190
--webserver_port=45161
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:58:35.358867 15747 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:58:35.360332 15747 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:58:35.370405 15753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:58:35.371181 15754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:58:36.503376 15756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:58:36.506562 15755 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1134 milliseconds
W20250814 13:58:36.507371 15747 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.136s user 0.333s sys 0.797s
W20250814 13:58:36.507599 15747 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.137s user 0.333s sys 0.797s
I20250814 13:58:36.507762 15747 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 13:58:36.508797 15747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:58:36.513572 15747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:58:36.514899 15747 hybrid_clock.cc:648] HybridClock initialized: now 1755179916514860 us; error 44 us; skew 500 ppm
I20250814 13:58:36.515661 15747 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:58:36.522377 15747 webserver.cc:480] Webserver started at http://127.11.132.190:45161/ using document root <none> and password file <none>
I20250814 13:58:36.523191 15747 fs_manager.cc:362] Metadata directory not provided
I20250814 13:58:36.523361 15747 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:58:36.530884 15747 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.002s sys 0.002s
I20250814 13:58:36.535722 15763 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:58:36.536713 15747 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20250814 13:58:36.536998 15747 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
uuid: "2ff0f6a4351645cd82f1dc14d3ef065d"
format_stamp: "Formatted at 2025-08-14 13:58:07 on dist-test-slave-q1tc"
I20250814 13:58:36.538780 15747 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:58:36.611351 15747 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:58:36.612771 15747 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:58:36.613207 15747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:58:36.678807 15747 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.190:39441
I20250814 13:58:36.678877 15814 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.190:39441 every 8 connection(s)
I20250814 13:58:36.681485 15747 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250814 13:58:36.689693 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 15747
I20250814 13:58:36.690770 15815 sys_catalog.cc:263] Verifying existing consensus state
I20250814 13:58:36.691032 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.132.129:42009
--local_ip_for_outbound_sockets=127.11.132.129
--tserver_master_addrs=127.11.132.190:39441
--webserver_port=44059
--webserver_interface=127.11.132.129
--builtin_ntp_servers=127.11.132.148:37521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250814 13:58:36.695051 15815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d: Bootstrap starting.
I20250814 13:58:36.709273 15815 log.cc:826] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d: Log is configured to *not* fsync() on all Append() calls
I20250814 13:58:36.804466 15815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=15 ignored=0} mutations{seen=23 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 13:58:36.805263 15815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d: Bootstrap complete.
I20250814 13:58:36.823026 15815 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff0f6a4351645cd82f1dc14d3ef065d" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 39441 } }
I20250814 13:58:36.825016 15815 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ff0f6a4351645cd82f1dc14d3ef065d, State: Initialized, Role: FOLLOWER
I20250814 13:58:36.825735 15815 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff0f6a4351645cd82f1dc14d3ef065d" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 39441 } }
I20250814 13:58:36.826177 15815 raft_consensus.cc:397] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [term 5 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 13:58:36.826411 15815 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [term 5 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 13:58:36.826689 15815 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [term 5 FOLLOWER]: Advancing to term 6
I20250814 13:58:36.831498 15815 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff0f6a4351645cd82f1dc14d3ef065d" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 39441 } }
I20250814 13:58:36.832089 15815 leader_election.cc:304] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ff0f6a4351645cd82f1dc14d3ef065d; no voters:
I20250814 13:58:36.834196 15815 leader_election.cc:290] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [CANDIDATE]: Term 6 election: Requested vote from peers
I20250814 13:58:36.834625 15819 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [term 6 FOLLOWER]: Leader election won for term 6
I20250814 13:58:36.837740 15819 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [term 6 LEADER]: Becoming Leader. State: Replica: 2ff0f6a4351645cd82f1dc14d3ef065d, State: Running, Role: LEADER
I20250814 13:58:36.838624 15819 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 6, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff0f6a4351645cd82f1dc14d3ef065d" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 39441 } }
I20250814 13:58:36.839100 15815 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [sys.catalog]: configured and running, proceeding with master startup.
I20250814 13:58:36.846549 15821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ff0f6a4351645cd82f1dc14d3ef065d. Latest consensus state: current_term: 6 leader_uuid: "2ff0f6a4351645cd82f1dc14d3ef065d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff0f6a4351645cd82f1dc14d3ef065d" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 39441 } } }
I20250814 13:58:36.848201 15820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 leader_uuid: "2ff0f6a4351645cd82f1dc14d3ef065d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff0f6a4351645cd82f1dc14d3ef065d" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 39441 } } }
I20250814 13:58:36.848863 15820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [sys.catalog]: This master's current role is: LEADER
I20250814 13:58:36.850064 15821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d [sys.catalog]: This master's current role is: LEADER
I20250814 13:58:36.855885 15826 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250814 13:58:36.870635 15826 catalog_manager.cc:671] Loaded metadata for table TestTable [id=5601a1f0afa44323a253ee6eb0ee8bc8]
I20250814 13:58:36.872318 15826 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=730812e3c05f49c8980b7153f836960f]
I20250814 13:58:36.873909 15826 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=dbc2e364d16e40cb809d6ebb7a2c8df6]
I20250814 13:58:36.882292 15826 tablet_loader.cc:96] loaded metadata for tablet 1db867af4d4b43dca1cf4ddc26d3321b (table TestTable2 [id=730812e3c05f49c8980b7153f836960f])
I20250814 13:58:36.883841 15826 tablet_loader.cc:96] loaded metadata for tablet 24df6cb14940420ea85c32a197b4fbc6 (table TestTable [id=5601a1f0afa44323a253ee6eb0ee8bc8])
I20250814 13:58:36.885370 15826 tablet_loader.cc:96] loaded metadata for tablet e5e55ced4bfc4fb6b6991fc92964cbaa (table TestTable1 [id=dbc2e364d16e40cb809d6ebb7a2c8df6])
I20250814 13:58:36.886780 15826 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250814 13:58:36.892393 15826 catalog_manager.cc:1261] Loaded cluster ID: da09c53bd0d74cc9a334120407ff837f
I20250814 13:58:36.892733 15826 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250814 13:58:36.900848 15826 catalog_manager.cc:1506] Loading token signing keys...
I20250814 13:58:36.906567 15826 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 2ff0f6a4351645cd82f1dc14d3ef065d: Loaded TSK: 0
I20250814 13:58:36.908114 15826 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250814 13:58:37.042526 15817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:58:37.042986 15817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:58:37.043463 15817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:58:37.073320 15817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:58:37.074111 15817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.132.129
I20250814 13:58:37.107097 15817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:37521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.132.129:42009
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.11.132.129
--webserver_port=44059
--tserver_master_addrs=127.11.132.190:39441
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.132.129
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:58:37.108345 15817 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:58:37.109853 15817 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:58:37.121873 15843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:58:37.123059 15844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:58:38.501590 15846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:58:38.504316 15845 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1375 milliseconds
W20250814 13:58:38.504776 15817 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.382s user 0.435s sys 0.914s
W20250814 13:58:38.505110 15817 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.382s user 0.438s sys 0.914s
I20250814 13:58:38.505366 15817 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 13:58:38.506623 15817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:58:38.512459 15817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:58:38.513885 15817 hybrid_clock.cc:648] HybridClock initialized: now 1755179918513819 us; error 70 us; skew 500 ppm
I20250814 13:58:38.514818 15817 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:58:38.523455 15817 webserver.cc:480] Webserver started at http://127.11.132.129:44059/ using document root <none> and password file <none>
I20250814 13:58:38.524633 15817 fs_manager.cc:362] Metadata directory not provided
I20250814 13:58:38.524951 15817 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:58:38.535297 15817 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.009s sys 0.000s
I20250814 13:58:38.541193 15853 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:58:38.542425 15817 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250814 13:58:38.542805 15817 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "eff1057ae98849458f656363167a5337"
format_stamp: "Formatted at 2025-08-14 13:58:09 on dist-test-slave-q1tc"
I20250814 13:58:38.545468 15817 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:58:38.616719 15817 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:58:38.618520 15817 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:58:38.619031 15817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:58:38.622212 15817 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 13:58:38.629356 15860 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250814 13:58:38.640167 15817 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250814 13:58:38.640452 15817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.000s sys 0.002s
I20250814 13:58:38.640780 15817 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250814 13:58:38.648123 15860 tablet_bootstrap.cc:492] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337: Bootstrap starting.
I20250814 13:58:38.652628 15817 ts_tablet_manager.cc:610] Registered 2 tablets
I20250814 13:58:38.652868 15817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.012s user 0.009s sys 0.000s
I20250814 13:58:38.710613 15860 log.cc:826] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337: Log is configured to *not* fsync() on all Append() calls
I20250814 13:58:38.834321 15860 tablet_bootstrap.cc:492] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 13:58:38.835597 15860 tablet_bootstrap.cc:492] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337: Bootstrap complete.
I20250814 13:58:38.837518 15860 ts_tablet_manager.cc:1397] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337: Time spent bootstrapping tablet: real 0.190s user 0.149s sys 0.031s
I20250814 13:58:38.839660 15817 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.129:42009
I20250814 13:58:38.839811 15967 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.129:42009 every 8 connection(s)
I20250814 13:58:38.843092 15817 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250814 13:58:38.845018 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 15817
I20250814 13:58:38.846773 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.132.130:42415
--local_ip_for_outbound_sockets=127.11.132.130
--tserver_master_addrs=127.11.132.190:39441
--webserver_port=44439
--webserver_interface=127.11.132.130
--builtin_ntp_servers=127.11.132.148:37521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250814 13:58:38.863502 15860 raft_consensus.cc:357] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } } peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } attrs { promote: false } }
I20250814 13:58:38.867538 15860 raft_consensus.cc:738] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff1057ae98849458f656363167a5337, State: Initialized, Role: FOLLOWER
I20250814 13:58:38.868870 15860 consensus_queue.cc:260] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } } peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } attrs { promote: false } }
I20250814 13:58:38.873263 15860 ts_tablet_manager.cc:1428] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337: Time spent starting tablet: real 0.035s user 0.034s sys 0.000s
I20250814 13:58:38.874051 15860 tablet_bootstrap.cc:492] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337: Bootstrap starting.
I20250814 13:58:38.902837 15968 heartbeater.cc:344] Connected to a master server at 127.11.132.190:39441
I20250814 13:58:38.903321 15968 heartbeater.cc:461] Registering TS with master...
I20250814 13:58:38.904572 15968 heartbeater.cc:507] Master 127.11.132.190:39441 requested a full tablet report, sending...
I20250814 13:58:38.909997 15780 ts_manager.cc:194] Registered new tserver with Master: eff1057ae98849458f656363167a5337 (127.11.132.129:42009)
I20250814 13:58:38.914630 15780 catalog_manager.cc:5582] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 reported cstate change: config changed from index -1 to 11, term changed from 0 to 2, VOTER 513c7d5f4e3f41e5b9f8918182af4955 (127.11.132.130) added, VOTER ac4ac7370c8c4b0c967fa990af77a067 (127.11.132.131) added, VOTER eff1057ae98849458f656363167a5337 (127.11.132.129) added. New cstate: current_term: 2 committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } } peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } attrs { promote: false } } }
I20250814 13:58:38.982283 15780 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.132.129:50481
I20250814 13:58:38.986434 15968 heartbeater.cc:499] Master 127.11.132.190:39441 was elected leader, sending a full tablet report...
I20250814 13:58:39.009747 15860 tablet_bootstrap.cc:492] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 13:58:39.010380 15860 tablet_bootstrap.cc:492] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337: Bootstrap complete.
I20250814 13:58:39.011483 15860 ts_tablet_manager.cc:1397] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337: Time spent bootstrapping tablet: real 0.138s user 0.115s sys 0.016s
I20250814 13:58:39.013011 15860 raft_consensus.cc:357] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } } peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } attrs { promote: false } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } }
I20250814 13:58:39.013556 15860 raft_consensus.cc:738] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff1057ae98849458f656363167a5337, State: Initialized, Role: FOLLOWER
I20250814 13:58:39.014137 15860 consensus_queue.cc:260] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } } peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } attrs { promote: false } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } }
I20250814 13:58:39.015903 15860 ts_tablet_manager.cc:1428] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20250814 13:58:39.237691 15969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:58:39.238160 15969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:58:39.238631 15969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:58:39.269217 15969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:58:39.270040 15969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.132.130
I20250814 13:58:39.303061 15969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:37521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.132.130:42415
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.11.132.130
--webserver_port=44439
--tserver_master_addrs=127.11.132.190:39441
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.132.130
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:58:39.304323 15969 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:58:39.305823 15969 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:58:39.317124 15984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 13:58:40.110888 15991 raft_consensus.cc:491] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250814 13:58:40.111958 15991 raft_consensus.cc:513] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } } peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } attrs { promote: false } }
I20250814 13:58:40.121991 15991 leader_election.cc:290] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 513c7d5f4e3f41e5b9f8918182af4955 (127.11.132.130:42415), ac4ac7370c8c4b0c967fa990af77a067 (127.11.132.131:44563)
W20250814 13:58:40.126478 15857 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.130:42415: connect: Connection refused (error 111)
W20250814 13:58:40.136312 15857 leader_election.cc:336] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 513c7d5f4e3f41e5b9f8918182af4955 (127.11.132.130:42415): Network error: Client connection negotiation failed: client connection to 127.11.132.130:42415: connect: Connection refused (error 111)
W20250814 13:58:40.137398 15857 leader_election.cc:336] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ac4ac7370c8c4b0c967fa990af77a067 (127.11.132.131:44563): Network error: Client connection negotiation failed: client connection to 127.11.132.131:44563: connect: Connection refused (error 111)
I20250814 13:58:40.137972 15857 leader_election.cc:304] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eff1057ae98849458f656363167a5337; no voters: 513c7d5f4e3f41e5b9f8918182af4955, ac4ac7370c8c4b0c967fa990af77a067
I20250814 13:58:40.138960 15991 raft_consensus.cc:2747] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250814 13:58:40.367785 15991 raft_consensus.cc:491] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250814 13:58:40.368402 15991 raft_consensus.cc:513] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } } peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } attrs { promote: false } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } }
I20250814 13:58:40.370946 15991 leader_election.cc:290] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac4ac7370c8c4b0c967fa990af77a067 (127.11.132.131:44563), 513c7d5f4e3f41e5b9f8918182af4955 (127.11.132.130:42415)
W20250814 13:58:40.382053 15857 leader_election.cc:336] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ac4ac7370c8c4b0c967fa990af77a067 (127.11.132.131:44563): Network error: Client connection negotiation failed: client connection to 127.11.132.131:44563: connect: Connection refused (error 111)
W20250814 13:58:40.382977 15857 leader_election.cc:336] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 513c7d5f4e3f41e5b9f8918182af4955 (127.11.132.130:42415): Network error: Client connection negotiation failed: client connection to 127.11.132.130:42415: connect: Connection refused (error 111)
I20250814 13:58:40.383656 15857 leader_election.cc:304] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eff1057ae98849458f656363167a5337; no voters: 513c7d5f4e3f41e5b9f8918182af4955, ac4ac7370c8c4b0c967fa990af77a067
I20250814 13:58:40.384663 15991 raft_consensus.cc:2747] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250814 13:58:39.318195 15985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:58:40.720119 15983 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 15969
W20250814 13:58:41.038697 15983 kernel_stack_watchdog.cc:198] Thread 15969 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 399ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250814 13:58:41.039172 15969 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.721s user 0.538s sys 1.128s
W20250814 13:58:41.039518 15969 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.721s user 0.538s sys 1.128s
W20250814 13:58:41.040163 15987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:58:41.043190 15986 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1723 milliseconds
I20250814 13:58:41.043222 15969 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 13:58:41.044368 15969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:58:41.046295 15969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:58:41.047616 15969 hybrid_clock.cc:648] HybridClock initialized: now 1755179921047579 us; error 42 us; skew 500 ppm
I20250814 13:58:41.048352 15969 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:58:41.054026 15969 webserver.cc:480] Webserver started at http://127.11.132.130:44439/ using document root <none> and password file <none>
I20250814 13:58:41.054909 15969 fs_manager.cc:362] Metadata directory not provided
I20250814 13:58:41.055110 15969 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:58:41.062024 15969 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20250814 13:58:41.066373 15999 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:58:41.067216 15969 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250814 13:58:41.067481 15969 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "513c7d5f4e3f41e5b9f8918182af4955"
format_stamp: "Formatted at 2025-08-14 13:58:11 on dist-test-slave-q1tc"
I20250814 13:58:41.069180 15969 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:58:41.117031 15969 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:58:41.118396 15969 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:58:41.118791 15969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:58:41.121228 15969 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 13:58:41.126502 16006 ts_tablet_manager.cc:542] Loading tablet metadata (0/3 complete)
I20250814 13:58:41.140534 15969 ts_tablet_manager.cc:579] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20250814 13:58:41.140759 15969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.001s sys 0.000s
I20250814 13:58:41.141009 15969 ts_tablet_manager.cc:594] Registering tablets (0/3 complete)
I20250814 13:58:41.145728 16006 tablet_bootstrap.cc:492] T 24df6cb14940420ea85c32a197b4fbc6 P 513c7d5f4e3f41e5b9f8918182af4955: Bootstrap starting.
I20250814 13:58:41.151018 15969 ts_tablet_manager.cc:610] Registered 3 tablets
I20250814 13:58:41.151206 15969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.010s user 0.009s sys 0.000s
I20250814 13:58:41.194113 16006 log.cc:826] T 24df6cb14940420ea85c32a197b4fbc6 P 513c7d5f4e3f41e5b9f8918182af4955: Log is configured to *not* fsync() on all Append() calls
I20250814 13:58:41.275249 16006 tablet_bootstrap.cc:492] T 24df6cb14940420ea85c32a197b4fbc6 P 513c7d5f4e3f41e5b9f8918182af4955: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 13:58:41.276031 16006 tablet_bootstrap.cc:492] T 24df6cb14940420ea85c32a197b4fbc6 P 513c7d5f4e3f41e5b9f8918182af4955: Bootstrap complete.
I20250814 13:58:41.277824 16006 ts_tablet_manager.cc:1397] T 24df6cb14940420ea85c32a197b4fbc6 P 513c7d5f4e3f41e5b9f8918182af4955: Time spent bootstrapping tablet: real 0.132s user 0.105s sys 0.026s
I20250814 13:58:41.293558 16006 raft_consensus.cc:357] T 24df6cb14940420ea85c32a197b4fbc6 P 513c7d5f4e3f41e5b9f8918182af4955 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } } peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } attrs { promote: false } }
I20250814 13:58:41.295411 16006 raft_consensus.cc:738] T 24df6cb14940420ea85c32a197b4fbc6 P 513c7d5f4e3f41e5b9f8918182af4955 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 513c7d5f4e3f41e5b9f8918182af4955, State: Initialized, Role: FOLLOWER
I20250814 13:58:41.296270 16006 consensus_queue.cc:260] T 24df6cb14940420ea85c32a197b4fbc6 P 513c7d5f4e3f41e5b9f8918182af4955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } } peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } attrs { promote: false } }
I20250814 13:58:41.301412 16006 ts_tablet_manager.cc:1428] T 24df6cb14940420ea85c32a197b4fbc6 P 513c7d5f4e3f41e5b9f8918182af4955: Time spent starting tablet: real 0.023s user 0.023s sys 0.000s
I20250814 13:58:41.302331 16006 tablet_bootstrap.cc:492] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955: Bootstrap starting.
I20250814 13:58:41.318653 15969 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.130:42415
I20250814 13:58:41.318715 16114 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.130:42415 every 8 connection(s)
I20250814 13:58:41.323680 15969 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250814 13:58:41.329149 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 15969
I20250814 13:58:41.331315 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.132.131:44563
--local_ip_for_outbound_sockets=127.11.132.131
--tserver_master_addrs=127.11.132.190:39441
--webserver_port=39163
--webserver_interface=127.11.132.131
--builtin_ntp_servers=127.11.132.148:37521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250814 13:58:41.358290 16115 heartbeater.cc:344] Connected to a master server at 127.11.132.190:39441
I20250814 13:58:41.358788 16115 heartbeater.cc:461] Registering TS with master...
I20250814 13:58:41.360064 16115 heartbeater.cc:507] Master 127.11.132.190:39441 requested a full tablet report, sending...
I20250814 13:58:41.364998 15780 ts_manager.cc:194] Registered new tserver with Master: 513c7d5f4e3f41e5b9f8918182af4955 (127.11.132.130:42415)
I20250814 13:58:41.370003 15780 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.132.130:38595
I20250814 13:58:41.403673 16006 tablet_bootstrap.cc:492] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 13:58:41.404348 16006 tablet_bootstrap.cc:492] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955: Bootstrap complete.
I20250814 13:58:41.405568 16006 ts_tablet_manager.cc:1397] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955: Time spent bootstrapping tablet: real 0.103s user 0.082s sys 0.020s
I20250814 13:58:41.407051 16006 raft_consensus.cc:357] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } }
I20250814 13:58:41.407379 16006 raft_consensus.cc:738] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 513c7d5f4e3f41e5b9f8918182af4955, State: Initialized, Role: FOLLOWER
I20250814 13:58:41.407796 16006 consensus_queue.cc:260] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } }
I20250814 13:58:41.408139 16006 raft_consensus.cc:397] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 13:58:41.408367 16006 raft_consensus.cc:491] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 13:58:41.408636 16006 raft_consensus.cc:3058] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955 [term 2 FOLLOWER]: Advancing to term 3
I20250814 13:58:41.413980 16006 raft_consensus.cc:513] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } }
I20250814 13:58:41.414530 16006 leader_election.cc:304] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 513c7d5f4e3f41e5b9f8918182af4955; no voters:
I20250814 13:58:41.414958 16006 leader_election.cc:290] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250814 13:58:41.415222 16099 raft_consensus.cc:2802] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955 [term 3 FOLLOWER]: Leader election won for term 3
I20250814 13:58:41.418107 16099 raft_consensus.cc:695] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955 [term 3 LEADER]: Becoming Leader. State: Replica: 513c7d5f4e3f41e5b9f8918182af4955, State: Running, Role: LEADER
I20250814 13:58:41.418358 16006 ts_tablet_manager.cc:1428] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20250814 13:58:41.418568 16115 heartbeater.cc:499] Master 127.11.132.190:39441 was elected leader, sending a full tablet report...
I20250814 13:58:41.418977 16006 tablet_bootstrap.cc:492] T e5e55ced4bfc4fb6b6991fc92964cbaa P 513c7d5f4e3f41e5b9f8918182af4955: Bootstrap starting.
I20250814 13:58:41.419230 16099 consensus_queue.cc:237] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } }
I20250814 13:58:41.434743 15780 catalog_manager.cc:5582] T 1db867af4d4b43dca1cf4ddc26d3321b P 513c7d5f4e3f41e5b9f8918182af4955 reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "513c7d5f4e3f41e5b9f8918182af4955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } health_report { overall_health: HEALTHY } } }
I20250814 13:58:41.508978 16006 tablet_bootstrap.cc:492] T e5e55ced4bfc4fb6b6991fc92964cbaa P 513c7d5f4e3f41e5b9f8918182af4955: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 13:58:41.509629 16006 tablet_bootstrap.cc:492] T e5e55ced4bfc4fb6b6991fc92964cbaa P 513c7d5f4e3f41e5b9f8918182af4955: Bootstrap complete.
I20250814 13:58:41.510651 16006 ts_tablet_manager.cc:1397] T e5e55ced4bfc4fb6b6991fc92964cbaa P 513c7d5f4e3f41e5b9f8918182af4955: Time spent bootstrapping tablet: real 0.092s user 0.077s sys 0.012s
I20250814 13:58:41.512256 16006 raft_consensus.cc:357] T e5e55ced4bfc4fb6b6991fc92964cbaa P 513c7d5f4e3f41e5b9f8918182af4955 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } } peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } attrs { promote: false } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } }
I20250814 13:58:41.512694 16006 raft_consensus.cc:738] T e5e55ced4bfc4fb6b6991fc92964cbaa P 513c7d5f4e3f41e5b9f8918182af4955 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 513c7d5f4e3f41e5b9f8918182af4955, State: Initialized, Role: FOLLOWER
I20250814 13:58:41.513131 16006 consensus_queue.cc:260] T e5e55ced4bfc4fb6b6991fc92964cbaa P 513c7d5f4e3f41e5b9f8918182af4955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } } peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } attrs { promote: false } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } }
I20250814 13:58:41.514300 16006 ts_tablet_manager.cc:1428] T e5e55ced4bfc4fb6b6991fc92964cbaa P 513c7d5f4e3f41e5b9f8918182af4955: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
W20250814 13:58:41.642445 16119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:58:41.642979 16119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:58:41.643471 16119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:58:41.672734 16119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:58:41.673532 16119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.132.131
I20250814 13:58:41.705422 16119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:37521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.132.131:44563
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.11.132.131
--webserver_port=39163
--tserver_master_addrs=127.11.132.190:39441
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.132.131
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:58:41.706626 16119 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:58:41.708012 16119 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:58:41.719452 16135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 13:58:41.880373 16141 raft_consensus.cc:491] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250814 13:58:41.881196 16141 raft_consensus.cc:513] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } } peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } attrs { promote: false } }
I20250814 13:58:41.890408 16141 leader_election.cc:290] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 513c7d5f4e3f41e5b9f8918182af4955 (127.11.132.130:42415), ac4ac7370c8c4b0c967fa990af77a067 (127.11.132.131:44563)
W20250814 13:58:41.925659 15857 leader_election.cc:336] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ac4ac7370c8c4b0c967fa990af77a067 (127.11.132.131:44563): Network error: Client connection negotiation failed: client connection to 127.11.132.131:44563: connect: Connection refused (error 111)
I20250814 13:58:41.930167 16069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24df6cb14940420ea85c32a197b4fbc6" candidate_uuid: "eff1057ae98849458f656363167a5337" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "513c7d5f4e3f41e5b9f8918182af4955" is_pre_election: true
I20250814 13:58:41.931016 16069 raft_consensus.cc:2466] T 24df6cb14940420ea85c32a197b4fbc6 P 513c7d5f4e3f41e5b9f8918182af4955 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff1057ae98849458f656363167a5337 in term 2.
I20250814 13:58:41.933133 15857 leader_election.cc:304] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 513c7d5f4e3f41e5b9f8918182af4955, eff1057ae98849458f656363167a5337; no voters: ac4ac7370c8c4b0c967fa990af77a067
I20250814 13:58:41.934141 16141 raft_consensus.cc:2802] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250814 13:58:41.934521 16141 raft_consensus.cc:491] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250814 13:58:41.934911 16141 raft_consensus.cc:3058] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [term 2 FOLLOWER]: Advancing to term 3
I20250814 13:58:41.947077 16141 raft_consensus.cc:513] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } } peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } attrs { promote: false } }
I20250814 13:58:41.950138 16069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24df6cb14940420ea85c32a197b4fbc6" candidate_uuid: "eff1057ae98849458f656363167a5337" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "513c7d5f4e3f41e5b9f8918182af4955"
I20250814 13:58:41.950781 16069 raft_consensus.cc:3058] T 24df6cb14940420ea85c32a197b4fbc6 P 513c7d5f4e3f41e5b9f8918182af4955 [term 2 FOLLOWER]: Advancing to term 3
I20250814 13:58:41.959049 16069 raft_consensus.cc:2466] T 24df6cb14940420ea85c32a197b4fbc6 P 513c7d5f4e3f41e5b9f8918182af4955 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff1057ae98849458f656363167a5337 in term 3.
W20250814 13:58:41.965353 15857 leader_election.cc:336] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ac4ac7370c8c4b0c967fa990af77a067 (127.11.132.131:44563): Network error: Client connection negotiation failed: client connection to 127.11.132.131:44563: connect: Connection refused (error 111)
I20250814 13:58:41.966107 16141 leader_election.cc:290] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [CANDIDATE]: Term 3 election: Requested vote from peers 513c7d5f4e3f41e5b9f8918182af4955 (127.11.132.130:42415), ac4ac7370c8c4b0c967fa990af77a067 (127.11.132.131:44563)
I20250814 13:58:41.967388 15857 leader_election.cc:304] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 513c7d5f4e3f41e5b9f8918182af4955, eff1057ae98849458f656363167a5337; no voters: ac4ac7370c8c4b0c967fa990af77a067
I20250814 13:58:41.970949 16141 raft_consensus.cc:2802] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [term 3 FOLLOWER]: Leader election won for term 3
I20250814 13:58:41.972311 16141 raft_consensus.cc:695] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [term 3 LEADER]: Becoming Leader. State: Replica: eff1057ae98849458f656363167a5337, State: Running, Role: LEADER
I20250814 13:58:41.974045 16141 consensus_queue.cc:237] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } } peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } attrs { promote: false } }
I20250814 13:58:41.993227 15780 catalog_manager.cc:5582] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 reported cstate change: term changed from 2 to 3, leader changed from <none> to eff1057ae98849458f656363167a5337 (127.11.132.129). New cstate: current_term: 3 leader_uuid: "eff1057ae98849458f656363167a5337" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250814 13:58:42.317374 16141 raft_consensus.cc:491] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250814 13:58:42.318032 16141 raft_consensus.cc:513] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } } peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } attrs { promote: false } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } }
I20250814 13:58:42.320384 16141 leader_election.cc:290] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac4ac7370c8c4b0c967fa990af77a067 (127.11.132.131:44563), 513c7d5f4e3f41e5b9f8918182af4955 (127.11.132.130:42415)
I20250814 13:58:42.323422 16069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5e55ced4bfc4fb6b6991fc92964cbaa" candidate_uuid: "eff1057ae98849458f656363167a5337" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "513c7d5f4e3f41e5b9f8918182af4955" is_pre_election: true
I20250814 13:58:42.324206 16069 raft_consensus.cc:2466] T e5e55ced4bfc4fb6b6991fc92964cbaa P 513c7d5f4e3f41e5b9f8918182af4955 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff1057ae98849458f656363167a5337 in term 2.
I20250814 13:58:42.326190 15857 leader_election.cc:304] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 513c7d5f4e3f41e5b9f8918182af4955, eff1057ae98849458f656363167a5337; no voters:
I20250814 13:58:42.328416 16141 raft_consensus.cc:2802] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250814 13:58:42.328824 16141 raft_consensus.cc:491] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250814 13:58:42.329231 16141 raft_consensus.cc:3058] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337 [term 2 FOLLOWER]: Advancing to term 3
W20250814 13:58:42.337229 15857 leader_election.cc:336] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ac4ac7370c8c4b0c967fa990af77a067 (127.11.132.131:44563): Network error: Client connection negotiation failed: client connection to 127.11.132.131:44563: connect: Connection refused (error 111)
I20250814 13:58:42.349668 16141 raft_consensus.cc:513] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } } peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } attrs { promote: false } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } }
W20250814 13:58:42.356884 15857 leader_election.cc:336] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ac4ac7370c8c4b0c967fa990af77a067 (127.11.132.131:44563): Network error: Client connection negotiation failed: client connection to 127.11.132.131:44563: connect: Connection refused (error 111)
I20250814 13:58:42.358461 16141 leader_election.cc:290] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337 [CANDIDATE]: Term 3 election: Requested vote from peers ac4ac7370c8c4b0c967fa990af77a067 (127.11.132.131:44563), 513c7d5f4e3f41e5b9f8918182af4955 (127.11.132.130:42415)
I20250814 13:58:42.359239 16069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5e55ced4bfc4fb6b6991fc92964cbaa" candidate_uuid: "eff1057ae98849458f656363167a5337" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "513c7d5f4e3f41e5b9f8918182af4955"
I20250814 13:58:42.359879 16069 raft_consensus.cc:3058] T e5e55ced4bfc4fb6b6991fc92964cbaa P 513c7d5f4e3f41e5b9f8918182af4955 [term 2 FOLLOWER]: Advancing to term 3
I20250814 13:58:42.370086 16069 raft_consensus.cc:2466] T e5e55ced4bfc4fb6b6991fc92964cbaa P 513c7d5f4e3f41e5b9f8918182af4955 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff1057ae98849458f656363167a5337 in term 3.
I20250814 13:58:42.373997 15857 leader_election.cc:304] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 513c7d5f4e3f41e5b9f8918182af4955, eff1057ae98849458f656363167a5337; no voters: ac4ac7370c8c4b0c967fa990af77a067
I20250814 13:58:42.375298 16141 raft_consensus.cc:2802] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337 [term 3 FOLLOWER]: Leader election won for term 3
I20250814 13:58:42.375819 16141 raft_consensus.cc:695] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337 [term 3 LEADER]: Becoming Leader. State: Replica: eff1057ae98849458f656363167a5337, State: Running, Role: LEADER
I20250814 13:58:42.376609 16141 consensus_queue.cc:237] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } } peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } attrs { promote: false } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } }
I20250814 13:58:42.405114 15780 catalog_manager.cc:5582] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337 reported cstate change: term changed from 2 to 3, leader changed from ac4ac7370c8c4b0c967fa990af77a067 (127.11.132.131) to eff1057ae98849458f656363167a5337 (127.11.132.129). New cstate: current_term: 3 leader_uuid: "eff1057ae98849458f656363167a5337" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250814 13:58:42.520687 16069 raft_consensus.cc:1273] T 24df6cb14940420ea85c32a197b4fbc6 P 513c7d5f4e3f41e5b9f8918182af4955 [term 3 FOLLOWER]: Refusing update from remote peer eff1057ae98849458f656363167a5337: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20250814 13:58:42.522933 16141 consensus_queue.cc:1035] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [LEADER]: Connected to new peer: Peer: permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20250814 13:58:42.552042 15857 consensus_peers.cc:489] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 -> Peer ac4ac7370c8c4b0c967fa990af77a067 (127.11.132.131:44563): Couldn't send request to peer ac4ac7370c8c4b0c967fa990af77a067. Status: Network error: Client connection negotiation failed: client connection to 127.11.132.131:44563: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250814 13:58:42.623739 15923 consensus_queue.cc:237] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } }
I20250814 13:58:42.631700 16069 raft_consensus.cc:1273] T 24df6cb14940420ea85c32a197b4fbc6 P 513c7d5f4e3f41e5b9f8918182af4955 [term 3 FOLLOWER]: Refusing update from remote peer eff1057ae98849458f656363167a5337: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20250814 13:58:42.633975 16146 consensus_queue.cc:1035] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [LEADER]: Connected to new peer: Peer: permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20250814 13:58:42.642588 16141 raft_consensus.cc:2953] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [term 3 LEADER]: Committing config change with OpId 3.13: config changed from index 11 to 13, VOTER ac4ac7370c8c4b0c967fa990af77a067 (127.11.132.131) evicted. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } } }
I20250814 13:58:42.655524 16068 raft_consensus.cc:2953] T 24df6cb14940420ea85c32a197b4fbc6 P 513c7d5f4e3f41e5b9f8918182af4955 [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 11 to 13, VOTER ac4ac7370c8c4b0c967fa990af77a067 (127.11.132.131) evicted. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } } }
I20250814 13:58:42.671619 15767 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 24df6cb14940420ea85c32a197b4fbc6 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250814 13:58:42.678588 15780 catalog_manager.cc:5582] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 reported cstate change: config changed from index 11 to 13, VOTER ac4ac7370c8c4b0c967fa990af77a067 (127.11.132.131) evicted. New cstate: current_term: 3 leader_uuid: "eff1057ae98849458f656363167a5337" committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250814 13:58:42.692260 15780 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet 24df6cb14940420ea85c32a197b4fbc6 on TS ac4ac7370c8c4b0c967fa990af77a067: Not found: failed to reset TS proxy: Could not find TS for UUID ac4ac7370c8c4b0c967fa990af77a067
I20250814 13:58:42.696244 15923 consensus_queue.cc:237] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 11, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } }
I20250814 13:58:42.719039 16141 raft_consensus.cc:2953] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 13 to 14, VOTER 513c7d5f4e3f41e5b9f8918182af4955 (127.11.132.130) evicted. New config: { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } } }
I20250814 13:58:42.733861 15767 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 24df6cb14940420ea85c32a197b4fbc6 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250814 13:58:42.739852 15780 catalog_manager.cc:5582] T 24df6cb14940420ea85c32a197b4fbc6 P eff1057ae98849458f656363167a5337 reported cstate change: config changed from index 13 to 14, VOTER 513c7d5f4e3f41e5b9f8918182af4955 (127.11.132.130) evicted. New cstate: current_term: 3 leader_uuid: "eff1057ae98849458f656363167a5337" committed_config { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } health_report { overall_health: HEALTHY } } }
W20250814 13:58:42.749204 15765 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 24df6cb14940420ea85c32a197b4fbc6 on TS ac4ac7370c8c4b0c967fa990af77a067 failed: Not found: failed to reset TS proxy: Could not find TS for UUID ac4ac7370c8c4b0c967fa990af77a067
I20250814 13:58:42.783996 16049 tablet_service.cc:1515] Processing DeleteTablet for tablet 24df6cb14940420ea85c32a197b4fbc6 with delete_type TABLET_DATA_TOMBSTONED (TS 513c7d5f4e3f41e5b9f8918182af4955 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:34358
I20250814 13:58:42.800300 16159 tablet_replica.cc:331] stopping tablet replica
I20250814 13:58:42.801640 16159 raft_consensus.cc:2241] T 24df6cb14940420ea85c32a197b4fbc6 P 513c7d5f4e3f41e5b9f8918182af4955 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250814 13:58:42.802543 16159 raft_consensus.cc:2270] T 24df6cb14940420ea85c32a197b4fbc6 P 513c7d5f4e3f41e5b9f8918182af4955 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250814 13:58:42.806741 16159 ts_tablet_manager.cc:1905] T 24df6cb14940420ea85c32a197b4fbc6 P 513c7d5f4e3f41e5b9f8918182af4955: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250814 13:58:42.829123 16159 ts_tablet_manager.cc:1918] T 24df6cb14940420ea85c32a197b4fbc6 P 513c7d5f4e3f41e5b9f8918182af4955: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.13
I20250814 13:58:42.829682 16159 log.cc:1199] T 24df6cb14940420ea85c32a197b4fbc6 P 513c7d5f4e3f41e5b9f8918182af4955: Deleting WAL directory at /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal/wals/24df6cb14940420ea85c32a197b4fbc6
I20250814 13:58:42.832046 15767 catalog_manager.cc:4928] TS 513c7d5f4e3f41e5b9f8918182af4955 (127.11.132.130:42415): tablet 24df6cb14940420ea85c32a197b4fbc6 (table TestTable [id=5601a1f0afa44323a253ee6eb0ee8bc8]) successfully deleted
W20250814 13:58:42.886526 15857 consensus_peers.cc:489] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337 -> Peer ac4ac7370c8c4b0c967fa990af77a067 (127.11.132.131:44563): Couldn't send request to peer ac4ac7370c8c4b0c967fa990af77a067. Status: Network error: Client connection negotiation failed: client connection to 127.11.132.131:44563: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250814 13:58:42.977820 16068 raft_consensus.cc:1273] T e5e55ced4bfc4fb6b6991fc92964cbaa P 513c7d5f4e3f41e5b9f8918182af4955 [term 3 FOLLOWER]: Refusing update from remote peer eff1057ae98849458f656363167a5337: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20250814 13:58:42.979990 16141 consensus_queue.cc:1035] T e5e55ced4bfc4fb6b6991fc92964cbaa P eff1057ae98849458f656363167a5337 [LEADER]: Connected to new peer: Peer: permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20250814 13:58:43.122036 16134 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 16119
W20250814 13:58:43.201903 16119 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.482s user 0.456s sys 0.883s
W20250814 13:58:43.202391 16119 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.483s user 0.456s sys 0.883s
W20250814 13:58:41.719851 16136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:58:43.203572 16137 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1483 milliseconds
I20250814 13:58:43.203962 16119 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250814 13:58:43.204087 16138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 13:58:43.207536 16119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:58:43.209504 16119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:58:43.210826 16119 hybrid_clock.cc:648] HybridClock initialized: now 1755179923210783 us; error 38 us; skew 500 ppm
I20250814 13:58:43.211578 16119 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:58:43.217164 16119 webserver.cc:480] Webserver started at http://127.11.132.131:39163/ using document root <none> and password file <none>
I20250814 13:58:43.218152 16119 fs_manager.cc:362] Metadata directory not provided
I20250814 13:58:43.218364 16119 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:58:43.225936 16119 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20250814 13:58:43.230074 16165 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:58:43.230981 16119 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20250814 13:58:43.231279 16119 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ac4ac7370c8c4b0c967fa990af77a067"
format_stamp: "Formatted at 2025-08-14 13:58:13 on dist-test-slave-q1tc"
I20250814 13:58:43.233042 16119 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:58:43.278119 16119 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:58:43.279378 16119 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:58:43.279764 16119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:58:43.282100 16119 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 13:58:43.288547 16172 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250814 13:58:43.299188 16119 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250814 13:58:43.299409 16119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.012s user 0.001s sys 0.001s
I20250814 13:58:43.299647 16119 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250814 13:58:43.304581 16172 tablet_bootstrap.cc:492] T 24df6cb14940420ea85c32a197b4fbc6 P ac4ac7370c8c4b0c967fa990af77a067: Bootstrap starting.
I20250814 13:58:43.307050 16119 ts_tablet_manager.cc:610] Registered 2 tablets
I20250814 13:58:43.307255 16119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.008s user 0.006s sys 0.000s
I20250814 13:58:43.365327 16172 log.cc:826] T 24df6cb14940420ea85c32a197b4fbc6 P ac4ac7370c8c4b0c967fa990af77a067: Log is configured to *not* fsync() on all Append() calls
I20250814 13:58:43.458294 16172 tablet_bootstrap.cc:492] T 24df6cb14940420ea85c32a197b4fbc6 P ac4ac7370c8c4b0c967fa990af77a067: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 13:58:43.459545 16172 tablet_bootstrap.cc:492] T 24df6cb14940420ea85c32a197b4fbc6 P ac4ac7370c8c4b0c967fa990af77a067: Bootstrap complete.
I20250814 13:58:43.460901 16172 ts_tablet_manager.cc:1397] T 24df6cb14940420ea85c32a197b4fbc6 P ac4ac7370c8c4b0c967fa990af77a067: Time spent bootstrapping tablet: real 0.157s user 0.101s sys 0.050s
I20250814 13:58:43.476338 16172 raft_consensus.cc:357] T 24df6cb14940420ea85c32a197b4fbc6 P ac4ac7370c8c4b0c967fa990af77a067 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } } peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } attrs { promote: false } }
I20250814 13:58:43.478668 16172 raft_consensus.cc:738] T 24df6cb14940420ea85c32a197b4fbc6 P ac4ac7370c8c4b0c967fa990af77a067 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac4ac7370c8c4b0c967fa990af77a067, State: Initialized, Role: FOLLOWER
I20250814 13:58:43.479686 16172 consensus_queue.cc:260] T 24df6cb14940420ea85c32a197b4fbc6 P ac4ac7370c8c4b0c967fa990af77a067 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } } peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } attrs { promote: false } }
I20250814 13:58:43.483543 16172 ts_tablet_manager.cc:1428] T 24df6cb14940420ea85c32a197b4fbc6 P ac4ac7370c8c4b0c967fa990af77a067: Time spent starting tablet: real 0.022s user 0.021s sys 0.001s
I20250814 13:58:43.484238 16172 tablet_bootstrap.cc:492] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067: Bootstrap starting.
I20250814 13:58:43.491219 16119 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.131:44563
I20250814 13:58:43.491562 16280 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.131:44563 every 8 connection(s)
I20250814 13:58:43.493763 16119 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250814 13:58:43.502552 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 16119
I20250814 13:58:43.520380 16281 heartbeater.cc:344] Connected to a master server at 127.11.132.190:39441
I20250814 13:58:43.520757 16281 heartbeater.cc:461] Registering TS with master...
I20250814 13:58:43.521966 16281 heartbeater.cc:507] Master 127.11.132.190:39441 requested a full tablet report, sending...
I20250814 13:58:43.525876 15779 ts_manager.cc:194] Registered new tserver with Master: ac4ac7370c8c4b0c967fa990af77a067 (127.11.132.131:44563)
I20250814 13:58:43.528867 15779 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.132.131:35421
I20250814 13:58:43.539579 11794 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250814 13:58:43.543633 11794 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20250814 13:58:43.546823 11794 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 24df6cb14940420ea85c32a197b4fbc6: tablet_id: "24df6cb14940420ea85c32a197b4fbc6" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
I20250814 13:58:43.576301 16172 tablet_bootstrap.cc:492] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 13:58:43.576911 16172 tablet_bootstrap.cc:492] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067: Bootstrap complete.
I20250814 13:58:43.577967 16172 ts_tablet_manager.cc:1397] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067: Time spent bootstrapping tablet: real 0.094s user 0.075s sys 0.016s
I20250814 13:58:43.579416 16172 raft_consensus.cc:357] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } } peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } attrs { promote: false } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } }
I20250814 13:58:43.579841 16172 raft_consensus.cc:738] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac4ac7370c8c4b0c967fa990af77a067, State: Initialized, Role: FOLLOWER
I20250814 13:58:43.580274 16172 consensus_queue.cc:260] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ac4ac7370c8c4b0c967fa990af77a067" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 44563 } } peers { permanent_uuid: "eff1057ae98849458f656363167a5337" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 42009 } attrs { promote: false } } peers { permanent_uuid: "513c7d5f4e3f41e5b9f8918182af4955" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 42415 } attrs { promote: false } }
I20250814 13:58:43.581120 16281 heartbeater.cc:499] Master 127.11.132.190:39441 was elected leader, sending a full tablet report...
I20250814 13:58:43.581521 16172 ts_tablet_manager.cc:1428] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20250814 13:58:43.829205 16235 raft_consensus.cc:3058] T e5e55ced4bfc4fb6b6991fc92964cbaa P ac4ac7370c8c4b0c967fa990af77a067 [term 2 FOLLOWER]: Advancing to term 3
I20250814 13:58:43.925685 16215 tablet_service.cc:1515] Processing DeleteTablet for tablet 24df6cb14940420ea85c32a197b4fbc6 with delete_type TABLET_DATA_TOMBSTONED (TS ac4ac7370c8c4b0c967fa990af77a067 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:57072
I20250814 13:58:43.930498 16293 tablet_replica.cc:331] stopping tablet replica
I20250814 13:58:43.931120 16293 raft_consensus.cc:2241] T 24df6cb14940420ea85c32a197b4fbc6 P ac4ac7370c8c4b0c967fa990af77a067 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250814 13:58:43.931558 16293 raft_consensus.cc:2270] T 24df6cb14940420ea85c32a197b4fbc6 P ac4ac7370c8c4b0c967fa990af77a067 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250814 13:58:43.933593 16293 ts_tablet_manager.cc:1905] T 24df6cb14940420ea85c32a197b4fbc6 P ac4ac7370c8c4b0c967fa990af77a067: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250814 13:58:43.942807 16293 ts_tablet_manager.cc:1918] T 24df6cb14940420ea85c32a197b4fbc6 P ac4ac7370c8c4b0c967fa990af77a067: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.11
I20250814 13:58:43.943058 16293 log.cc:1199] T 24df6cb14940420ea85c32a197b4fbc6 P ac4ac7370c8c4b0c967fa990af77a067: Deleting WAL directory at /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal/wals/24df6cb14940420ea85c32a197b4fbc6
I20250814 13:58:43.944236 15767 catalog_manager.cc:4928] TS ac4ac7370c8c4b0c967fa990af77a067 (127.11.132.131:44563): tablet 24df6cb14940420ea85c32a197b4fbc6 (table TestTable [id=5601a1f0afa44323a253ee6eb0ee8bc8]) successfully deleted
W20250814 13:58:44.550311 11794 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 24df6cb14940420ea85c32a197b4fbc6: tablet_id: "24df6cb14940420ea85c32a197b4fbc6" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250814 13:58:45.554468 11794 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 24df6cb14940420ea85c32a197b4fbc6: tablet_id: "24df6cb14940420ea85c32a197b4fbc6" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250814 13:58:46.557977 11794 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 24df6cb14940420ea85c32a197b4fbc6: tablet_id: "24df6cb14940420ea85c32a197b4fbc6" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250814 13:58:47.561272 11794 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 24df6cb14940420ea85c32a197b4fbc6: tablet_id: "24df6cb14940420ea85c32a197b4fbc6" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250814 13:58:48.564540 11794 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 24df6cb14940420ea85c32a197b4fbc6: tablet_id: "24df6cb14940420ea85c32a197b4fbc6" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250814 13:58:49.567655 11794 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 24df6cb14940420ea85c32a197b4fbc6: tablet_id: "24df6cb14940420ea85c32a197b4fbc6" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250814 13:58:50.571736 11794 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 24df6cb14940420ea85c32a197b4fbc6: tablet_id: "24df6cb14940420ea85c32a197b4fbc6" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250814 13:58:51.575224 11794 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 24df6cb14940420ea85c32a197b4fbc6: tablet_id: "24df6cb14940420ea85c32a197b4fbc6" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250814 13:58:52.578713 11794 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 24df6cb14940420ea85c32a197b4fbc6: tablet_id: "24df6cb14940420ea85c32a197b4fbc6" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250814 13:58:53.581928 11794 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 24df6cb14940420ea85c32a197b4fbc6: tablet_id: "24df6cb14940420ea85c32a197b4fbc6" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250814 13:58:54.585419 11794 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 24df6cb14940420ea85c32a197b4fbc6: tablet_id: "24df6cb14940420ea85c32a197b4fbc6" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250814 13:58:55.588928 11794 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 24df6cb14940420ea85c32a197b4fbc6: tablet_id: "24df6cb14940420ea85c32a197b4fbc6" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250814 13:58:56.592337 11794 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 24df6cb14940420ea85c32a197b4fbc6: tablet_id: "24df6cb14940420ea85c32a197b4fbc6" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250814 13:58:57.595646 11794 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 24df6cb14940420ea85c32a197b4fbc6: tablet_id: "24df6cb14940420ea85c32a197b4fbc6" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250814 13:58:58.598724 11794 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 24df6cb14940420ea85c32a197b4fbc6: tablet_id: "24df6cb14940420ea85c32a197b4fbc6" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250814 13:58:59.601902 11794 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 24df6cb14940420ea85c32a197b4fbc6: tablet_id: "24df6cb14940420ea85c32a197b4fbc6" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250814 13:59:00.605618 11794 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 24df6cb14940420ea85c32a197b4fbc6: tablet_id: "24df6cb14940420ea85c32a197b4fbc6" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250814 13:59:01.609076 11794 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 24df6cb14940420ea85c32a197b4fbc6: tablet_id: "24df6cb14940420ea85c32a197b4fbc6" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250814 13:59:02.612285 11794 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 24df6cb14940420ea85c32a197b4fbc6: tablet_id: "24df6cb14940420ea85c32a197b4fbc6" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/kudu-admin-test.cc:3914: Failure
Failed
Bad status: Not found: not all replicas of tablets comprising table TestTable are registered yet
I20250814 13:59:03.615269 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 15817
I20250814 13:59:03.640309 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 15969
I20250814 13:59:03.663805 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 16119
I20250814 13:59:03.688608 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 15747
2025-08-14T13:59:03Z chronyd exiting
I20250814 13:59:03.732236 11794 test_util.cc:183] -----------------------------------------------
I20250814 13:59:03.732440 11794 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755179819681315-11794-0
[ FAILED ] AdminCliTest.TestRebuildTables (58141 ms)
[----------] 5 tests from AdminCliTest (123984 ms total)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest
[ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4
I20250814 13:59:03.736039 11794 test_util.cc:276] Using random seed: -1391447340
I20250814 13:59:03.739951 11794 ts_itest-base.cc:115] Starting cluster with:
I20250814 13:59:03.740084 11794 ts_itest-base.cc:116] --------------
I20250814 13:59:03.740244 11794 ts_itest-base.cc:117] 5 tablet servers
I20250814 13:59:03.740379 11794 ts_itest-base.cc:118] 3 replicas per TS
I20250814 13:59:03.740513 11794 ts_itest-base.cc:119] --------------
2025-08-14T13:59:03Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-14T13:59:03Z Disabled control of system clock
I20250814 13:59:03.778226 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.190:45291
--webserver_interface=127.11.132.190
--webserver_port=0
--builtin_ntp_servers=127.11.132.148:43813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.132.190:45291
--raft_prepare_replacement_before_eviction=true with env {}
W20250814 13:59:04.058704 16313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:59:04.059237 16313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:59:04.059661 16313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:59:04.087641 16313 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250814 13:59:04.087961 16313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250814 13:59:04.088189 16313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:59:04.088403 16313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250814 13:59:04.088614 16313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250814 13:59:04.118772 16313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:43813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.132.190:45291
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.190:45291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.11.132.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:59:04.119982 16313 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:59:04.121490 16313 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:59:04.131040 16319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:59:04.131582 16320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:59:05.230847 16322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:59:05.233915 16313 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.103s user 0.343s sys 0.750s
W20250814 13:59:05.234259 16313 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.103s user 0.343s sys 0.750s
W20250814 13:59:05.234815 16321 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1096 milliseconds
I20250814 13:59:05.234877 16313 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 13:59:05.236003 16313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:59:05.238463 16313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:59:05.239802 16313 hybrid_clock.cc:648] HybridClock initialized: now 1755179945239766 us; error 44 us; skew 500 ppm
I20250814 13:59:05.240528 16313 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:59:05.247689 16313 webserver.cc:480] Webserver started at http://127.11.132.190:33687/ using document root <none> and password file <none>
I20250814 13:59:05.248566 16313 fs_manager.cc:362] Metadata directory not provided
I20250814 13:59:05.248767 16313 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:59:05.249234 16313 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 13:59:05.253080 16313 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "28fb79496d8a49b6825fbf79dd63c728"
format_stamp: "Formatted at 2025-08-14 13:59:05 on dist-test-slave-q1tc"
I20250814 13:59:05.254323 16313 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "28fb79496d8a49b6825fbf79dd63c728"
format_stamp: "Formatted at 2025-08-14 13:59:05 on dist-test-slave-q1tc"
I20250814 13:59:05.261523 16313 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20250814 13:59:05.267012 16329 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:59:05.268033 16313 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250814 13:59:05.268307 16313 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
uuid: "28fb79496d8a49b6825fbf79dd63c728"
format_stamp: "Formatted at 2025-08-14 13:59:05 on dist-test-slave-q1tc"
I20250814 13:59:05.268599 16313 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:59:05.347856 16313 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:59:05.349426 16313 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:59:05.349857 16313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:59:05.417498 16313 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.190:45291
I20250814 13:59:05.417567 16380 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.190:45291 every 8 connection(s)
I20250814 13:59:05.420029 16313 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250814 13:59:05.422901 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 16313
I20250814 13:59:05.423476 11794 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250814 13:59:05.425423 16381 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:59:05.448127 16381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28fb79496d8a49b6825fbf79dd63c728: Bootstrap starting.
I20250814 13:59:05.454437 16381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28fb79496d8a49b6825fbf79dd63c728: Neither blocks nor log segments found. Creating new log.
I20250814 13:59:05.456262 16381 log.cc:826] T 00000000000000000000000000000000 P 28fb79496d8a49b6825fbf79dd63c728: Log is configured to *not* fsync() on all Append() calls
I20250814 13:59:05.460206 16381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28fb79496d8a49b6825fbf79dd63c728: No bootstrap required, opened a new log
I20250814 13:59:05.476845 16381 raft_consensus.cc:357] T 00000000000000000000000000000000 P 28fb79496d8a49b6825fbf79dd63c728 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28fb79496d8a49b6825fbf79dd63c728" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 45291 } }
I20250814 13:59:05.477429 16381 raft_consensus.cc:383] T 00000000000000000000000000000000 P 28fb79496d8a49b6825fbf79dd63c728 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 13:59:05.477615 16381 raft_consensus.cc:738] T 00000000000000000000000000000000 P 28fb79496d8a49b6825fbf79dd63c728 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28fb79496d8a49b6825fbf79dd63c728, State: Initialized, Role: FOLLOWER
I20250814 13:59:05.478161 16381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28fb79496d8a49b6825fbf79dd63c728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28fb79496d8a49b6825fbf79dd63c728" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 45291 } }
I20250814 13:59:05.478574 16381 raft_consensus.cc:397] T 00000000000000000000000000000000 P 28fb79496d8a49b6825fbf79dd63c728 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 13:59:05.478786 16381 raft_consensus.cc:491] T 00000000000000000000000000000000 P 28fb79496d8a49b6825fbf79dd63c728 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 13:59:05.479023 16381 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 28fb79496d8a49b6825fbf79dd63c728 [term 0 FOLLOWER]: Advancing to term 1
I20250814 13:59:05.483196 16381 raft_consensus.cc:513] T 00000000000000000000000000000000 P 28fb79496d8a49b6825fbf79dd63c728 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28fb79496d8a49b6825fbf79dd63c728" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 45291 } }
I20250814 13:59:05.483794 16381 leader_election.cc:304] T 00000000000000000000000000000000 P 28fb79496d8a49b6825fbf79dd63c728 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28fb79496d8a49b6825fbf79dd63c728; no voters:
I20250814 13:59:05.485399 16381 leader_election.cc:290] T 00000000000000000000000000000000 P 28fb79496d8a49b6825fbf79dd63c728 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250814 13:59:05.486111 16386 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 28fb79496d8a49b6825fbf79dd63c728 [term 1 FOLLOWER]: Leader election won for term 1
I20250814 13:59:05.488170 16386 raft_consensus.cc:695] T 00000000000000000000000000000000 P 28fb79496d8a49b6825fbf79dd63c728 [term 1 LEADER]: Becoming Leader. State: Replica: 28fb79496d8a49b6825fbf79dd63c728, State: Running, Role: LEADER
I20250814 13:59:05.489013 16386 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28fb79496d8a49b6825fbf79dd63c728 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28fb79496d8a49b6825fbf79dd63c728" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 45291 } }
I20250814 13:59:05.490149 16381 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28fb79496d8a49b6825fbf79dd63c728 [sys.catalog]: configured and running, proceeding with master startup.
I20250814 13:59:05.496165 16388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28fb79496d8a49b6825fbf79dd63c728 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28fb79496d8a49b6825fbf79dd63c728" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28fb79496d8a49b6825fbf79dd63c728" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 45291 } } }
I20250814 13:59:05.496963 16388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28fb79496d8a49b6825fbf79dd63c728 [sys.catalog]: This master's current role is: LEADER
I20250814 13:59:05.501204 16387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28fb79496d8a49b6825fbf79dd63c728 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28fb79496d8a49b6825fbf79dd63c728. Latest consensus state: current_term: 1 leader_uuid: "28fb79496d8a49b6825fbf79dd63c728" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28fb79496d8a49b6825fbf79dd63c728" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 45291 } } }
I20250814 13:59:05.502427 16387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28fb79496d8a49b6825fbf79dd63c728 [sys.catalog]: This master's current role is: LEADER
I20250814 13:59:05.504776 16394 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250814 13:59:05.515565 16394 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250814 13:59:05.529518 16394 catalog_manager.cc:1349] Generated new cluster ID: cb26f351280b4b8083d6d73271ccc6b0
I20250814 13:59:05.529778 16394 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250814 13:59:05.549459 16394 catalog_manager.cc:1372] Generated new certificate authority record
I20250814 13:59:05.550916 16394 catalog_manager.cc:1506] Loading token signing keys...
I20250814 13:59:05.566915 16394 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 28fb79496d8a49b6825fbf79dd63c728: Generated new TSK 0
I20250814 13:59:05.567672 16394 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250814 13:59:05.588088 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.132.129:0
--local_ip_for_outbound_sockets=127.11.132.129
--webserver_interface=127.11.132.129
--webserver_port=0
--tserver_master_addrs=127.11.132.190:45291
--builtin_ntp_servers=127.11.132.148:43813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250814 13:59:05.869938 16405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:59:05.870398 16405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:59:05.870836 16405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:59:05.898398 16405 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250814 13:59:05.898820 16405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:59:05.899606 16405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.132.129
I20250814 13:59:05.930943 16405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:43813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.132.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.11.132.129
--webserver_port=0
--tserver_master_addrs=127.11.132.190:45291
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.132.129
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:59:05.932077 16405 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:59:05.933531 16405 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:59:05.944864 16411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:59:05.946377 16412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:59:07.348516 16410 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 16405
W20250814 13:59:07.781839 16410 kernel_stack_watchdog.cc:198] Thread 16405 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250814 13:59:07.782394 16405 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.837s user 0.628s sys 1.176s
W20250814 13:59:07.782789 16405 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.837s user 0.628s sys 1.176s
W20250814 13:59:07.783914 16414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:59:07.786909 16413 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1836 milliseconds
I20250814 13:59:07.786972 16405 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 13:59:07.788084 16405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:59:07.790584 16405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:59:07.792004 16405 hybrid_clock.cc:648] HybridClock initialized: now 1755179947791947 us; error 50 us; skew 500 ppm
I20250814 13:59:07.792752 16405 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:59:07.799088 16405 webserver.cc:480] Webserver started at http://127.11.132.129:38035/ using document root <none> and password file <none>
I20250814 13:59:07.799898 16405 fs_manager.cc:362] Metadata directory not provided
I20250814 13:59:07.800088 16405 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:59:07.800495 16405 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 13:59:07.806164 16405 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "9ccce75b11cc4de0bf834185c617d49f"
format_stamp: "Formatted at 2025-08-14 13:59:07 on dist-test-slave-q1tc"
I20250814 13:59:07.807204 16405 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "9ccce75b11cc4de0bf834185c617d49f"
format_stamp: "Formatted at 2025-08-14 13:59:07 on dist-test-slave-q1tc"
I20250814 13:59:07.813949 16405 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20250814 13:59:07.819376 16421 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:59:07.820418 16405 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250814 13:59:07.820695 16405 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "9ccce75b11cc4de0bf834185c617d49f"
format_stamp: "Formatted at 2025-08-14 13:59:07 on dist-test-slave-q1tc"
I20250814 13:59:07.820973 16405 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:59:07.869400 16405 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:59:07.870760 16405 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:59:07.871153 16405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:59:07.874015 16405 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 13:59:07.878147 16405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 13:59:07.878343 16405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 13:59:07.878561 16405 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 13:59:07.878701 16405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 13:59:08.039366 16405 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.129:33485
I20250814 13:59:08.039541 16533 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.129:33485 every 8 connection(s)
I20250814 13:59:08.041790 16405 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250814 13:59:08.042588 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 16405
I20250814 13:59:08.043059 11794 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250814 13:59:08.051981 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.132.130:0
--local_ip_for_outbound_sockets=127.11.132.130
--webserver_interface=127.11.132.130
--webserver_port=0
--tserver_master_addrs=127.11.132.190:45291
--builtin_ntp_servers=127.11.132.148:43813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250814 13:59:08.083195 16534 heartbeater.cc:344] Connected to a master server at 127.11.132.190:45291
I20250814 13:59:08.083745 16534 heartbeater.cc:461] Registering TS with master...
I20250814 13:59:08.085222 16534 heartbeater.cc:507] Master 127.11.132.190:45291 requested a full tablet report, sending...
I20250814 13:59:08.088362 16346 ts_manager.cc:194] Registered new tserver with Master: 9ccce75b11cc4de0bf834185c617d49f (127.11.132.129:33485)
I20250814 13:59:08.090941 16346 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.132.129:35373
W20250814 13:59:08.350656 16538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:59:08.351104 16538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:59:08.351593 16538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:59:08.380086 16538 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250814 13:59:08.380438 16538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:59:08.381245 16538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.132.130
I20250814 13:59:08.412935 16538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:43813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.132.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.11.132.130
--webserver_port=0
--tserver_master_addrs=127.11.132.190:45291
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.132.130
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:59:08.414165 16538 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:59:08.415854 16538 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:59:08.428086 16544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 13:59:09.094691 16534 heartbeater.cc:499] Master 127.11.132.190:45291 was elected leader, sending a full tablet report...
W20250814 13:59:09.831085 16543 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 16538
W20250814 13:59:09.847435 16543 kernel_stack_watchdog.cc:198] Thread 16538 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 399ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250814 13:59:08.428537 16545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:59:09.848181 16538 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.419s user 0.001s sys 0.001s
W20250814 13:59:09.848510 16538 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.420s user 0.001s sys 0.001s
W20250814 13:59:09.848925 16546 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1419 milliseconds
W20250814 13:59:09.850114 16548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 13:59:09.850060 16538 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 13:59:09.852975 16538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:59:09.855545 16538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:59:09.856942 16538 hybrid_clock.cc:648] HybridClock initialized: now 1755179949856907 us; error 45 us; skew 500 ppm
I20250814 13:59:09.857831 16538 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:59:09.863801 16538 webserver.cc:480] Webserver started at http://127.11.132.130:44297/ using document root <none> and password file <none>
I20250814 13:59:09.864697 16538 fs_manager.cc:362] Metadata directory not provided
I20250814 13:59:09.864902 16538 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:59:09.865391 16538 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 13:59:09.869712 16538 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "4a259efff20b4211b1f493536e4a0daa"
format_stamp: "Formatted at 2025-08-14 13:59:09 on dist-test-slave-q1tc"
I20250814 13:59:09.870733 16538 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "4a259efff20b4211b1f493536e4a0daa"
format_stamp: "Formatted at 2025-08-14 13:59:09 on dist-test-slave-q1tc"
I20250814 13:59:09.877578 16538 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20250814 13:59:09.882593 16554 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:59:09.883402 16538 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.004s
I20250814 13:59:09.883666 16538 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "4a259efff20b4211b1f493536e4a0daa"
format_stamp: "Formatted at 2025-08-14 13:59:09 on dist-test-slave-q1tc"
I20250814 13:59:09.883997 16538 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:59:09.938946 16538 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:59:09.940248 16538 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:59:09.940644 16538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:59:09.942871 16538 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 13:59:09.946780 16538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 13:59:09.946964 16538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 13:59:09.947176 16538 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 13:59:09.947324 16538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 13:59:10.093775 16538 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.130:45829
I20250814 13:59:10.093894 16667 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.130:45829 every 8 connection(s)
I20250814 13:59:10.096268 16538 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250814 13:59:10.104771 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 16538
I20250814 13:59:10.105361 11794 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250814 13:59:10.112031 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.132.131:0
--local_ip_for_outbound_sockets=127.11.132.131
--webserver_interface=127.11.132.131
--webserver_port=0
--tserver_master_addrs=127.11.132.190:45291
--builtin_ntp_servers=127.11.132.148:43813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250814 13:59:10.116456 16668 heartbeater.cc:344] Connected to a master server at 127.11.132.190:45291
I20250814 13:59:10.116824 16668 heartbeater.cc:461] Registering TS with master...
I20250814 13:59:10.117802 16668 heartbeater.cc:507] Master 127.11.132.190:45291 requested a full tablet report, sending...
I20250814 13:59:10.119663 16346 ts_manager.cc:194] Registered new tserver with Master: 4a259efff20b4211b1f493536e4a0daa (127.11.132.130:45829)
I20250814 13:59:10.120729 16346 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.132.130:48463
W20250814 13:59:10.393474 16672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:59:10.393941 16672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:59:10.394428 16672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:59:10.424010 16672 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250814 13:59:10.424407 16672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:59:10.425202 16672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.132.131
I20250814 13:59:10.457625 16672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:43813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.132.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.11.132.131
--webserver_port=0
--tserver_master_addrs=127.11.132.190:45291
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.132.131
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:59:10.458829 16672 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:59:10.460229 16672 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:59:10.471132 16678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 13:59:11.123425 16668 heartbeater.cc:499] Master 127.11.132.190:45291 was elected leader, sending a full tablet report...
W20250814 13:59:11.875845 16677 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 16672
W20250814 13:59:10.471735 16679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:59:12.069468 16681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:59:12.066694 16672 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.595s user 0.576s sys 1.015s
W20250814 13:59:12.070348 16672 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.599s user 0.577s sys 1.015s
W20250814 13:59:12.070684 16680 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1597 milliseconds
I20250814 13:59:12.070619 16672 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 13:59:12.074644 16672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:59:12.076656 16672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:59:12.077983 16672 hybrid_clock.cc:648] HybridClock initialized: now 1755179952077950 us; error 33 us; skew 500 ppm
I20250814 13:59:12.078703 16672 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:59:12.084273 16672 webserver.cc:480] Webserver started at http://127.11.132.131:34407/ using document root <none> and password file <none>
I20250814 13:59:12.085198 16672 fs_manager.cc:362] Metadata directory not provided
I20250814 13:59:12.085415 16672 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:59:12.085827 16672 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 13:59:12.089613 16672 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "79d62f6b438e42a79538ea17ade318fb"
format_stamp: "Formatted at 2025-08-14 13:59:12 on dist-test-slave-q1tc"
I20250814 13:59:12.090595 16672 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "79d62f6b438e42a79538ea17ade318fb"
format_stamp: "Formatted at 2025-08-14 13:59:12 on dist-test-slave-q1tc"
I20250814 13:59:12.097435 16672 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.002s
I20250814 13:59:12.102563 16688 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:59:12.103461 16672 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250814 13:59:12.103734 16672 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "79d62f6b438e42a79538ea17ade318fb"
format_stamp: "Formatted at 2025-08-14 13:59:12 on dist-test-slave-q1tc"
I20250814 13:59:12.104018 16672 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:59:12.153708 16672 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:59:12.155033 16672 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:59:12.155491 16672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:59:12.157794 16672 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 13:59:12.161254 16672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 13:59:12.161441 16672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 13:59:12.161654 16672 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 13:59:12.161814 16672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 13:59:12.293570 16672 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.131:43661
I20250814 13:59:12.293668 16800 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.131:43661 every 8 connection(s)
I20250814 13:59:12.296351 16672 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250814 13:59:12.305379 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 16672
I20250814 13:59:12.305787 11794 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250814 13:59:12.311952 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.132.132:0
--local_ip_for_outbound_sockets=127.11.132.132
--webserver_interface=127.11.132.132
--webserver_port=0
--tserver_master_addrs=127.11.132.190:45291
--builtin_ntp_servers=127.11.132.148:43813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250814 13:59:12.335127 16801 heartbeater.cc:344] Connected to a master server at 127.11.132.190:45291
I20250814 13:59:12.335567 16801 heartbeater.cc:461] Registering TS with master...
I20250814 13:59:12.336637 16801 heartbeater.cc:507] Master 127.11.132.190:45291 requested a full tablet report, sending...
I20250814 13:59:12.338619 16346 ts_manager.cc:194] Registered new tserver with Master: 79d62f6b438e42a79538ea17ade318fb (127.11.132.131:43661)
I20250814 13:59:12.339776 16346 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.132.131:59633
W20250814 13:59:12.592142 16805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:59:12.592650 16805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:59:12.593137 16805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:59:12.622694 16805 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250814 13:59:12.623101 16805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:59:12.623874 16805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.132.132
I20250814 13:59:12.655352 16805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:43813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.132.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.11.132.132
--webserver_port=0
--tserver_master_addrs=127.11.132.190:45291
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.132.132
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:59:12.656541 16805 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:59:12.658025 16805 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:59:12.668434 16811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 13:59:13.342645 16801 heartbeater.cc:499] Master 127.11.132.190:45291 was elected leader, sending a full tablet report...
W20250814 13:59:12.671653 16812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:59:13.786123 16814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:59:13.788371 16813 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1113 milliseconds
I20250814 13:59:13.788466 16805 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 13:59:13.789676 16805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:59:13.792441 16805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:59:13.793884 16805 hybrid_clock.cc:648] HybridClock initialized: now 1755179953793854 us; error 42 us; skew 500 ppm
I20250814 13:59:13.794624 16805 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:59:13.800722 16805 webserver.cc:480] Webserver started at http://127.11.132.132:36457/ using document root <none> and password file <none>
I20250814 13:59:13.801604 16805 fs_manager.cc:362] Metadata directory not provided
I20250814 13:59:13.801786 16805 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:59:13.802263 16805 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 13:59:13.806615 16805 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "31630b89ca744c9d97aba939876a95c7"
format_stamp: "Formatted at 2025-08-14 13:59:13 on dist-test-slave-q1tc"
I20250814 13:59:13.807637 16805 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "31630b89ca744c9d97aba939876a95c7"
format_stamp: "Formatted at 2025-08-14 13:59:13 on dist-test-slave-q1tc"
I20250814 13:59:13.814040 16805 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250814 13:59:13.819348 16821 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:59:13.820279 16805 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20250814 13:59:13.820570 16805 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "31630b89ca744c9d97aba939876a95c7"
format_stamp: "Formatted at 2025-08-14 13:59:13 on dist-test-slave-q1tc"
I20250814 13:59:13.820865 16805 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:59:13.876021 16805 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:59:13.877439 16805 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:59:13.877826 16805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:59:13.880409 16805 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 13:59:13.884168 16805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 13:59:13.884400 16805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 13:59:13.884646 16805 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 13:59:13.884796 16805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 13:59:14.007370 16805 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.132:34107
I20250814 13:59:14.007470 16933 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.132:34107 every 8 connection(s)
I20250814 13:59:14.009747 16805 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250814 13:59:14.020027 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 16805
I20250814 13:59:14.020481 11794 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250814 13:59:14.027690 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.132.133:0
--local_ip_for_outbound_sockets=127.11.132.133
--webserver_interface=127.11.132.133
--webserver_port=0
--tserver_master_addrs=127.11.132.190:45291
--builtin_ntp_servers=127.11.132.148:43813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250814 13:59:14.031318 16934 heartbeater.cc:344] Connected to a master server at 127.11.132.190:45291
I20250814 13:59:14.031819 16934 heartbeater.cc:461] Registering TS with master...
I20250814 13:59:14.033187 16934 heartbeater.cc:507] Master 127.11.132.190:45291 requested a full tablet report, sending...
I20250814 13:59:14.035480 16346 ts_manager.cc:194] Registered new tserver with Master: 31630b89ca744c9d97aba939876a95c7 (127.11.132.132:34107)
I20250814 13:59:14.036818 16346 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.132.132:36977
W20250814 13:59:14.319449 16938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:59:14.319907 16938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:59:14.320444 16938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:59:14.351020 16938 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250814 13:59:14.351428 16938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:59:14.352145 16938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.132.133
I20250814 13:59:14.383879 16938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:43813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.132.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.11.132.133
--webserver_port=0
--tserver_master_addrs=127.11.132.190:45291
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.132.133
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:59:14.385164 16938 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:59:14.386657 16938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:59:14.398386 16945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 13:59:15.055183 16934 heartbeater.cc:499] Master 127.11.132.190:45291 was elected leader, sending a full tablet report...
W20250814 13:59:14.399377 16946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:59:15.537209 16948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:59:15.539233 16947 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1136 milliseconds
I20250814 13:59:15.539312 16938 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 13:59:15.540355 16938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:59:15.542785 16938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:59:15.544167 16938 hybrid_clock.cc:648] HybridClock initialized: now 1755179955544123 us; error 45 us; skew 500 ppm
I20250814 13:59:15.544940 16938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:59:15.550916 16938 webserver.cc:480] Webserver started at http://127.11.132.133:35777/ using document root <none> and password file <none>
I20250814 13:59:15.551832 16938 fs_manager.cc:362] Metadata directory not provided
I20250814 13:59:15.552044 16938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:59:15.552477 16938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 13:59:15.556651 16938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "81ba3c82836e48048a21416cc7f47d49"
format_stamp: "Formatted at 2025-08-14 13:59:15 on dist-test-slave-q1tc"
I20250814 13:59:15.557657 16938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "81ba3c82836e48048a21416cc7f47d49"
format_stamp: "Formatted at 2025-08-14 13:59:15 on dist-test-slave-q1tc"
I20250814 13:59:15.564129 16938 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250814 13:59:15.569480 16955 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:59:15.570308 16938 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250814 13:59:15.570576 16938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "81ba3c82836e48048a21416cc7f47d49"
format_stamp: "Formatted at 2025-08-14 13:59:15 on dist-test-slave-q1tc"
I20250814 13:59:15.570847 16938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:59:15.617753 16938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:59:15.619074 16938 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:59:15.619462 16938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:59:15.621899 16938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 13:59:15.625659 16938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 13:59:15.625837 16938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 13:59:15.626106 16938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 13:59:15.626260 16938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 13:59:15.754607 16938 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.133:40731
I20250814 13:59:15.754760 17067 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.133:40731 every 8 connection(s)
I20250814 13:59:15.757018 16938 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250814 13:59:15.765422 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 16938
I20250814 13:59:15.765829 11794 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250814 13:59:15.776295 17068 heartbeater.cc:344] Connected to a master server at 127.11.132.190:45291
I20250814 13:59:15.776659 17068 heartbeater.cc:461] Registering TS with master...
I20250814 13:59:15.777617 17068 heartbeater.cc:507] Master 127.11.132.190:45291 requested a full tablet report, sending...
I20250814 13:59:15.779675 16346 ts_manager.cc:194] Registered new tserver with Master: 81ba3c82836e48048a21416cc7f47d49 (127.11.132.133:40731)
I20250814 13:59:15.780970 16346 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.132.133:52697
I20250814 13:59:15.785811 11794 external_mini_cluster.cc:949] 5 TS(s) registered with all masters
I20250814 13:59:15.818549 16346 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36058:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250814 13:59:15.892792 16602 tablet_service.cc:1468] Processing CreateTablet for tablet 44fc3f9557be4e7ea1194fa63afa2d18 (DEFAULT_TABLE table=TestTable [id=fc05c622aa6145248d52dda757770436]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 13:59:15.894553 16602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44fc3f9557be4e7ea1194fa63afa2d18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:59:15.895556 17003 tablet_service.cc:1468] Processing CreateTablet for tablet 44fc3f9557be4e7ea1194fa63afa2d18 (DEFAULT_TABLE table=TestTable [id=fc05c622aa6145248d52dda757770436]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 13:59:15.896070 16469 tablet_service.cc:1468] Processing CreateTablet for tablet 44fc3f9557be4e7ea1194fa63afa2d18 (DEFAULT_TABLE table=TestTable [id=fc05c622aa6145248d52dda757770436]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 13:59:15.897261 17003 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44fc3f9557be4e7ea1194fa63afa2d18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:59:15.897969 16469 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44fc3f9557be4e7ea1194fa63afa2d18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:59:15.918731 17088 tablet_bootstrap.cc:492] T 44fc3f9557be4e7ea1194fa63afa2d18 P 9ccce75b11cc4de0bf834185c617d49f: Bootstrap starting.
I20250814 13:59:15.922576 17089 tablet_bootstrap.cc:492] T 44fc3f9557be4e7ea1194fa63afa2d18 P 4a259efff20b4211b1f493536e4a0daa: Bootstrap starting.
I20250814 13:59:15.926532 17090 tablet_bootstrap.cc:492] T 44fc3f9557be4e7ea1194fa63afa2d18 P 81ba3c82836e48048a21416cc7f47d49: Bootstrap starting.
I20250814 13:59:15.928241 17088 tablet_bootstrap.cc:654] T 44fc3f9557be4e7ea1194fa63afa2d18 P 9ccce75b11cc4de0bf834185c617d49f: Neither blocks nor log segments found. Creating new log.
I20250814 13:59:15.929752 17089 tablet_bootstrap.cc:654] T 44fc3f9557be4e7ea1194fa63afa2d18 P 4a259efff20b4211b1f493536e4a0daa: Neither blocks nor log segments found. Creating new log.
I20250814 13:59:15.930645 17088 log.cc:826] T 44fc3f9557be4e7ea1194fa63afa2d18 P 9ccce75b11cc4de0bf834185c617d49f: Log is configured to *not* fsync() on all Append() calls
I20250814 13:59:15.931473 17090 tablet_bootstrap.cc:654] T 44fc3f9557be4e7ea1194fa63afa2d18 P 81ba3c82836e48048a21416cc7f47d49: Neither blocks nor log segments found. Creating new log.
I20250814 13:59:15.931918 17089 log.cc:826] T 44fc3f9557be4e7ea1194fa63afa2d18 P 4a259efff20b4211b1f493536e4a0daa: Log is configured to *not* fsync() on all Append() calls
I20250814 13:59:15.934219 17090 log.cc:826] T 44fc3f9557be4e7ea1194fa63afa2d18 P 81ba3c82836e48048a21416cc7f47d49: Log is configured to *not* fsync() on all Append() calls
I20250814 13:59:15.936353 17088 tablet_bootstrap.cc:492] T 44fc3f9557be4e7ea1194fa63afa2d18 P 9ccce75b11cc4de0bf834185c617d49f: No bootstrap required, opened a new log
I20250814 13:59:15.936954 17088 ts_tablet_manager.cc:1397] T 44fc3f9557be4e7ea1194fa63afa2d18 P 9ccce75b11cc4de0bf834185c617d49f: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.002s
I20250814 13:59:15.938488 17089 tablet_bootstrap.cc:492] T 44fc3f9557be4e7ea1194fa63afa2d18 P 4a259efff20b4211b1f493536e4a0daa: No bootstrap required, opened a new log
I20250814 13:59:15.938825 17089 ts_tablet_manager.cc:1397] T 44fc3f9557be4e7ea1194fa63afa2d18 P 4a259efff20b4211b1f493536e4a0daa: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.000s
I20250814 13:59:15.939415 17090 tablet_bootstrap.cc:492] T 44fc3f9557be4e7ea1194fa63afa2d18 P 81ba3c82836e48048a21416cc7f47d49: No bootstrap required, opened a new log
I20250814 13:59:15.939842 17090 ts_tablet_manager.cc:1397] T 44fc3f9557be4e7ea1194fa63afa2d18 P 81ba3c82836e48048a21416cc7f47d49: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.010s
I20250814 13:59:15.955390 17089 raft_consensus.cc:357] T 44fc3f9557be4e7ea1194fa63afa2d18 P 4a259efff20b4211b1f493536e4a0daa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ba3c82836e48048a21416cc7f47d49" member_type: VOTER last_known_addr { host: "127.11.132.133" port: 40731 } } peers { permanent_uuid: "4a259efff20b4211b1f493536e4a0daa" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 45829 } } peers { permanent_uuid: "9ccce75b11cc4de0bf834185c617d49f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 33485 } }
I20250814 13:59:15.956094 17089 raft_consensus.cc:383] T 44fc3f9557be4e7ea1194fa63afa2d18 P 4a259efff20b4211b1f493536e4a0daa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 13:59:15.956367 17089 raft_consensus.cc:738] T 44fc3f9557be4e7ea1194fa63afa2d18 P 4a259efff20b4211b1f493536e4a0daa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a259efff20b4211b1f493536e4a0daa, State: Initialized, Role: FOLLOWER
I20250814 13:59:15.957093 17089 consensus_queue.cc:260] T 44fc3f9557be4e7ea1194fa63afa2d18 P 4a259efff20b4211b1f493536e4a0daa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ba3c82836e48048a21416cc7f47d49" member_type: VOTER last_known_addr { host: "127.11.132.133" port: 40731 } } peers { permanent_uuid: "4a259efff20b4211b1f493536e4a0daa" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 45829 } } peers { permanent_uuid: "9ccce75b11cc4de0bf834185c617d49f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 33485 } }
I20250814 13:59:15.961935 17089 ts_tablet_manager.cc:1428] T 44fc3f9557be4e7ea1194fa63afa2d18 P 4a259efff20b4211b1f493536e4a0daa: Time spent starting tablet: real 0.023s user 0.018s sys 0.004s
I20250814 13:59:15.962280 17088 raft_consensus.cc:357] T 44fc3f9557be4e7ea1194fa63afa2d18 P 9ccce75b11cc4de0bf834185c617d49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ba3c82836e48048a21416cc7f47d49" member_type: VOTER last_known_addr { host: "127.11.132.133" port: 40731 } } peers { permanent_uuid: "4a259efff20b4211b1f493536e4a0daa" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 45829 } } peers { permanent_uuid: "9ccce75b11cc4de0bf834185c617d49f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 33485 } }
I20250814 13:59:15.963204 17088 raft_consensus.cc:383] T 44fc3f9557be4e7ea1194fa63afa2d18 P 9ccce75b11cc4de0bf834185c617d49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 13:59:15.963493 17088 raft_consensus.cc:738] T 44fc3f9557be4e7ea1194fa63afa2d18 P 9ccce75b11cc4de0bf834185c617d49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ccce75b11cc4de0bf834185c617d49f, State: Initialized, Role: FOLLOWER
I20250814 13:59:15.964322 17088 consensus_queue.cc:260] T 44fc3f9557be4e7ea1194fa63afa2d18 P 9ccce75b11cc4de0bf834185c617d49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ba3c82836e48048a21416cc7f47d49" member_type: VOTER last_known_addr { host: "127.11.132.133" port: 40731 } } peers { permanent_uuid: "4a259efff20b4211b1f493536e4a0daa" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 45829 } } peers { permanent_uuid: "9ccce75b11cc4de0bf834185c617d49f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 33485 } }
I20250814 13:59:15.964712 17090 raft_consensus.cc:357] T 44fc3f9557be4e7ea1194fa63afa2d18 P 81ba3c82836e48048a21416cc7f47d49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ba3c82836e48048a21416cc7f47d49" member_type: VOTER last_known_addr { host: "127.11.132.133" port: 40731 } } peers { permanent_uuid: "4a259efff20b4211b1f493536e4a0daa" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 45829 } } peers { permanent_uuid: "9ccce75b11cc4de0bf834185c617d49f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 33485 } }
I20250814 13:59:15.965607 17090 raft_consensus.cc:383] T 44fc3f9557be4e7ea1194fa63afa2d18 P 81ba3c82836e48048a21416cc7f47d49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 13:59:15.966017 17090 raft_consensus.cc:738] T 44fc3f9557be4e7ea1194fa63afa2d18 P 81ba3c82836e48048a21416cc7f47d49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81ba3c82836e48048a21416cc7f47d49, State: Initialized, Role: FOLLOWER
I20250814 13:59:15.966981 17090 consensus_queue.cc:260] T 44fc3f9557be4e7ea1194fa63afa2d18 P 81ba3c82836e48048a21416cc7f47d49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ba3c82836e48048a21416cc7f47d49" member_type: VOTER last_known_addr { host: "127.11.132.133" port: 40731 } } peers { permanent_uuid: "4a259efff20b4211b1f493536e4a0daa" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 45829 } } peers { permanent_uuid: "9ccce75b11cc4de0bf834185c617d49f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 33485 } }
I20250814 13:59:15.970559 17068 heartbeater.cc:499] Master 127.11.132.190:45291 was elected leader, sending a full tablet report...
I20250814 13:59:15.971779 17090 ts_tablet_manager.cc:1428] T 44fc3f9557be4e7ea1194fa63afa2d18 P 81ba3c82836e48048a21416cc7f47d49: Time spent starting tablet: real 0.032s user 0.025s sys 0.007s
I20250814 13:59:15.973109 17088 ts_tablet_manager.cc:1428] T 44fc3f9557be4e7ea1194fa63afa2d18 P 9ccce75b11cc4de0bf834185c617d49f: Time spent starting tablet: real 0.036s user 0.024s sys 0.010s
W20250814 13:59:16.011224 17069 tablet.cc:2378] T 44fc3f9557be4e7ea1194fa63afa2d18 P 81ba3c82836e48048a21416cc7f47d49: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250814 13:59:16.078298 16535 tablet.cc:2378] T 44fc3f9557be4e7ea1194fa63afa2d18 P 9ccce75b11cc4de0bf834185c617d49f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250814 13:59:16.107710 16669 tablet.cc:2378] T 44fc3f9557be4e7ea1194fa63afa2d18 P 4a259efff20b4211b1f493536e4a0daa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250814 13:59:16.133600 17094 raft_consensus.cc:491] T 44fc3f9557be4e7ea1194fa63afa2d18 P 4a259efff20b4211b1f493536e4a0daa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250814 13:59:16.134065 17094 raft_consensus.cc:513] T 44fc3f9557be4e7ea1194fa63afa2d18 P 4a259efff20b4211b1f493536e4a0daa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ba3c82836e48048a21416cc7f47d49" member_type: VOTER last_known_addr { host: "127.11.132.133" port: 40731 } } peers { permanent_uuid: "4a259efff20b4211b1f493536e4a0daa" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 45829 } } peers { permanent_uuid: "9ccce75b11cc4de0bf834185c617d49f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 33485 } }
I20250814 13:59:16.136308 17094 leader_election.cc:290] T 44fc3f9557be4e7ea1194fa63afa2d18 P 4a259efff20b4211b1f493536e4a0daa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81ba3c82836e48048a21416cc7f47d49 (127.11.132.133:40731), 9ccce75b11cc4de0bf834185c617d49f (127.11.132.129:33485)
I20250814 13:59:16.149173 17023 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "44fc3f9557be4e7ea1194fa63afa2d18" candidate_uuid: "4a259efff20b4211b1f493536e4a0daa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81ba3c82836e48048a21416cc7f47d49" is_pre_election: true
I20250814 13:59:16.149302 16489 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "44fc3f9557be4e7ea1194fa63afa2d18" candidate_uuid: "4a259efff20b4211b1f493536e4a0daa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ccce75b11cc4de0bf834185c617d49f" is_pre_election: true
I20250814 13:59:16.149792 17023 raft_consensus.cc:2466] T 44fc3f9557be4e7ea1194fa63afa2d18 P 81ba3c82836e48048a21416cc7f47d49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a259efff20b4211b1f493536e4a0daa in term 0.
I20250814 13:59:16.149930 16489 raft_consensus.cc:2466] T 44fc3f9557be4e7ea1194fa63afa2d18 P 9ccce75b11cc4de0bf834185c617d49f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a259efff20b4211b1f493536e4a0daa in term 0.
I20250814 13:59:16.150857 16556 leader_election.cc:304] T 44fc3f9557be4e7ea1194fa63afa2d18 P 4a259efff20b4211b1f493536e4a0daa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a259efff20b4211b1f493536e4a0daa, 81ba3c82836e48048a21416cc7f47d49; no voters:
I20250814 13:59:16.151495 17094 raft_consensus.cc:2802] T 44fc3f9557be4e7ea1194fa63afa2d18 P 4a259efff20b4211b1f493536e4a0daa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250814 13:59:16.151827 17094 raft_consensus.cc:491] T 44fc3f9557be4e7ea1194fa63afa2d18 P 4a259efff20b4211b1f493536e4a0daa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250814 13:59:16.152107 17094 raft_consensus.cc:3058] T 44fc3f9557be4e7ea1194fa63afa2d18 P 4a259efff20b4211b1f493536e4a0daa [term 0 FOLLOWER]: Advancing to term 1
I20250814 13:59:16.156342 17094 raft_consensus.cc:513] T 44fc3f9557be4e7ea1194fa63afa2d18 P 4a259efff20b4211b1f493536e4a0daa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ba3c82836e48048a21416cc7f47d49" member_type: VOTER last_known_addr { host: "127.11.132.133" port: 40731 } } peers { permanent_uuid: "4a259efff20b4211b1f493536e4a0daa" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 45829 } } peers { permanent_uuid: "9ccce75b11cc4de0bf834185c617d49f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 33485 } }
I20250814 13:59:16.157508 17094 leader_election.cc:290] T 44fc3f9557be4e7ea1194fa63afa2d18 P 4a259efff20b4211b1f493536e4a0daa [CANDIDATE]: Term 1 election: Requested vote from peers 81ba3c82836e48048a21416cc7f47d49 (127.11.132.133:40731), 9ccce75b11cc4de0bf834185c617d49f (127.11.132.129:33485)
I20250814 13:59:16.158051 17023 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "44fc3f9557be4e7ea1194fa63afa2d18" candidate_uuid: "4a259efff20b4211b1f493536e4a0daa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81ba3c82836e48048a21416cc7f47d49"
I20250814 13:59:16.158393 17023 raft_consensus.cc:3058] T 44fc3f9557be4e7ea1194fa63afa2d18 P 81ba3c82836e48048a21416cc7f47d49 [term 0 FOLLOWER]: Advancing to term 1
I20250814 13:59:16.158318 16489 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "44fc3f9557be4e7ea1194fa63afa2d18" candidate_uuid: "4a259efff20b4211b1f493536e4a0daa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ccce75b11cc4de0bf834185c617d49f"
I20250814 13:59:16.158668 16489 raft_consensus.cc:3058] T 44fc3f9557be4e7ea1194fa63afa2d18 P 9ccce75b11cc4de0bf834185c617d49f [term 0 FOLLOWER]: Advancing to term 1
I20250814 13:59:16.162573 17023 raft_consensus.cc:2466] T 44fc3f9557be4e7ea1194fa63afa2d18 P 81ba3c82836e48048a21416cc7f47d49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a259efff20b4211b1f493536e4a0daa in term 1.
I20250814 13:59:16.162843 16489 raft_consensus.cc:2466] T 44fc3f9557be4e7ea1194fa63afa2d18 P 9ccce75b11cc4de0bf834185c617d49f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a259efff20b4211b1f493536e4a0daa in term 1.
I20250814 13:59:16.163333 16556 leader_election.cc:304] T 44fc3f9557be4e7ea1194fa63afa2d18 P 4a259efff20b4211b1f493536e4a0daa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a259efff20b4211b1f493536e4a0daa, 81ba3c82836e48048a21416cc7f47d49; no voters:
I20250814 13:59:16.163849 17094 raft_consensus.cc:2802] T 44fc3f9557be4e7ea1194fa63afa2d18 P 4a259efff20b4211b1f493536e4a0daa [term 1 FOLLOWER]: Leader election won for term 1
I20250814 13:59:16.165354 17094 raft_consensus.cc:695] T 44fc3f9557be4e7ea1194fa63afa2d18 P 4a259efff20b4211b1f493536e4a0daa [term 1 LEADER]: Becoming Leader. State: Replica: 4a259efff20b4211b1f493536e4a0daa, State: Running, Role: LEADER
I20250814 13:59:16.166157 17094 consensus_queue.cc:237] T 44fc3f9557be4e7ea1194fa63afa2d18 P 4a259efff20b4211b1f493536e4a0daa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ba3c82836e48048a21416cc7f47d49" member_type: VOTER last_known_addr { host: "127.11.132.133" port: 40731 } } peers { permanent_uuid: "4a259efff20b4211b1f493536e4a0daa" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 45829 } } peers { permanent_uuid: "9ccce75b11cc4de0bf834185c617d49f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 33485 } }
I20250814 13:59:16.175685 16346 catalog_manager.cc:5582] T 44fc3f9557be4e7ea1194fa63afa2d18 P 4a259efff20b4211b1f493536e4a0daa reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a259efff20b4211b1f493536e4a0daa (127.11.132.130). New cstate: current_term: 1 leader_uuid: "4a259efff20b4211b1f493536e4a0daa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ba3c82836e48048a21416cc7f47d49" member_type: VOTER last_known_addr { host: "127.11.132.133" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a259efff20b4211b1f493536e4a0daa" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ccce75b11cc4de0bf834185c617d49f" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 33485 } health_report { overall_health: UNKNOWN } } }
I20250814 13:59:16.193835 11794 external_mini_cluster.cc:949] 5 TS(s) registered with all masters
I20250814 13:59:16.196810 11794 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 9ccce75b11cc4de0bf834185c617d49f to finish bootstrapping
I20250814 13:59:16.209638 11794 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 4a259efff20b4211b1f493536e4a0daa to finish bootstrapping
I20250814 13:59:16.219024 11794 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 81ba3c82836e48048a21416cc7f47d49 to finish bootstrapping
I20250814 13:59:16.229893 11794 test_util.cc:276] Using random seed: -1378953486
I20250814 13:59:16.252787 11794 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20250814 13:59:16.253628 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 16405
W20250814 13:59:16.283842 16558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.129:33485: connect: Connection refused (error 111)
I20250814 13:59:16.285600 17023 raft_consensus.cc:1273] T 44fc3f9557be4e7ea1194fa63afa2d18 P 81ba3c82836e48048a21416cc7f47d49 [term 1 FOLLOWER]: Refusing update from remote peer 4a259efff20b4211b1f493536e4a0daa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250814 13:59:16.287396 17094 consensus_queue.cc:1035] T 44fc3f9557be4e7ea1194fa63afa2d18 P 4a259efff20b4211b1f493536e4a0daa [LEADER]: Connected to new peer: Peer: permanent_uuid: "81ba3c82836e48048a21416cc7f47d49" member_type: VOTER last_known_addr { host: "127.11.132.133" port: 40731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250814 13:59:16.293270 16558 consensus_peers.cc:489] T 44fc3f9557be4e7ea1194fa63afa2d18 P 4a259efff20b4211b1f493536e4a0daa -> Peer 9ccce75b11cc4de0bf834185c617d49f (127.11.132.129:33485): Couldn't send request to peer 9ccce75b11cc4de0bf834185c617d49f. Status: Network error: Client connection negotiation failed: client connection to 127.11.132.129:33485: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250814 13:59:16.312525 17109 mvcc.cc:204] Tried to move back new op lower bound from 7189217100922695680 to 7189217100463083520. Current Snapshot: MvccSnapshot[applied={T|T < 7189217100922695680}]
I20250814 13:59:18.810197 16602 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250814 13:59:18.813544 16869 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250814 13:59:18.854132 16736 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250814 13:59:18.880095 17003 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250814 13:59:18.934293 16558 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.132.129:33485, user_credentials={real_user=slave}} blocked reactor thread for 44585.1us
W20250814 13:59:18.959857 16558 consensus_peers.cc:489] T 44fc3f9557be4e7ea1194fa63afa2d18 P 4a259efff20b4211b1f493536e4a0daa -> Peer 9ccce75b11cc4de0bf834185c617d49f (127.11.132.129:33485): Couldn't send request to peer 9ccce75b11cc4de0bf834185c617d49f. Status: Network error: Client connection negotiation failed: client connection to 127.11.132.129:33485: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250814 13:59:20.581126 16869 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250814 13:59:20.600996 16602 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250814 13:59:20.615995 16736 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250814 13:59:20.643591 17003 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250814 13:59:21.548352 16558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.129:33485: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250814 13:59:21.550483 16558 consensus_peers.cc:489] T 44fc3f9557be4e7ea1194fa63afa2d18 P 4a259efff20b4211b1f493536e4a0daa -> Peer 9ccce75b11cc4de0bf834185c617d49f (127.11.132.129:33485): Couldn't send request to peer 9ccce75b11cc4de0bf834185c617d49f. Status: Network error: Client connection negotiation failed: client connection to 127.11.132.129:33485: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250814 13:59:22.153512 17064 debug-util.cc:398] Leaking SignalData structure 0x7b08000d53e0 after lost signal to thread 16940
W20250814 13:59:22.154426 17064 debug-util.cc:398] Leaking SignalData structure 0x7b08000e8760 after lost signal to thread 17067
I20250814 13:59:23.049643 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 16538
I20250814 13:59:23.091946 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 16672
I20250814 13:59:23.121398 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 16805
I20250814 13:59:23.150719 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 16938
I20250814 13:59:23.185786 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 16313
2025-08-14T13:59:23Z chronyd exiting
[ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4 (19508 ms)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest (19509 ms total)
[----------] 1 test from ListTableCliSimpleParamTest
[ RUN ] ListTableCliSimpleParamTest.TestListTables/2
I20250814 13:59:23.244969 11794 test_util.cc:276] Using random seed: -1371938415
I20250814 13:59:23.248821 11794 ts_itest-base.cc:115] Starting cluster with:
I20250814 13:59:23.248958 11794 ts_itest-base.cc:116] --------------
I20250814 13:59:23.249107 11794 ts_itest-base.cc:117] 1 tablet servers
I20250814 13:59:23.249249 11794 ts_itest-base.cc:118] 1 replicas per TS
I20250814 13:59:23.249349 11794 ts_itest-base.cc:119] --------------
2025-08-14T13:59:23Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-14T13:59:23Z Disabled control of system clock
I20250814 13:59:23.288290 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.190:40113
--webserver_interface=127.11.132.190
--webserver_port=0
--builtin_ntp_servers=127.11.132.148:42299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.132.190:40113 with env {}
W20250814 13:59:23.572662 17213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:59:23.573252 17213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:59:23.573652 17213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:59:23.602553 17213 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250814 13:59:23.602806 17213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:59:23.603005 17213 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250814 13:59:23.603193 17213 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250814 13:59:23.636188 17213 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:42299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.132.190:40113
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.190:40113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.11.132.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:59:23.637399 17213 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:59:23.638924 17213 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:59:23.649713 17219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:59:23.650239 17220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:59:24.860544 17222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:59:24.863268 17221 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1211 milliseconds
W20250814 13:59:24.863943 17213 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.214s user 0.423s sys 0.786s
W20250814 13:59:24.864171 17213 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.215s user 0.423s sys 0.786s
I20250814 13:59:24.864393 17213 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 13:59:24.865491 17213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:59:24.869539 17213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:59:24.870862 17213 hybrid_clock.cc:648] HybridClock initialized: now 1755179964870834 us; error 46 us; skew 500 ppm
I20250814 13:59:24.871599 17213 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:59:24.878378 17213 webserver.cc:480] Webserver started at http://127.11.132.190:35837/ using document root <none> and password file <none>
I20250814 13:59:24.879182 17213 fs_manager.cc:362] Metadata directory not provided
I20250814 13:59:24.879379 17213 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:59:24.879777 17213 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 13:59:24.883803 17213 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "039b1bc255d04bfc93681081b02e0a1d"
format_stamp: "Formatted at 2025-08-14 13:59:24 on dist-test-slave-q1tc"
I20250814 13:59:24.884804 17213 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "039b1bc255d04bfc93681081b02e0a1d"
format_stamp: "Formatted at 2025-08-14 13:59:24 on dist-test-slave-q1tc"
I20250814 13:59:24.891971 17213 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20250814 13:59:24.897543 17229 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:59:24.898492 17213 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250814 13:59:24.898758 17213 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
uuid: "039b1bc255d04bfc93681081b02e0a1d"
format_stamp: "Formatted at 2025-08-14 13:59:24 on dist-test-slave-q1tc"
I20250814 13:59:24.899044 17213 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:59:24.970880 17213 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:59:24.972075 17213 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:59:24.972435 17213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:59:25.036677 17213 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.190:40113
I20250814 13:59:25.036748 17280 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.190:40113 every 8 connection(s)
I20250814 13:59:25.039187 17213 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250814 13:59:25.043749 17281 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:59:25.049616 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 17213
I20250814 13:59:25.049971 11794 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755179819681315-11794-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250814 13:59:25.060521 17281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 039b1bc255d04bfc93681081b02e0a1d: Bootstrap starting.
I20250814 13:59:25.066638 17281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 039b1bc255d04bfc93681081b02e0a1d: Neither blocks nor log segments found. Creating new log.
I20250814 13:59:25.068066 17281 log.cc:826] T 00000000000000000000000000000000 P 039b1bc255d04bfc93681081b02e0a1d: Log is configured to *not* fsync() on all Append() calls
I20250814 13:59:25.072618 17281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 039b1bc255d04bfc93681081b02e0a1d: No bootstrap required, opened a new log
I20250814 13:59:25.088682 17281 raft_consensus.cc:357] T 00000000000000000000000000000000 P 039b1bc255d04bfc93681081b02e0a1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "039b1bc255d04bfc93681081b02e0a1d" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 40113 } }
I20250814 13:59:25.089286 17281 raft_consensus.cc:383] T 00000000000000000000000000000000 P 039b1bc255d04bfc93681081b02e0a1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 13:59:25.089527 17281 raft_consensus.cc:738] T 00000000000000000000000000000000 P 039b1bc255d04bfc93681081b02e0a1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039b1bc255d04bfc93681081b02e0a1d, State: Initialized, Role: FOLLOWER
I20250814 13:59:25.090122 17281 consensus_queue.cc:260] T 00000000000000000000000000000000 P 039b1bc255d04bfc93681081b02e0a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "039b1bc255d04bfc93681081b02e0a1d" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 40113 } }
I20250814 13:59:25.090668 17281 raft_consensus.cc:397] T 00000000000000000000000000000000 P 039b1bc255d04bfc93681081b02e0a1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 13:59:25.090927 17281 raft_consensus.cc:491] T 00000000000000000000000000000000 P 039b1bc255d04bfc93681081b02e0a1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 13:59:25.091284 17281 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 039b1bc255d04bfc93681081b02e0a1d [term 0 FOLLOWER]: Advancing to term 1
I20250814 13:59:25.094975 17281 raft_consensus.cc:513] T 00000000000000000000000000000000 P 039b1bc255d04bfc93681081b02e0a1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "039b1bc255d04bfc93681081b02e0a1d" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 40113 } }
I20250814 13:59:25.095526 17281 leader_election.cc:304] T 00000000000000000000000000000000 P 039b1bc255d04bfc93681081b02e0a1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 039b1bc255d04bfc93681081b02e0a1d; no voters:
I20250814 13:59:25.097093 17281 leader_election.cc:290] T 00000000000000000000000000000000 P 039b1bc255d04bfc93681081b02e0a1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250814 13:59:25.097810 17286 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 039b1bc255d04bfc93681081b02e0a1d [term 1 FOLLOWER]: Leader election won for term 1
I20250814 13:59:25.100102 17286 raft_consensus.cc:695] T 00000000000000000000000000000000 P 039b1bc255d04bfc93681081b02e0a1d [term 1 LEADER]: Becoming Leader. State: Replica: 039b1bc255d04bfc93681081b02e0a1d, State: Running, Role: LEADER
I20250814 13:59:25.100869 17281 sys_catalog.cc:564] T 00000000000000000000000000000000 P 039b1bc255d04bfc93681081b02e0a1d [sys.catalog]: configured and running, proceeding with master startup.
I20250814 13:59:25.100807 17286 consensus_queue.cc:237] T 00000000000000000000000000000000 P 039b1bc255d04bfc93681081b02e0a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "039b1bc255d04bfc93681081b02e0a1d" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 40113 } }
I20250814 13:59:25.107744 17287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 039b1bc255d04bfc93681081b02e0a1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "039b1bc255d04bfc93681081b02e0a1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "039b1bc255d04bfc93681081b02e0a1d" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 40113 } } }
I20250814 13:59:25.107645 17288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 039b1bc255d04bfc93681081b02e0a1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 039b1bc255d04bfc93681081b02e0a1d. Latest consensus state: current_term: 1 leader_uuid: "039b1bc255d04bfc93681081b02e0a1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "039b1bc255d04bfc93681081b02e0a1d" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 40113 } } }
I20250814 13:59:25.108361 17287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 039b1bc255d04bfc93681081b02e0a1d [sys.catalog]: This master's current role is: LEADER
I20250814 13:59:25.108456 17288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 039b1bc255d04bfc93681081b02e0a1d [sys.catalog]: This master's current role is: LEADER
I20250814 13:59:25.113344 17292 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250814 13:59:25.124137 17292 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250814 13:59:25.138931 17292 catalog_manager.cc:1349] Generated new cluster ID: 18efc41aba2a47eeb5193b09de9f9a3c
I20250814 13:59:25.139127 17292 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250814 13:59:25.169499 17292 catalog_manager.cc:1372] Generated new certificate authority record
I20250814 13:59:25.171110 17292 catalog_manager.cc:1506] Loading token signing keys...
I20250814 13:59:25.184110 17292 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 039b1bc255d04bfc93681081b02e0a1d: Generated new TSK 0
I20250814 13:59:25.184769 17292 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250814 13:59:25.191919 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.132.129:0
--local_ip_for_outbound_sockets=127.11.132.129
--webserver_interface=127.11.132.129
--webserver_port=0
--tserver_master_addrs=127.11.132.190:40113
--builtin_ntp_servers=127.11.132.148:42299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250814 13:59:25.474663 17305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:59:25.475117 17305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:59:25.475540 17305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:59:25.503697 17305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:59:25.504436 17305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.132.129
I20250814 13:59:25.535509 17305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:42299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.132.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.11.132.129
--webserver_port=0
--tserver_master_addrs=127.11.132.190:40113
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.132.129
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:59:25.536739 17305 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:59:25.538211 17305 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:59:25.549633 17311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:59:25.551024 17312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:59:26.951381 17314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:59:26.954151 17305 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.404s user 0.451s sys 0.932s
W20250814 13:59:26.954577 17305 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.404s user 0.452s sys 0.933s
W20250814 13:59:26.955176 17313 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1399 milliseconds
I20250814 13:59:26.955263 17305 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 13:59:26.956900 17305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:59:26.961609 17305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:59:26.963100 17305 hybrid_clock.cc:648] HybridClock initialized: now 1755179966963019 us; error 78 us; skew 500 ppm
I20250814 13:59:26.964210 17305 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:59:26.973441 17305 webserver.cc:480] Webserver started at http://127.11.132.129:41735/ using document root <none> and password file <none>
I20250814 13:59:26.974591 17305 fs_manager.cc:362] Metadata directory not provided
I20250814 13:59:26.974823 17305 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:59:26.975337 17305 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 13:59:26.981477 17305 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "444d16b0b98c4c9582de026d71e18775"
format_stamp: "Formatted at 2025-08-14 13:59:26 on dist-test-slave-q1tc"
I20250814 13:59:26.982761 17305 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "444d16b0b98c4c9582de026d71e18775"
format_stamp: "Formatted at 2025-08-14 13:59:26 on dist-test-slave-q1tc"
I20250814 13:59:26.991703 17305 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.005s sys 0.005s
I20250814 13:59:26.998723 17321 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:59:26.999954 17305 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250814 13:59:27.000289 17305 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "444d16b0b98c4c9582de026d71e18775"
format_stamp: "Formatted at 2025-08-14 13:59:26 on dist-test-slave-q1tc"
I20250814 13:59:27.000672 17305 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:59:27.076714 17305 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:59:27.078109 17305 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:59:27.078508 17305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:59:27.081310 17305 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 13:59:27.085448 17305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 13:59:27.085625 17305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 13:59:27.085878 17305 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 13:59:27.086050 17305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 13:59:27.254267 17305 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.129:36319
I20250814 13:59:27.254364 17433 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.129:36319 every 8 connection(s)
I20250814 13:59:27.256935 17305 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250814 13:59:27.265431 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 17305
I20250814 13:59:27.265843 11794 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755179819681315-11794-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250814 13:59:27.280326 17434 heartbeater.cc:344] Connected to a master server at 127.11.132.190:40113
I20250814 13:59:27.280781 17434 heartbeater.cc:461] Registering TS with master...
I20250814 13:59:27.281965 17434 heartbeater.cc:507] Master 127.11.132.190:40113 requested a full tablet report, sending...
I20250814 13:59:27.284449 17246 ts_manager.cc:194] Registered new tserver with Master: 444d16b0b98c4c9582de026d71e18775 (127.11.132.129:36319)
I20250814 13:59:27.286332 17246 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.132.129:48885
I20250814 13:59:27.297216 11794 external_mini_cluster.cc:949] 1 TS(s) registered with all masters
I20250814 13:59:27.330054 17246 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43650:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250814 13:59:27.388914 17369 tablet_service.cc:1468] Processing CreateTablet for tablet 00305cdd44a74168b8835ac38913aefc (DEFAULT_TABLE table=TestTable [id=26c709fa67bb4ec3b1e6459aeabab5df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 13:59:27.390228 17369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00305cdd44a74168b8835ac38913aefc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:59:27.410956 17449 tablet_bootstrap.cc:492] T 00305cdd44a74168b8835ac38913aefc P 444d16b0b98c4c9582de026d71e18775: Bootstrap starting.
I20250814 13:59:27.416254 17449 tablet_bootstrap.cc:654] T 00305cdd44a74168b8835ac38913aefc P 444d16b0b98c4c9582de026d71e18775: Neither blocks nor log segments found. Creating new log.
I20250814 13:59:27.418000 17449 log.cc:826] T 00305cdd44a74168b8835ac38913aefc P 444d16b0b98c4c9582de026d71e18775: Log is configured to *not* fsync() on all Append() calls
I20250814 13:59:27.422170 17449 tablet_bootstrap.cc:492] T 00305cdd44a74168b8835ac38913aefc P 444d16b0b98c4c9582de026d71e18775: No bootstrap required, opened a new log
I20250814 13:59:27.422502 17449 ts_tablet_manager.cc:1397] T 00305cdd44a74168b8835ac38913aefc P 444d16b0b98c4c9582de026d71e18775: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.006s
I20250814 13:59:27.438643 17449 raft_consensus.cc:357] T 00305cdd44a74168b8835ac38913aefc P 444d16b0b98c4c9582de026d71e18775 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "444d16b0b98c4c9582de026d71e18775" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 36319 } }
I20250814 13:59:27.439109 17449 raft_consensus.cc:383] T 00305cdd44a74168b8835ac38913aefc P 444d16b0b98c4c9582de026d71e18775 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 13:59:27.439352 17449 raft_consensus.cc:738] T 00305cdd44a74168b8835ac38913aefc P 444d16b0b98c4c9582de026d71e18775 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 444d16b0b98c4c9582de026d71e18775, State: Initialized, Role: FOLLOWER
I20250814 13:59:27.439978 17449 consensus_queue.cc:260] T 00305cdd44a74168b8835ac38913aefc P 444d16b0b98c4c9582de026d71e18775 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "444d16b0b98c4c9582de026d71e18775" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 36319 } }
I20250814 13:59:27.440454 17449 raft_consensus.cc:397] T 00305cdd44a74168b8835ac38913aefc P 444d16b0b98c4c9582de026d71e18775 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 13:59:27.440675 17449 raft_consensus.cc:491] T 00305cdd44a74168b8835ac38913aefc P 444d16b0b98c4c9582de026d71e18775 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 13:59:27.440935 17449 raft_consensus.cc:3058] T 00305cdd44a74168b8835ac38913aefc P 444d16b0b98c4c9582de026d71e18775 [term 0 FOLLOWER]: Advancing to term 1
I20250814 13:59:27.445556 17449 raft_consensus.cc:513] T 00305cdd44a74168b8835ac38913aefc P 444d16b0b98c4c9582de026d71e18775 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "444d16b0b98c4c9582de026d71e18775" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 36319 } }
I20250814 13:59:27.446254 17449 leader_election.cc:304] T 00305cdd44a74168b8835ac38913aefc P 444d16b0b98c4c9582de026d71e18775 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 444d16b0b98c4c9582de026d71e18775; no voters:
I20250814 13:59:27.448287 17449 leader_election.cc:290] T 00305cdd44a74168b8835ac38913aefc P 444d16b0b98c4c9582de026d71e18775 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250814 13:59:27.448638 17451 raft_consensus.cc:2802] T 00305cdd44a74168b8835ac38913aefc P 444d16b0b98c4c9582de026d71e18775 [term 1 FOLLOWER]: Leader election won for term 1
I20250814 13:59:27.451574 17434 heartbeater.cc:499] Master 127.11.132.190:40113 was elected leader, sending a full tablet report...
I20250814 13:59:27.452857 17449 ts_tablet_manager.cc:1428] T 00305cdd44a74168b8835ac38913aefc P 444d16b0b98c4c9582de026d71e18775: Time spent starting tablet: real 0.030s user 0.027s sys 0.003s
I20250814 13:59:27.453490 17451 raft_consensus.cc:695] T 00305cdd44a74168b8835ac38913aefc P 444d16b0b98c4c9582de026d71e18775 [term 1 LEADER]: Becoming Leader. State: Replica: 444d16b0b98c4c9582de026d71e18775, State: Running, Role: LEADER
I20250814 13:59:27.454077 17451 consensus_queue.cc:237] T 00305cdd44a74168b8835ac38913aefc P 444d16b0b98c4c9582de026d71e18775 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "444d16b0b98c4c9582de026d71e18775" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 36319 } }
I20250814 13:59:27.465879 17245 catalog_manager.cc:5582] T 00305cdd44a74168b8835ac38913aefc P 444d16b0b98c4c9582de026d71e18775 reported cstate change: term changed from 0 to 1, leader changed from <none> to 444d16b0b98c4c9582de026d71e18775 (127.11.132.129). New cstate: current_term: 1 leader_uuid: "444d16b0b98c4c9582de026d71e18775" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "444d16b0b98c4c9582de026d71e18775" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 36319 } health_report { overall_health: HEALTHY } } }
I20250814 13:59:27.489820 11794 external_mini_cluster.cc:949] 1 TS(s) registered with all masters
I20250814 13:59:27.492769 11794 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 444d16b0b98c4c9582de026d71e18775 to finish bootstrapping
I20250814 13:59:30.116210 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 17305
I20250814 13:59:30.140383 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 17213
2025-08-14T13:59:30Z chronyd exiting
[ OK ] ListTableCliSimpleParamTest.TestListTables/2 (6957 ms)
[----------] 1 test from ListTableCliSimpleParamTest (6957 ms total)
[----------] 1 test from ListTableCliParamTest
[ RUN ] ListTableCliParamTest.ListTabletWithPartitionInfo/4
I20250814 13:59:30.203871 11794 test_util.cc:276] Using random seed: -1364979521
[ OK ] ListTableCliParamTest.ListTabletWithPartitionInfo/4 (18 ms)
[----------] 1 test from ListTableCliParamTest (18 ms total)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest
[ RUN ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0
2025-08-14T13:59:30Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-14T13:59:30Z Disabled control of system clock
I20250814 13:59:30.260928 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.190:46139
--webserver_interface=127.11.132.190
--webserver_port=0
--builtin_ntp_servers=127.11.132.148:37633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.132.190:46139 with env {}
W20250814 13:59:30.541440 17478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:59:30.541950 17478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:59:30.542398 17478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:59:30.571048 17478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250814 13:59:30.571350 17478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:59:30.571591 17478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250814 13:59:30.571825 17478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250814 13:59:30.604772 17478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:37633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.132.190:46139
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.190:46139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.132.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:59:30.606036 17478 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:59:30.607568 17478 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:59:30.618284 17484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:59:30.618988 17485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:59:31.760927 17487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:59:31.763286 17486 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1140 milliseconds
I20250814 13:59:31.763414 17478 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 13:59:31.764526 17478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:59:31.767683 17478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:59:31.769094 17478 hybrid_clock.cc:648] HybridClock initialized: now 1755179971769038 us; error 30 us; skew 500 ppm
I20250814 13:59:31.769796 17478 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:59:31.775605 17478 webserver.cc:480] Webserver started at http://127.11.132.190:39569/ using document root <none> and password file <none>
I20250814 13:59:31.776424 17478 fs_manager.cc:362] Metadata directory not provided
I20250814 13:59:31.776625 17478 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:59:31.777078 17478 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 13:59:31.781167 17478 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/master-0/data/instance:
uuid: "7238765ca9c743a081144f2e6d472e11"
format_stamp: "Formatted at 2025-08-14 13:59:31 on dist-test-slave-q1tc"
I20250814 13:59:31.782135 17478 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/master-0/wal/instance:
uuid: "7238765ca9c743a081144f2e6d472e11"
format_stamp: "Formatted at 2025-08-14 13:59:31 on dist-test-slave-q1tc"
I20250814 13:59:31.788643 17478 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250814 13:59:31.793514 17494 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:59:31.794518 17478 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250814 13:59:31.794795 17478 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/master-0/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/master-0/wal
uuid: "7238765ca9c743a081144f2e6d472e11"
format_stamp: "Formatted at 2025-08-14 13:59:31 on dist-test-slave-q1tc"
I20250814 13:59:31.795078 17478 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:59:31.849542 17478 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:59:31.850894 17478 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:59:31.851311 17478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:59:31.917587 17478 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.190:46139
I20250814 13:59:31.917655 17545 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.190:46139 every 8 connection(s)
I20250814 13:59:31.920199 17478 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/master-0/data/info.pb
I20250814 13:59:31.924938 17546 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:59:31.929929 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 17478
I20250814 13:59:31.930330 11794 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/master-0/wal/instance
I20250814 13:59:31.942329 17546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7238765ca9c743a081144f2e6d472e11: Bootstrap starting.
I20250814 13:59:31.948872 17546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7238765ca9c743a081144f2e6d472e11: Neither blocks nor log segments found. Creating new log.
I20250814 13:59:31.950724 17546 log.cc:826] T 00000000000000000000000000000000 P 7238765ca9c743a081144f2e6d472e11: Log is configured to *not* fsync() on all Append() calls
I20250814 13:59:31.955111 17546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7238765ca9c743a081144f2e6d472e11: No bootstrap required, opened a new log
I20250814 13:59:31.971606 17546 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7238765ca9c743a081144f2e6d472e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7238765ca9c743a081144f2e6d472e11" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 46139 } }
I20250814 13:59:31.972368 17546 raft_consensus.cc:383] T 00000000000000000000000000000000 P 7238765ca9c743a081144f2e6d472e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 13:59:31.972599 17546 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7238765ca9c743a081144f2e6d472e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7238765ca9c743a081144f2e6d472e11, State: Initialized, Role: FOLLOWER
I20250814 13:59:31.973274 17546 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7238765ca9c743a081144f2e6d472e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7238765ca9c743a081144f2e6d472e11" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 46139 } }
I20250814 13:59:31.973728 17546 raft_consensus.cc:397] T 00000000000000000000000000000000 P 7238765ca9c743a081144f2e6d472e11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 13:59:31.973985 17546 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7238765ca9c743a081144f2e6d472e11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 13:59:31.974336 17546 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 7238765ca9c743a081144f2e6d472e11 [term 0 FOLLOWER]: Advancing to term 1
I20250814 13:59:31.978650 17546 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7238765ca9c743a081144f2e6d472e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7238765ca9c743a081144f2e6d472e11" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 46139 } }
I20250814 13:59:31.979239 17546 leader_election.cc:304] T 00000000000000000000000000000000 P 7238765ca9c743a081144f2e6d472e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7238765ca9c743a081144f2e6d472e11; no voters:
I20250814 13:59:31.980777 17546 leader_election.cc:290] T 00000000000000000000000000000000 P 7238765ca9c743a081144f2e6d472e11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250814 13:59:31.981527 17551 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 7238765ca9c743a081144f2e6d472e11 [term 1 FOLLOWER]: Leader election won for term 1
I20250814 13:59:31.983843 17551 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7238765ca9c743a081144f2e6d472e11 [term 1 LEADER]: Becoming Leader. State: Replica: 7238765ca9c743a081144f2e6d472e11, State: Running, Role: LEADER
I20250814 13:59:31.984407 17551 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7238765ca9c743a081144f2e6d472e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7238765ca9c743a081144f2e6d472e11" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 46139 } }
I20250814 13:59:31.984735 17546 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7238765ca9c743a081144f2e6d472e11 [sys.catalog]: configured and running, proceeding with master startup.
I20250814 13:59:31.996538 17552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7238765ca9c743a081144f2e6d472e11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7238765ca9c743a081144f2e6d472e11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7238765ca9c743a081144f2e6d472e11" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 46139 } } }
I20250814 13:59:31.997017 17552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7238765ca9c743a081144f2e6d472e11 [sys.catalog]: This master's current role is: LEADER
I20250814 13:59:31.998894 17553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7238765ca9c743a081144f2e6d472e11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7238765ca9c743a081144f2e6d472e11. Latest consensus state: current_term: 1 leader_uuid: "7238765ca9c743a081144f2e6d472e11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7238765ca9c743a081144f2e6d472e11" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 46139 } } }
I20250814 13:59:31.999528 17553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7238765ca9c743a081144f2e6d472e11 [sys.catalog]: This master's current role is: LEADER
I20250814 13:59:31.999760 17561 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250814 13:59:32.011348 17561 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250814 13:59:32.027864 17561 catalog_manager.cc:1349] Generated new cluster ID: 927c7e3a046045c8bf71ee23788bfcaa
I20250814 13:59:32.028111 17561 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250814 13:59:32.048147 17561 catalog_manager.cc:1372] Generated new certificate authority record
I20250814 13:59:32.049456 17561 catalog_manager.cc:1506] Loading token signing keys...
I20250814 13:59:32.059365 17561 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 7238765ca9c743a081144f2e6d472e11: Generated new TSK 0
I20250814 13:59:32.060277 17561 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250814 13:59:32.079602 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.132.129:0
--local_ip_for_outbound_sockets=127.11.132.129
--webserver_interface=127.11.132.129
--webserver_port=0
--tserver_master_addrs=127.11.132.190:46139
--builtin_ntp_servers=127.11.132.148:37633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20250814 13:59:32.375941 17570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:59:32.376474 17570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:59:32.376971 17570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:59:32.407130 17570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:59:32.407955 17570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.132.129
I20250814 13:59:32.440199 17570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:37633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.132.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.132.129
--webserver_port=0
--tserver_master_addrs=127.11.132.190:46139
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.132.129
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:59:32.441438 17570 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:59:32.442885 17570 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:59:32.455201 17576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:59:32.459251 17579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:59:32.457271 17577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:59:33.823647 17578 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1363 milliseconds
I20250814 13:59:33.823786 17570 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 13:59:33.824955 17570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:59:33.827680 17570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:59:33.829165 17570 hybrid_clock.cc:648] HybridClock initialized: now 1755179973829136 us; error 40 us; skew 500 ppm
I20250814 13:59:33.829910 17570 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:59:33.836228 17570 webserver.cc:480] Webserver started at http://127.11.132.129:38669/ using document root <none> and password file <none>
I20250814 13:59:33.837133 17570 fs_manager.cc:362] Metadata directory not provided
I20250814 13:59:33.837328 17570 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:59:33.837815 17570 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 13:59:33.842063 17570 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-0/data/instance:
uuid: "7eb2f9678de6476c9f17aad6dc2a25ca"
format_stamp: "Formatted at 2025-08-14 13:59:33 on dist-test-slave-q1tc"
I20250814 13:59:33.843093 17570 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-0/wal/instance:
uuid: "7eb2f9678de6476c9f17aad6dc2a25ca"
format_stamp: "Formatted at 2025-08-14 13:59:33 on dist-test-slave-q1tc"
I20250814 13:59:33.849815 17570 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250814 13:59:33.855361 17586 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:59:33.856333 17570 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250814 13:59:33.856638 17570 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-0/wal
uuid: "7eb2f9678de6476c9f17aad6dc2a25ca"
format_stamp: "Formatted at 2025-08-14 13:59:33 on dist-test-slave-q1tc"
I20250814 13:59:33.856932 17570 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:59:33.917439 17570 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:59:33.918919 17570 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:59:33.919322 17570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:59:33.922219 17570 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 13:59:33.926828 17570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 13:59:33.927057 17570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 13:59:33.927311 17570 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 13:59:33.927475 17570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 13:59:34.089542 17570 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.129:35349
I20250814 13:59:34.089694 17698 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.129:35349 every 8 connection(s)
I20250814 13:59:34.091835 17570 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-0/data/info.pb
I20250814 13:59:34.102247 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 17570
I20250814 13:59:34.102705 11794 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-0/wal/instance
I20250814 13:59:34.109193 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.132.130:0
--local_ip_for_outbound_sockets=127.11.132.130
--webserver_interface=127.11.132.130
--webserver_port=0
--tserver_master_addrs=127.11.132.190:46139
--builtin_ntp_servers=127.11.132.148:37633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250814 13:59:34.113461 17699 heartbeater.cc:344] Connected to a master server at 127.11.132.190:46139
I20250814 13:59:34.113847 17699 heartbeater.cc:461] Registering TS with master...
I20250814 13:59:34.114729 17699 heartbeater.cc:507] Master 127.11.132.190:46139 requested a full tablet report, sending...
I20250814 13:59:34.117341 17511 ts_manager.cc:194] Registered new tserver with Master: 7eb2f9678de6476c9f17aad6dc2a25ca (127.11.132.129:35349)
I20250814 13:59:34.119936 17511 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.132.129:50019
W20250814 13:59:34.403667 17703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:59:34.404163 17703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:59:34.404716 17703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:59:34.434644 17703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:59:34.435462 17703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.132.130
I20250814 13:59:34.468596 17703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:37633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.132.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.132.130
--webserver_port=0
--tserver_master_addrs=127.11.132.190:46139
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.132.130
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:59:34.469882 17703 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:59:34.471357 17703 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:59:34.482306 17709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 13:59:35.123097 17699 heartbeater.cc:499] Master 127.11.132.190:46139 was elected leader, sending a full tablet report...
W20250814 13:59:34.483143 17710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:59:35.603299 17712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:59:35.607390 17711 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1123 milliseconds
W20250814 13:59:35.607746 17703 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.125s user 0.412s sys 0.710s
W20250814 13:59:35.608043 17703 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.126s user 0.412s sys 0.710s
I20250814 13:59:35.608198 17703 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 13:59:35.609174 17703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:59:35.611166 17703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:59:35.612475 17703 hybrid_clock.cc:648] HybridClock initialized: now 1755179975612428 us; error 56 us; skew 500 ppm
I20250814 13:59:35.613245 17703 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:59:35.620211 17703 webserver.cc:480] Webserver started at http://127.11.132.130:40215/ using document root <none> and password file <none>
I20250814 13:59:35.621421 17703 fs_manager.cc:362] Metadata directory not provided
I20250814 13:59:35.621637 17703 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:59:35.622044 17703 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 13:59:35.626142 17703 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-1/data/instance:
uuid: "274c7e6393224516bcafe58490c5987e"
format_stamp: "Formatted at 2025-08-14 13:59:35 on dist-test-slave-q1tc"
I20250814 13:59:35.627244 17703 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-1/wal/instance:
uuid: "274c7e6393224516bcafe58490c5987e"
format_stamp: "Formatted at 2025-08-14 13:59:35 on dist-test-slave-q1tc"
I20250814 13:59:35.634310 17703 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20250814 13:59:35.640015 17719 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:59:35.641049 17703 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20250814 13:59:35.641425 17703 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-1/wal
uuid: "274c7e6393224516bcafe58490c5987e"
format_stamp: "Formatted at 2025-08-14 13:59:35 on dist-test-slave-q1tc"
I20250814 13:59:35.641759 17703 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:59:35.712692 17703 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:59:35.714084 17703 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:59:35.714476 17703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:59:35.716877 17703 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 13:59:35.720479 17703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 13:59:35.720656 17703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 13:59:35.720886 17703 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 13:59:35.721024 17703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 13:59:35.845880 17703 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.130:46609
I20250814 13:59:35.845942 17831 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.130:46609 every 8 connection(s)
I20250814 13:59:35.848520 17703 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-1/data/info.pb
I20250814 13:59:35.855011 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 17703
I20250814 13:59:35.855460 11794 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-1/wal/instance
I20250814 13:59:35.860911 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.132.131:0
--local_ip_for_outbound_sockets=127.11.132.131
--webserver_interface=127.11.132.131
--webserver_port=0
--tserver_master_addrs=127.11.132.190:46139
--builtin_ntp_servers=127.11.132.148:37633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250814 13:59:35.870862 17832 heartbeater.cc:344] Connected to a master server at 127.11.132.190:46139
I20250814 13:59:35.871248 17832 heartbeater.cc:461] Registering TS with master...
I20250814 13:59:35.872114 17832 heartbeater.cc:507] Master 127.11.132.190:46139 requested a full tablet report, sending...
I20250814 13:59:35.874038 17511 ts_manager.cc:194] Registered new tserver with Master: 274c7e6393224516bcafe58490c5987e (127.11.132.130:46609)
I20250814 13:59:35.875229 17511 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.132.130:45393
W20250814 13:59:36.150982 17836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:59:36.151418 17836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:59:36.151929 17836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:59:36.183694 17836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:59:36.184500 17836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.132.131
I20250814 13:59:36.217267 17836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:37633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.132.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.132.131
--webserver_port=0
--tserver_master_addrs=127.11.132.190:46139
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.132.131
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:59:36.218451 17836 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:59:36.220027 17836 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:59:36.231287 17842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 13:59:36.879377 17832 heartbeater.cc:499] Master 127.11.132.190:46139 was elected leader, sending a full tablet report...
W20250814 13:59:36.231933 17843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:59:37.366997 17844 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1133 milliseconds
W20250814 13:59:37.367597 17845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:59:37.370685 17836 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.139s user 0.343s sys 0.790s
W20250814 13:59:37.371003 17836 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.140s user 0.343s sys 0.790s
I20250814 13:59:37.371214 17836 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 13:59:37.372463 17836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:59:37.374496 17836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:59:37.375809 17836 hybrid_clock.cc:648] HybridClock initialized: now 1755179977375764 us; error 53 us; skew 500 ppm
I20250814 13:59:37.376545 17836 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:59:37.383153 17836 webserver.cc:480] Webserver started at http://127.11.132.131:41633/ using document root <none> and password file <none>
I20250814 13:59:37.384088 17836 fs_manager.cc:362] Metadata directory not provided
I20250814 13:59:37.384303 17836 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:59:37.384779 17836 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 13:59:37.389096 17836 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-2/data/instance:
uuid: "e59346ea53fe40c0b8cca6aeaefe269a"
format_stamp: "Formatted at 2025-08-14 13:59:37 on dist-test-slave-q1tc"
I20250814 13:59:37.390228 17836 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-2/wal/instance:
uuid: "e59346ea53fe40c0b8cca6aeaefe269a"
format_stamp: "Formatted at 2025-08-14 13:59:37 on dist-test-slave-q1tc"
I20250814 13:59:37.397388 17836 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20250814 13:59:37.403395 17852 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:59:37.404465 17836 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20250814 13:59:37.404812 17836 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-2/wal
uuid: "e59346ea53fe40c0b8cca6aeaefe269a"
format_stamp: "Formatted at 2025-08-14 13:59:37 on dist-test-slave-q1tc"
I20250814 13:59:37.405144 17836 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:59:37.472453 17836 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:59:37.473755 17836 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:59:37.474126 17836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:59:37.476598 17836 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 13:59:37.480381 17836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 13:59:37.480584 17836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 13:59:37.480808 17836 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 13:59:37.480952 17836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 13:59:37.604368 17836 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.131:45543
I20250814 13:59:37.604471 17964 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.131:45543 every 8 connection(s)
I20250814 13:59:37.606842 17836 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-2/data/info.pb
I20250814 13:59:37.613927 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 17836
I20250814 13:59:37.614347 11794 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-2/wal/instance
I20250814 13:59:37.625905 17965 heartbeater.cc:344] Connected to a master server at 127.11.132.190:46139
I20250814 13:59:37.626219 17965 heartbeater.cc:461] Registering TS with master...
I20250814 13:59:37.627403 17965 heartbeater.cc:507] Master 127.11.132.190:46139 requested a full tablet report, sending...
I20250814 13:59:37.629208 17511 ts_manager.cc:194] Registered new tserver with Master: e59346ea53fe40c0b8cca6aeaefe269a (127.11.132.131:45543)
I20250814 13:59:37.630484 17511 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.132.131:38261
I20250814 13:59:37.633599 11794 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250814 13:59:37.655925 11794 test_util.cc:276] Using random seed: -1357527456
I20250814 13:59:37.691049 17511 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46340:
name: "pre_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250814 13:59:37.693183 17511 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table pre_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250814 13:59:37.741156 17634 tablet_service.cc:1468] Processing CreateTablet for tablet 53f2621a8d3147f18044b93f74c391d3 (DEFAULT_TABLE table=pre_rebuild [id=428276c4c2a44cca915958cdf3463b12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 13:59:37.743110 17634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53f2621a8d3147f18044b93f74c391d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:59:37.744854 17900 tablet_service.cc:1468] Processing CreateTablet for tablet 53f2621a8d3147f18044b93f74c391d3 (DEFAULT_TABLE table=pre_rebuild [id=428276c4c2a44cca915958cdf3463b12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 13:59:37.746789 17900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53f2621a8d3147f18044b93f74c391d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:59:37.749418 17767 tablet_service.cc:1468] Processing CreateTablet for tablet 53f2621a8d3147f18044b93f74c391d3 (DEFAULT_TABLE table=pre_rebuild [id=428276c4c2a44cca915958cdf3463b12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 13:59:37.751247 17767 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53f2621a8d3147f18044b93f74c391d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:59:37.772763 17989 tablet_bootstrap.cc:492] T 53f2621a8d3147f18044b93f74c391d3 P 274c7e6393224516bcafe58490c5987e: Bootstrap starting.
I20250814 13:59:37.773774 17990 tablet_bootstrap.cc:492] T 53f2621a8d3147f18044b93f74c391d3 P e59346ea53fe40c0b8cca6aeaefe269a: Bootstrap starting.
I20250814 13:59:37.776520 17991 tablet_bootstrap.cc:492] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca: Bootstrap starting.
I20250814 13:59:37.780948 17990 tablet_bootstrap.cc:654] T 53f2621a8d3147f18044b93f74c391d3 P e59346ea53fe40c0b8cca6aeaefe269a: Neither blocks nor log segments found. Creating new log.
I20250814 13:59:37.781740 17991 tablet_bootstrap.cc:654] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca: Neither blocks nor log segments found. Creating new log.
I20250814 13:59:37.781704 17989 tablet_bootstrap.cc:654] T 53f2621a8d3147f18044b93f74c391d3 P 274c7e6393224516bcafe58490c5987e: Neither blocks nor log segments found. Creating new log.
I20250814 13:59:37.783097 17990 log.cc:826] T 53f2621a8d3147f18044b93f74c391d3 P e59346ea53fe40c0b8cca6aeaefe269a: Log is configured to *not* fsync() on all Append() calls
I20250814 13:59:37.783282 17991 log.cc:826] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca: Log is configured to *not* fsync() on all Append() calls
I20250814 13:59:37.783804 17989 log.cc:826] T 53f2621a8d3147f18044b93f74c391d3 P 274c7e6393224516bcafe58490c5987e: Log is configured to *not* fsync() on all Append() calls
I20250814 13:59:37.788679 17991 tablet_bootstrap.cc:492] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca: No bootstrap required, opened a new log
I20250814 13:59:37.789178 17991 ts_tablet_manager.cc:1397] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.010s
I20250814 13:59:37.790082 17989 tablet_bootstrap.cc:492] T 53f2621a8d3147f18044b93f74c391d3 P 274c7e6393224516bcafe58490c5987e: No bootstrap required, opened a new log
I20250814 13:59:37.790407 17990 tablet_bootstrap.cc:492] T 53f2621a8d3147f18044b93f74c391d3 P e59346ea53fe40c0b8cca6aeaefe269a: No bootstrap required, opened a new log
I20250814 13:59:37.790577 17989 ts_tablet_manager.cc:1397] T 53f2621a8d3147f18044b93f74c391d3 P 274c7e6393224516bcafe58490c5987e: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.004s
I20250814 13:59:37.790781 17990 ts_tablet_manager.cc:1397] T 53f2621a8d3147f18044b93f74c391d3 P e59346ea53fe40c0b8cca6aeaefe269a: Time spent bootstrapping tablet: real 0.017s user 0.016s sys 0.000s
I20250814 13:59:37.808203 17990 raft_consensus.cc:357] T 53f2621a8d3147f18044b93f74c391d3 P e59346ea53fe40c0b8cca6aeaefe269a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 35349 } } peers { permanent_uuid: "274c7e6393224516bcafe58490c5987e" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46609 } } peers { permanent_uuid: "e59346ea53fe40c0b8cca6aeaefe269a" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 45543 } }
I20250814 13:59:37.808835 17990 raft_consensus.cc:383] T 53f2621a8d3147f18044b93f74c391d3 P e59346ea53fe40c0b8cca6aeaefe269a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 13:59:37.809121 17990 raft_consensus.cc:738] T 53f2621a8d3147f18044b93f74c391d3 P e59346ea53fe40c0b8cca6aeaefe269a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59346ea53fe40c0b8cca6aeaefe269a, State: Initialized, Role: FOLLOWER
I20250814 13:59:37.809762 17990 consensus_queue.cc:260] T 53f2621a8d3147f18044b93f74c391d3 P e59346ea53fe40c0b8cca6aeaefe269a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 35349 } } peers { permanent_uuid: "274c7e6393224516bcafe58490c5987e" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46609 } } peers { permanent_uuid: "e59346ea53fe40c0b8cca6aeaefe269a" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 45543 } }
I20250814 13:59:37.812592 17965 heartbeater.cc:499] Master 127.11.132.190:46139 was elected leader, sending a full tablet report...
I20250814 13:59:37.813576 17990 ts_tablet_manager.cc:1428] T 53f2621a8d3147f18044b93f74c391d3 P e59346ea53fe40c0b8cca6aeaefe269a: Time spent starting tablet: real 0.023s user 0.023s sys 0.000s
I20250814 13:59:37.814653 17991 raft_consensus.cc:357] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 35349 } } peers { permanent_uuid: "274c7e6393224516bcafe58490c5987e" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46609 } } peers { permanent_uuid: "e59346ea53fe40c0b8cca6aeaefe269a" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 45543 } }
I20250814 13:59:37.815500 17991 raft_consensus.cc:383] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 13:59:37.815788 17991 raft_consensus.cc:738] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb2f9678de6476c9f17aad6dc2a25ca, State: Initialized, Role: FOLLOWER
I20250814 13:59:37.816098 17989 raft_consensus.cc:357] T 53f2621a8d3147f18044b93f74c391d3 P 274c7e6393224516bcafe58490c5987e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 35349 } } peers { permanent_uuid: "274c7e6393224516bcafe58490c5987e" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46609 } } peers { permanent_uuid: "e59346ea53fe40c0b8cca6aeaefe269a" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 45543 } }
I20250814 13:59:37.816951 17989 raft_consensus.cc:383] T 53f2621a8d3147f18044b93f74c391d3 P 274c7e6393224516bcafe58490c5987e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 13:59:37.816661 17991 consensus_queue.cc:260] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 35349 } } peers { permanent_uuid: "274c7e6393224516bcafe58490c5987e" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46609 } } peers { permanent_uuid: "e59346ea53fe40c0b8cca6aeaefe269a" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 45543 } }
I20250814 13:59:37.817260 17989 raft_consensus.cc:738] T 53f2621a8d3147f18044b93f74c391d3 P 274c7e6393224516bcafe58490c5987e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 274c7e6393224516bcafe58490c5987e, State: Initialized, Role: FOLLOWER
I20250814 13:59:37.818143 17989 consensus_queue.cc:260] T 53f2621a8d3147f18044b93f74c391d3 P 274c7e6393224516bcafe58490c5987e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 35349 } } peers { permanent_uuid: "274c7e6393224516bcafe58490c5987e" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46609 } } peers { permanent_uuid: "e59346ea53fe40c0b8cca6aeaefe269a" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 45543 } }
I20250814 13:59:37.823191 17989 ts_tablet_manager.cc:1428] T 53f2621a8d3147f18044b93f74c391d3 P 274c7e6393224516bcafe58490c5987e: Time spent starting tablet: real 0.032s user 0.023s sys 0.007s
I20250814 13:59:37.826115 17991 ts_tablet_manager.cc:1428] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca: Time spent starting tablet: real 0.037s user 0.033s sys 0.003s
W20250814 13:59:37.854285 17700 tablet.cc:2378] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250814 13:59:37.854456 17833 tablet.cc:2378] T 53f2621a8d3147f18044b93f74c391d3 P 274c7e6393224516bcafe58490c5987e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250814 13:59:37.860827 17966 tablet.cc:2378] T 53f2621a8d3147f18044b93f74c391d3 P e59346ea53fe40c0b8cca6aeaefe269a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250814 13:59:37.977941 17996 raft_consensus.cc:491] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250814 13:59:37.978385 17996 raft_consensus.cc:513] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 35349 } } peers { permanent_uuid: "274c7e6393224516bcafe58490c5987e" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46609 } } peers { permanent_uuid: "e59346ea53fe40c0b8cca6aeaefe269a" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 45543 } }
I20250814 13:59:37.980579 17996 leader_election.cc:290] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 274c7e6393224516bcafe58490c5987e (127.11.132.130:46609), e59346ea53fe40c0b8cca6aeaefe269a (127.11.132.131:45543)
I20250814 13:59:37.992309 17920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53f2621a8d3147f18044b93f74c391d3" candidate_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59346ea53fe40c0b8cca6aeaefe269a" is_pre_election: true
I20250814 13:59:37.992306 17787 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53f2621a8d3147f18044b93f74c391d3" candidate_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "274c7e6393224516bcafe58490c5987e" is_pre_election: true
I20250814 13:59:37.992949 17920 raft_consensus.cc:2466] T 53f2621a8d3147f18044b93f74c391d3 P e59346ea53fe40c0b8cca6aeaefe269a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb2f9678de6476c9f17aad6dc2a25ca in term 0.
I20250814 13:59:37.992934 17787 raft_consensus.cc:2466] T 53f2621a8d3147f18044b93f74c391d3 P 274c7e6393224516bcafe58490c5987e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb2f9678de6476c9f17aad6dc2a25ca in term 0.
I20250814 13:59:37.993940 17590 leader_election.cc:304] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb2f9678de6476c9f17aad6dc2a25ca, e59346ea53fe40c0b8cca6aeaefe269a; no voters:
I20250814 13:59:37.994544 17996 raft_consensus.cc:2802] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250814 13:59:37.994832 17996 raft_consensus.cc:491] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250814 13:59:37.995077 17996 raft_consensus.cc:3058] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [term 0 FOLLOWER]: Advancing to term 1
I20250814 13:59:37.999269 17996 raft_consensus.cc:513] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 35349 } } peers { permanent_uuid: "274c7e6393224516bcafe58490c5987e" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46609 } } peers { permanent_uuid: "e59346ea53fe40c0b8cca6aeaefe269a" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 45543 } }
I20250814 13:59:38.000593 17996 leader_election.cc:290] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [CANDIDATE]: Term 1 election: Requested vote from peers 274c7e6393224516bcafe58490c5987e (127.11.132.130:46609), e59346ea53fe40c0b8cca6aeaefe269a (127.11.132.131:45543)
I20250814 13:59:38.001303 17787 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53f2621a8d3147f18044b93f74c391d3" candidate_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "274c7e6393224516bcafe58490c5987e"
I20250814 13:59:38.001523 17920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53f2621a8d3147f18044b93f74c391d3" candidate_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59346ea53fe40c0b8cca6aeaefe269a"
I20250814 13:59:38.001786 17787 raft_consensus.cc:3058] T 53f2621a8d3147f18044b93f74c391d3 P 274c7e6393224516bcafe58490c5987e [term 0 FOLLOWER]: Advancing to term 1
I20250814 13:59:38.001977 17920 raft_consensus.cc:3058] T 53f2621a8d3147f18044b93f74c391d3 P e59346ea53fe40c0b8cca6aeaefe269a [term 0 FOLLOWER]: Advancing to term 1
I20250814 13:59:38.008489 17787 raft_consensus.cc:2466] T 53f2621a8d3147f18044b93f74c391d3 P 274c7e6393224516bcafe58490c5987e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb2f9678de6476c9f17aad6dc2a25ca in term 1.
I20250814 13:59:38.008598 17920 raft_consensus.cc:2466] T 53f2621a8d3147f18044b93f74c391d3 P e59346ea53fe40c0b8cca6aeaefe269a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb2f9678de6476c9f17aad6dc2a25ca in term 1.
I20250814 13:59:38.009459 17589 leader_election.cc:304] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 274c7e6393224516bcafe58490c5987e, 7eb2f9678de6476c9f17aad6dc2a25ca; no voters:
I20250814 13:59:38.010023 17996 raft_consensus.cc:2802] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [term 1 FOLLOWER]: Leader election won for term 1
I20250814 13:59:38.011340 17996 raft_consensus.cc:695] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [term 1 LEADER]: Becoming Leader. State: Replica: 7eb2f9678de6476c9f17aad6dc2a25ca, State: Running, Role: LEADER
I20250814 13:59:38.011989 17996 consensus_queue.cc:237] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 35349 } } peers { permanent_uuid: "274c7e6393224516bcafe58490c5987e" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46609 } } peers { permanent_uuid: "e59346ea53fe40c0b8cca6aeaefe269a" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 45543 } }
I20250814 13:59:38.023130 17511 catalog_manager.cc:5582] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 7eb2f9678de6476c9f17aad6dc2a25ca (127.11.132.129). New cstate: current_term: 1 leader_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 35349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "274c7e6393224516bcafe58490c5987e" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59346ea53fe40c0b8cca6aeaefe269a" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 45543 } health_report { overall_health: UNKNOWN } } }
I20250814 13:59:38.200150 17920 raft_consensus.cc:1273] T 53f2621a8d3147f18044b93f74c391d3 P e59346ea53fe40c0b8cca6aeaefe269a [term 1 FOLLOWER]: Refusing update from remote peer 7eb2f9678de6476c9f17aad6dc2a25ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250814 13:59:38.200150 17787 raft_consensus.cc:1273] T 53f2621a8d3147f18044b93f74c391d3 P 274c7e6393224516bcafe58490c5987e [term 1 FOLLOWER]: Refusing update from remote peer 7eb2f9678de6476c9f17aad6dc2a25ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250814 13:59:38.201810 18000 consensus_queue.cc:1035] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59346ea53fe40c0b8cca6aeaefe269a" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 45543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250814 13:59:38.202567 17996 consensus_queue.cc:1035] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "274c7e6393224516bcafe58490c5987e" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250814 13:59:38.231770 18008 mvcc.cc:204] Tried to move back new op lower bound from 7189217190691880960 to 7189217189943676928. Current Snapshot: MvccSnapshot[applied={T|T < 7189217190691880960}]
W20250814 13:59:41.800808 17828 debug-util.cc:398] Leaking SignalData structure 0x7b08000c67a0 after lost signal to thread 17704
W20250814 13:59:41.801893 17828 debug-util.cc:398] Leaking SignalData structure 0x7b08000cc2a0 after lost signal to thread 17831
I20250814 13:59:43.230545 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 17478
W20250814 13:59:43.301880 17699 heartbeater.cc:646] Failed to heartbeat to 127.11.132.190:46139 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.132.190:46139: connect: Connection refused (error 111)
W20250814 13:59:43.344108 17965 heartbeater.cc:646] Failed to heartbeat to 127.11.132.190:46139 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.132.190:46139: connect: Connection refused (error 111)
W20250814 13:59:43.372341 17832 heartbeater.cc:646] Failed to heartbeat to 127.11.132.190:46139 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.132.190:46139: connect: Connection refused (error 111)
W20250814 13:59:43.585717 18039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:59:43.586300 18039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:59:43.629752 18039 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250814 13:59:45.102968 18047 debug-util.cc:398] Leaking SignalData structure 0x7b08000373e0 after lost signal to thread 18039
W20250814 13:59:45.103646 18047 kernel_stack_watchdog.cc:198] Thread 18039 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 383ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250814 13:59:45.331336 18039 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.614s user 0.001s sys 0.001s
W20250814 13:59:45.431125 18039 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.714s user 0.004s sys 0.010s
I20250814 13:59:45.504330 18039 minidump.cc:252] Setting minidump size limit to 20M
I20250814 13:59:45.506125 18039 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:59:45.507144 18039 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:59:45.516381 18075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:59:45.518010 18076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:59:45.520215 18078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 13:59:45.521340 18039 server_base.cc:1047] running on GCE node
I20250814 13:59:45.522239 18039 hybrid_clock.cc:584] initializing the hybrid clock with 'system' time source
I20250814 13:59:45.522642 18039 hybrid_clock.cc:648] HybridClock initialized: now 1755179985522623 us; error 317516 us; skew 500 ppm
I20250814 13:59:45.523324 18039 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:59:45.527415 18039 webserver.cc:480] Webserver started at http://0.0.0.0:46615/ using document root <none> and password file <none>
I20250814 13:59:45.528221 18039 fs_manager.cc:362] Metadata directory not provided
I20250814 13:59:45.528399 18039 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:59:45.528736 18039 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 13:59:45.532553 18039 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/master-0/data/instance:
uuid: "77c45e5eeed74f7e861f72e101ea9f5c"
format_stamp: "Formatted at 2025-08-14 13:59:45 on dist-test-slave-q1tc"
I20250814 13:59:45.533592 18039 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/master-0/wal/instance:
uuid: "77c45e5eeed74f7e861f72e101ea9f5c"
format_stamp: "Formatted at 2025-08-14 13:59:45 on dist-test-slave-q1tc"
I20250814 13:59:45.539000 18039 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250814 13:59:45.543521 18083 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:59:45.544348 18039 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250814 13:59:45.544605 18039 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/master-0/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/master-0/wal
uuid: "77c45e5eeed74f7e861f72e101ea9f5c"
format_stamp: "Formatted at 2025-08-14 13:59:45 on dist-test-slave-q1tc"
I20250814 13:59:45.544867 18039 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:59:45.615139 18039 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:59:45.617100 18039 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:59:45.617625 18039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:59:45.623268 18039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:59:45.638224 18039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c: Bootstrap starting.
I20250814 13:59:45.643289 18039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c: Neither blocks nor log segments found. Creating new log.
I20250814 13:59:45.644897 18039 log.cc:826] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c: Log is configured to *not* fsync() on all Append() calls
I20250814 13:59:45.648670 18039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c: No bootstrap required, opened a new log
I20250814 13:59:45.663684 18039 raft_consensus.cc:357] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c45e5eeed74f7e861f72e101ea9f5c" member_type: VOTER }
I20250814 13:59:45.664151 18039 raft_consensus.cc:383] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 13:59:45.664371 18039 raft_consensus.cc:738] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77c45e5eeed74f7e861f72e101ea9f5c, State: Initialized, Role: FOLLOWER
I20250814 13:59:45.664971 18039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c45e5eeed74f7e861f72e101ea9f5c" member_type: VOTER }
I20250814 13:59:45.665462 18039 raft_consensus.cc:397] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 13:59:45.665693 18039 raft_consensus.cc:491] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 13:59:45.665972 18039 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c [term 0 FOLLOWER]: Advancing to term 1
I20250814 13:59:45.669584 18039 raft_consensus.cc:513] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c45e5eeed74f7e861f72e101ea9f5c" member_type: VOTER }
I20250814 13:59:45.670137 18039 leader_election.cc:304] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77c45e5eeed74f7e861f72e101ea9f5c; no voters:
I20250814 13:59:45.671713 18039 leader_election.cc:290] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250814 13:59:45.671931 18090 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c [term 1 FOLLOWER]: Leader election won for term 1
I20250814 13:59:45.673894 18090 raft_consensus.cc:695] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c [term 1 LEADER]: Becoming Leader. State: Replica: 77c45e5eeed74f7e861f72e101ea9f5c, State: Running, Role: LEADER
I20250814 13:59:45.674678 18090 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c45e5eeed74f7e861f72e101ea9f5c" member_type: VOTER }
I20250814 13:59:45.681413 18092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77c45e5eeed74f7e861f72e101ea9f5c. Latest consensus state: current_term: 1 leader_uuid: "77c45e5eeed74f7e861f72e101ea9f5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c45e5eeed74f7e861f72e101ea9f5c" member_type: VOTER } }
I20250814 13:59:45.681871 18092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c [sys.catalog]: This master's current role is: LEADER
I20250814 13:59:45.681797 18091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77c45e5eeed74f7e861f72e101ea9f5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c45e5eeed74f7e861f72e101ea9f5c" member_type: VOTER } }
I20250814 13:59:45.682322 18091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c [sys.catalog]: This master's current role is: LEADER
I20250814 13:59:45.692751 18039 tablet_replica.cc:331] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c: stopping tablet replica
I20250814 13:59:45.693284 18039 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c [term 1 LEADER]: Raft consensus shutting down.
I20250814 13:59:45.693608 18039 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250814 13:59:45.695377 18039 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250814 13:59:45.695744 18039 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250814 13:59:45.746788 18039 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
W20250814 13:59:46.305658 18047 debug-util.cc:398] Leaking SignalData structure 0x7b08000347c0 after lost signal to thread 18039
I20250814 13:59:46.774273 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 17570
I20250814 13:59:46.813601 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 17703
I20250814 13:59:46.845347 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 17836
I20250814 13:59:46.879405 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.190:46139
--webserver_interface=127.11.132.190
--webserver_port=39569
--builtin_ntp_servers=127.11.132.148:37633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.132.190:46139 with env {}
W20250814 13:59:47.160221 18101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:59:47.160808 18101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:59:47.161290 18101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:59:47.190518 18101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250814 13:59:47.190793 18101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:59:47.191068 18101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250814 13:59:47.191293 18101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250814 13:59:47.223589 18101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:37633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.132.190:46139
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.132.190:46139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.132.190
--webserver_port=39569
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:59:47.224824 18101 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:59:47.226316 18101 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:59:47.235674 18107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:59:47.237569 18108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:59:48.375608 18110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:59:48.378372 18109 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1136 milliseconds
I20250814 13:59:48.378492 18101 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 13:59:48.379678 18101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:59:48.382134 18101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:59:48.383451 18101 hybrid_clock.cc:648] HybridClock initialized: now 1755179988383406 us; error 48 us; skew 500 ppm
I20250814 13:59:48.384249 18101 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:59:48.390697 18101 webserver.cc:480] Webserver started at http://127.11.132.190:39569/ using document root <none> and password file <none>
I20250814 13:59:48.391552 18101 fs_manager.cc:362] Metadata directory not provided
I20250814 13:59:48.391765 18101 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:59:48.400002 18101 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.001s
I20250814 13:59:48.404399 18117 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:59:48.405421 18101 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250814 13:59:48.405714 18101 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/master-0/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/master-0/wal
uuid: "77c45e5eeed74f7e861f72e101ea9f5c"
format_stamp: "Formatted at 2025-08-14 13:59:45 on dist-test-slave-q1tc"
I20250814 13:59:48.407524 18101 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:59:48.466254 18101 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:59:48.467631 18101 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:59:48.468052 18101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:59:48.534679 18101 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.190:46139
I20250814 13:59:48.534777 18168 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.190:46139 every 8 connection(s)
I20250814 13:59:48.537289 18101 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/master-0/data/info.pb
I20250814 13:59:48.545544 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 18101
I20250814 13:59:48.546854 18169 sys_catalog.cc:263] Verifying existing consensus state
I20250814 13:59:48.546901 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.132.129:35349
--local_ip_for_outbound_sockets=127.11.132.129
--tserver_master_addrs=127.11.132.190:46139
--webserver_port=38669
--webserver_interface=127.11.132.129
--builtin_ntp_servers=127.11.132.148:37633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250814 13:59:48.560505 18169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c: Bootstrap starting.
I20250814 13:59:48.575549 18169 log.cc:826] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c: Log is configured to *not* fsync() on all Append() calls
I20250814 13:59:48.590137 18169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 13:59:48.591114 18169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c: Bootstrap complete.
I20250814 13:59:48.613022 18169 raft_consensus.cc:357] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c45e5eeed74f7e861f72e101ea9f5c" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 46139 } }
I20250814 13:59:48.613691 18169 raft_consensus.cc:738] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77c45e5eeed74f7e861f72e101ea9f5c, State: Initialized, Role: FOLLOWER
I20250814 13:59:48.614382 18169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c45e5eeed74f7e861f72e101ea9f5c" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 46139 } }
I20250814 13:59:48.614840 18169 raft_consensus.cc:397] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 13:59:48.615088 18169 raft_consensus.cc:491] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 13:59:48.615386 18169 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c [term 1 FOLLOWER]: Advancing to term 2
I20250814 13:59:48.619318 18169 raft_consensus.cc:513] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c45e5eeed74f7e861f72e101ea9f5c" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 46139 } }
I20250814 13:59:48.619911 18169 leader_election.cc:304] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77c45e5eeed74f7e861f72e101ea9f5c; no voters:
I20250814 13:59:48.622016 18169 leader_election.cc:290] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c [CANDIDATE]: Term 2 election: Requested vote from peers
I20250814 13:59:48.622555 18173 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c [term 2 FOLLOWER]: Leader election won for term 2
I20250814 13:59:48.625701 18173 raft_consensus.cc:695] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c [term 2 LEADER]: Becoming Leader. State: Replica: 77c45e5eeed74f7e861f72e101ea9f5c, State: Running, Role: LEADER
I20250814 13:59:48.626575 18173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c45e5eeed74f7e861f72e101ea9f5c" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 46139 } }
I20250814 13:59:48.627408 18169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c [sys.catalog]: configured and running, proceeding with master startup.
I20250814 13:59:48.636188 18175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77c45e5eeed74f7e861f72e101ea9f5c. Latest consensus state: current_term: 2 leader_uuid: "77c45e5eeed74f7e861f72e101ea9f5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c45e5eeed74f7e861f72e101ea9f5c" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 46139 } } }
I20250814 13:59:48.637311 18174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "77c45e5eeed74f7e861f72e101ea9f5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c45e5eeed74f7e861f72e101ea9f5c" member_type: VOTER last_known_addr { host: "127.11.132.190" port: 46139 } } }
I20250814 13:59:48.637984 18174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c [sys.catalog]: This master's current role is: LEADER
I20250814 13:59:48.640883 18175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c [sys.catalog]: This master's current role is: LEADER
I20250814 13:59:48.652087 18180 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250814 13:59:48.664381 18180 catalog_manager.cc:671] Loaded metadata for table pre_rebuild [id=bb010748db0d4382aed9a4b91f12f538]
I20250814 13:59:48.672683 18180 tablet_loader.cc:96] loaded metadata for tablet 53f2621a8d3147f18044b93f74c391d3 (table pre_rebuild [id=bb010748db0d4382aed9a4b91f12f538])
I20250814 13:59:48.674659 18180 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250814 13:59:48.700261 18180 catalog_manager.cc:1349] Generated new cluster ID: a7ca848060c442aca2575e50d0c89c1b
I20250814 13:59:48.700526 18180 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250814 13:59:48.738481 18180 catalog_manager.cc:1372] Generated new certificate authority record
I20250814 13:59:48.740406 18180 catalog_manager.cc:1506] Loading token signing keys...
I20250814 13:59:48.767043 18180 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c: Generated new TSK 0
I20250814 13:59:48.768064 18180 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250814 13:59:48.893244 18171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:59:48.893712 18171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:59:48.894163 18171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:59:48.924954 18171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:59:48.925859 18171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.132.129
I20250814 13:59:48.961047 18171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:37633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.132.129:35349
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.132.129
--webserver_port=38669
--tserver_master_addrs=127.11.132.190:46139
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.132.129
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:59:48.962386 18171 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:59:48.963881 18171 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:59:48.976217 18197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:59:50.379449 18196 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 18171
W20250814 13:59:50.638041 18196 kernel_stack_watchdog.cc:198] Thread 18171 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250814 13:59:48.977430 18198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:59:50.639386 18171 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.662s user 0.000s sys 0.002s
W20250814 13:59:50.639732 18171 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.663s user 0.000s sys 0.002s
W20250814 13:59:50.644428 18199 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1666 milliseconds
W20250814 13:59:50.645123 18201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 13:59:50.645196 18171 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 13:59:50.646612 18171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:59:50.649142 18171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:59:50.650662 18171 hybrid_clock.cc:648] HybridClock initialized: now 1755179990650608 us; error 41 us; skew 500 ppm
I20250814 13:59:50.651695 18171 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:59:50.658887 18171 webserver.cc:480] Webserver started at http://127.11.132.129:38669/ using document root <none> and password file <none>
I20250814 13:59:50.660074 18171 fs_manager.cc:362] Metadata directory not provided
I20250814 13:59:50.660365 18171 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:59:50.670487 18171 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.000s sys 0.008s
I20250814 13:59:50.675997 18208 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:59:50.677166 18171 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250814 13:59:50.677462 18171 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-0/wal
uuid: "7eb2f9678de6476c9f17aad6dc2a25ca"
format_stamp: "Formatted at 2025-08-14 13:59:33 on dist-test-slave-q1tc"
I20250814 13:59:50.679386 18171 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:59:50.741978 18171 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:59:50.743338 18171 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:59:50.743762 18171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:59:50.746321 18171 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 13:59:50.752519 18215 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250814 13:59:50.763185 18171 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250814 13:59:50.763402 18171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.012s user 0.001s sys 0.001s
I20250814 13:59:50.763657 18171 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250814 13:59:50.768090 18171 ts_tablet_manager.cc:610] Registered 1 tablets
I20250814 13:59:50.768276 18171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.002s sys 0.000s
I20250814 13:59:50.768716 18215 tablet_bootstrap.cc:492] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca: Bootstrap starting.
I20250814 13:59:50.961314 18171 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.129:35349
I20250814 13:59:50.961578 18321 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.129:35349 every 8 connection(s)
I20250814 13:59:50.964111 18171 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-0/data/info.pb
I20250814 13:59:50.979229 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 18171
I20250814 13:59:50.982016 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.132.130:46609
--local_ip_for_outbound_sockets=127.11.132.130
--tserver_master_addrs=127.11.132.190:46139
--webserver_port=40215
--webserver_interface=127.11.132.130
--builtin_ntp_servers=127.11.132.148:37633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250814 13:59:51.032176 18322 heartbeater.cc:344] Connected to a master server at 127.11.132.190:46139
I20250814 13:59:51.032660 18322 heartbeater.cc:461] Registering TS with master...
I20250814 13:59:51.033768 18322 heartbeater.cc:507] Master 127.11.132.190:46139 requested a full tablet report, sending...
I20250814 13:59:51.037978 18134 ts_manager.cc:194] Registered new tserver with Master: 7eb2f9678de6476c9f17aad6dc2a25ca (127.11.132.129:35349)
I20250814 13:59:51.046470 18134 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.132.129:52951
I20250814 13:59:51.080504 18215 log.cc:826] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca: Log is configured to *not* fsync() on all Append() calls
W20250814 13:59:51.303694 18326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:59:51.304226 18326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:59:51.304735 18326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:59:51.334925 18326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:59:51.335703 18326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.132.130
I20250814 13:59:51.367771 18326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:37633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.132.130:46609
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.132.130
--webserver_port=40215
--tserver_master_addrs=127.11.132.190:46139
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.132.130
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:59:51.369045 18326 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:59:51.370520 18326 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:59:51.383745 18333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 13:59:52.051268 18322 heartbeater.cc:499] Master 127.11.132.190:46139 was elected leader, sending a full tablet report...
W20250814 13:59:52.787134 18332 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 18326
W20250814 13:59:51.385648 18334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:59:53.075423 18332 kernel_stack_watchdog.cc:198] Thread 18326 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250814 13:59:53.076536 18326 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.694s user 0.490s sys 1.144s
W20250814 13:59:53.076997 18326 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.695s user 0.491s sys 1.144s
I20250814 13:59:53.082710 18326 server_base.cc:1047] running on GCE node
W20250814 13:59:53.083006 18338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 13:59:53.084095 18326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:59:53.086440 18326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:59:53.087844 18326 hybrid_clock.cc:648] HybridClock initialized: now 1755179993087792 us; error 52 us; skew 500 ppm
I20250814 13:59:53.088600 18326 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:59:53.094432 18326 webserver.cc:480] Webserver started at http://127.11.132.130:40215/ using document root <none> and password file <none>
I20250814 13:59:53.095254 18326 fs_manager.cc:362] Metadata directory not provided
I20250814 13:59:53.095450 18326 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:59:53.102946 18326 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.002s sys 0.003s
I20250814 13:59:53.107455 18343 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:59:53.108527 18326 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20250814 13:59:53.108827 18326 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-1/wal
uuid: "274c7e6393224516bcafe58490c5987e"
format_stamp: "Formatted at 2025-08-14 13:59:35 on dist-test-slave-q1tc"
I20250814 13:59:53.110584 18326 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:59:53.111335 18215 tablet_bootstrap.cc:492] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 13:59:53.112075 18215 tablet_bootstrap.cc:492] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca: Bootstrap complete.
I20250814 13:59:53.113385 18215 ts_tablet_manager.cc:1397] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca: Time spent bootstrapping tablet: real 2.345s user 2.258s sys 0.068s
I20250814 13:59:53.123337 18215 raft_consensus.cc:357] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 35349 } } peers { permanent_uuid: "274c7e6393224516bcafe58490c5987e" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46609 } } peers { permanent_uuid: "e59346ea53fe40c0b8cca6aeaefe269a" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 45543 } }
I20250814 13:59:53.125303 18215 raft_consensus.cc:738] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb2f9678de6476c9f17aad6dc2a25ca, State: Initialized, Role: FOLLOWER
I20250814 13:59:53.125967 18215 consensus_queue.cc:260] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 35349 } } peers { permanent_uuid: "274c7e6393224516bcafe58490c5987e" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46609 } } peers { permanent_uuid: "e59346ea53fe40c0b8cca6aeaefe269a" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 45543 } }
I20250814 13:59:53.128701 18215 ts_tablet_manager.cc:1428] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20250814 13:59:53.162796 18326 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:59:53.164105 18326 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:59:53.164525 18326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:59:53.166832 18326 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 13:59:53.171828 18351 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250814 13:59:53.178514 18326 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250814 13:59:53.178730 18326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.002s sys 0.000s
I20250814 13:59:53.178972 18326 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250814 13:59:53.182948 18326 ts_tablet_manager.cc:610] Registered 1 tablets
I20250814 13:59:53.183113 18326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.000s sys 0.002s
I20250814 13:59:53.183559 18351 tablet_bootstrap.cc:492] T 53f2621a8d3147f18044b93f74c391d3 P 274c7e6393224516bcafe58490c5987e: Bootstrap starting.
I20250814 13:59:53.346323 18326 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.130:46609
I20250814 13:59:53.346619 18457 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.130:46609 every 8 connection(s)
I20250814 13:59:53.348801 18326 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-1/data/info.pb
I20250814 13:59:53.351408 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 18326
I20250814 13:59:53.353657 11794 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
/tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.132.131:45543
--local_ip_for_outbound_sockets=127.11.132.131
--tserver_master_addrs=127.11.132.190:46139
--webserver_port=41633
--webserver_interface=127.11.132.131
--builtin_ntp_servers=127.11.132.148:37633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250814 13:59:53.380832 18458 heartbeater.cc:344] Connected to a master server at 127.11.132.190:46139
I20250814 13:59:53.381624 18458 heartbeater.cc:461] Registering TS with master...
I20250814 13:59:53.383718 18458 heartbeater.cc:507] Master 127.11.132.190:46139 requested a full tablet report, sending...
I20250814 13:59:53.389451 18134 ts_manager.cc:194] Registered new tserver with Master: 274c7e6393224516bcafe58490c5987e (127.11.132.130:46609)
I20250814 13:59:53.392174 18134 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.132.130:34533
I20250814 13:59:53.404400 18351 log.cc:826] T 53f2621a8d3147f18044b93f74c391d3 P 274c7e6393224516bcafe58490c5987e: Log is configured to *not* fsync() on all Append() calls
W20250814 13:59:53.653733 18462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 13:59:53.654206 18462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 13:59:53.654690 18462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 13:59:53.683712 18462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 13:59:53.684563 18462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.132.131
I20250814 13:59:53.717346 18462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.132.148:37633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.132.131:45543
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.132.131
--webserver_port=41633
--tserver_master_addrs=127.11.132.190:46139
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.132.131
--log_dir=/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7565
TSAN enabled
I20250814 13:59:53.718566 18462 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 13:59:53.720026 18462 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 13:59:53.731818 18469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 13:59:54.395982 18458 heartbeater.cc:499] Master 127.11.132.190:46139 was elected leader, sending a full tablet report...
I20250814 13:59:54.945887 18475 raft_consensus.cc:491] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250814 13:59:54.949375 18475 raft_consensus.cc:513] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 35349 } } peers { permanent_uuid: "274c7e6393224516bcafe58490c5987e" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46609 } } peers { permanent_uuid: "e59346ea53fe40c0b8cca6aeaefe269a" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 45543 } }
I20250814 13:59:54.977268 18475 leader_election.cc:290] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 274c7e6393224516bcafe58490c5987e (127.11.132.130:46609), e59346ea53fe40c0b8cca6aeaefe269a (127.11.132.131:45543)
W20250814 13:59:54.990897 18212 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.132.131:45543: connect: Connection refused (error 111)
W20250814 13:59:55.005076 18212 leader_election.cc:336] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e59346ea53fe40c0b8cca6aeaefe269a (127.11.132.131:45543): Network error: Client connection negotiation failed: client connection to 127.11.132.131:45543: connect: Connection refused (error 111)
I20250814 13:59:55.009346 18413 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53f2621a8d3147f18044b93f74c391d3" candidate_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "274c7e6393224516bcafe58490c5987e" is_pre_election: true
W20250814 13:59:55.031201 18211 leader_election.cc:343] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 274c7e6393224516bcafe58490c5987e (127.11.132.130:46609): Illegal state: must be running to vote when last-logged opid is not known
I20250814 13:59:55.031737 18211 leader_election.cc:304] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7eb2f9678de6476c9f17aad6dc2a25ca; no voters: 274c7e6393224516bcafe58490c5987e, e59346ea53fe40c0b8cca6aeaefe269a
I20250814 13:59:55.032725 18475 raft_consensus.cc:2747] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250814 13:59:55.135898 18468 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 18462
W20250814 13:59:53.733353 18470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:59:55.193979 18472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 13:59:55.191864 18462 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.459s user 0.407s sys 1.003s
W20250814 13:59:55.194664 18462 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.462s user 0.407s sys 1.003s
I20250814 13:59:55.194943 18462 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250814 13:59:55.193702 18471 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1460 milliseconds
I20250814 13:59:55.198073 18462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 13:59:55.200496 18462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 13:59:55.201934 18462 hybrid_clock.cc:648] HybridClock initialized: now 1755179995201886 us; error 38 us; skew 500 ppm
I20250814 13:59:55.202973 18462 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 13:59:55.209837 18462 webserver.cc:480] Webserver started at http://127.11.132.131:41633/ using document root <none> and password file <none>
I20250814 13:59:55.210990 18462 fs_manager.cc:362] Metadata directory not provided
I20250814 13:59:55.211264 18462 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 13:59:55.220635 18462 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.000s sys 0.006s
I20250814 13:59:55.225428 18484 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 13:59:55.226444 18462 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20250814 13:59:55.226732 18462 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-2/wal
uuid: "e59346ea53fe40c0b8cca6aeaefe269a"
format_stamp: "Formatted at 2025-08-14 13:59:37 on dist-test-slave-q1tc"
I20250814 13:59:55.228595 18462 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 13:59:55.276194 18462 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 13:59:55.277532 18462 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 13:59:55.277971 18462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 13:59:55.280750 18462 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 13:59:55.286486 18491 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250814 13:59:55.293430 18462 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250814 13:59:55.293612 18462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.002s
I20250814 13:59:55.293869 18462 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250814 13:59:55.297958 18462 ts_tablet_manager.cc:610] Registered 1 tablets
I20250814 13:59:55.298209 18462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.003s sys 0.000s
I20250814 13:59:55.298549 18491 tablet_bootstrap.cc:492] T 53f2621a8d3147f18044b93f74c391d3 P e59346ea53fe40c0b8cca6aeaefe269a: Bootstrap starting.
I20250814 13:59:55.516938 18462 rpc_server.cc:307] RPC server started. Bound to: 127.11.132.131:45543
I20250814 13:59:55.517098 18597 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.132.131:45543 every 8 connection(s)
I20250814 13:59:55.520368 18462 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-2/data/info.pb
I20250814 13:59:55.521385 11794 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu as pid 18462
I20250814 13:59:55.623850 18598 heartbeater.cc:344] Connected to a master server at 127.11.132.190:46139
I20250814 13:59:55.624395 18598 heartbeater.cc:461] Registering TS with master...
I20250814 13:59:55.625607 18598 heartbeater.cc:507] Master 127.11.132.190:46139 requested a full tablet report, sending...
I20250814 13:59:55.628485 18491 log.cc:826] T 53f2621a8d3147f18044b93f74c391d3 P e59346ea53fe40c0b8cca6aeaefe269a: Log is configured to *not* fsync() on all Append() calls
I20250814 13:59:55.629129 18134 ts_manager.cc:194] Registered new tserver with Master: e59346ea53fe40c0b8cca6aeaefe269a (127.11.132.131:45543)
I20250814 13:59:55.631192 11794 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250814 13:59:55.631953 18134 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.132.131:42295
I20250814 13:59:56.176774 18351 tablet_bootstrap.cc:492] T 53f2621a8d3147f18044b93f74c391d3 P 274c7e6393224516bcafe58490c5987e: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 13:59:56.177489 18351 tablet_bootstrap.cc:492] T 53f2621a8d3147f18044b93f74c391d3 P 274c7e6393224516bcafe58490c5987e: Bootstrap complete.
I20250814 13:59:56.178746 18351 ts_tablet_manager.cc:1397] T 53f2621a8d3147f18044b93f74c391d3 P 274c7e6393224516bcafe58490c5987e: Time spent bootstrapping tablet: real 2.996s user 2.829s sys 0.076s
I20250814 13:59:56.183305 18351 raft_consensus.cc:357] T 53f2621a8d3147f18044b93f74c391d3 P 274c7e6393224516bcafe58490c5987e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 35349 } } peers { permanent_uuid: "274c7e6393224516bcafe58490c5987e" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46609 } } peers { permanent_uuid: "e59346ea53fe40c0b8cca6aeaefe269a" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 45543 } }
I20250814 13:59:56.185189 18351 raft_consensus.cc:738] T 53f2621a8d3147f18044b93f74c391d3 P 274c7e6393224516bcafe58490c5987e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 274c7e6393224516bcafe58490c5987e, State: Initialized, Role: FOLLOWER
I20250814 13:59:56.185809 18351 consensus_queue.cc:260] T 53f2621a8d3147f18044b93f74c391d3 P 274c7e6393224516bcafe58490c5987e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 35349 } } peers { permanent_uuid: "274c7e6393224516bcafe58490c5987e" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46609 } } peers { permanent_uuid: "e59346ea53fe40c0b8cca6aeaefe269a" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 45543 } }
I20250814 13:59:56.188671 18351 ts_tablet_manager.cc:1428] T 53f2621a8d3147f18044b93f74c391d3 P 274c7e6393224516bcafe58490c5987e: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20250814 13:59:56.635157 18598 heartbeater.cc:499] Master 127.11.132.190:46139 was elected leader, sending a full tablet report...
I20250814 13:59:56.828822 18611 raft_consensus.cc:491] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250814 13:59:56.829212 18611 raft_consensus.cc:513] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 35349 } } peers { permanent_uuid: "274c7e6393224516bcafe58490c5987e" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46609 } } peers { permanent_uuid: "e59346ea53fe40c0b8cca6aeaefe269a" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 45543 } }
I20250814 13:59:56.830654 18611 leader_election.cc:290] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 274c7e6393224516bcafe58490c5987e (127.11.132.130:46609), e59346ea53fe40c0b8cca6aeaefe269a (127.11.132.131:45543)
I20250814 13:59:56.831543 18413 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53f2621a8d3147f18044b93f74c391d3" candidate_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "274c7e6393224516bcafe58490c5987e" is_pre_election: true
I20250814 13:59:56.833725 18413 raft_consensus.cc:2466] T 53f2621a8d3147f18044b93f74c391d3 P 274c7e6393224516bcafe58490c5987e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb2f9678de6476c9f17aad6dc2a25ca in term 1.
I20250814 13:59:56.835316 18211 leader_election.cc:304] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 274c7e6393224516bcafe58490c5987e, 7eb2f9678de6476c9f17aad6dc2a25ca; no voters:
I20250814 13:59:56.836949 18611 raft_consensus.cc:2802] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250814 13:59:56.837359 18611 raft_consensus.cc:491] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250814 13:59:56.837786 18611 raft_consensus.cc:3058] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [term 1 FOLLOWER]: Advancing to term 2
I20250814 13:59:56.845381 18611 raft_consensus.cc:513] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 35349 } } peers { permanent_uuid: "274c7e6393224516bcafe58490c5987e" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46609 } } peers { permanent_uuid: "e59346ea53fe40c0b8cca6aeaefe269a" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 45543 } }
I20250814 13:59:56.847075 18611 leader_election.cc:290] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [CANDIDATE]: Term 2 election: Requested vote from peers 274c7e6393224516bcafe58490c5987e (127.11.132.130:46609), e59346ea53fe40c0b8cca6aeaefe269a (127.11.132.131:45543)
I20250814 13:59:56.847796 18413 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53f2621a8d3147f18044b93f74c391d3" candidate_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "274c7e6393224516bcafe58490c5987e"
I20250814 13:59:56.848214 18413 raft_consensus.cc:3058] T 53f2621a8d3147f18044b93f74c391d3 P 274c7e6393224516bcafe58490c5987e [term 1 FOLLOWER]: Advancing to term 2
I20250814 13:59:56.856696 18413 raft_consensus.cc:2466] T 53f2621a8d3147f18044b93f74c391d3 P 274c7e6393224516bcafe58490c5987e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb2f9678de6476c9f17aad6dc2a25ca in term 2.
I20250814 13:59:56.857551 18211 leader_election.cc:304] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 274c7e6393224516bcafe58490c5987e, 7eb2f9678de6476c9f17aad6dc2a25ca; no voters:
I20250814 13:59:56.858269 18611 raft_consensus.cc:2802] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [term 2 FOLLOWER]: Leader election won for term 2
I20250814 13:59:56.849156 18553 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53f2621a8d3147f18044b93f74c391d3" candidate_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "e59346ea53fe40c0b8cca6aeaefe269a" is_pre_election: true
I20250814 13:59:56.849339 18552 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53f2621a8d3147f18044b93f74c391d3" candidate_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "e59346ea53fe40c0b8cca6aeaefe269a"
W20250814 13:59:56.860970 18212 leader_election.cc:343] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer e59346ea53fe40c0b8cca6aeaefe269a (127.11.132.131:45543): Illegal state: must be running to vote when last-logged opid is not known
I20250814 13:59:56.861023 18611 raft_consensus.cc:695] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [term 2 LEADER]: Becoming Leader. State: Replica: 7eb2f9678de6476c9f17aad6dc2a25ca, State: Running, Role: LEADER
I20250814 13:59:56.862046 18611 consensus_queue.cc:237] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 35349 } } peers { permanent_uuid: "274c7e6393224516bcafe58490c5987e" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46609 } } peers { permanent_uuid: "e59346ea53fe40c0b8cca6aeaefe269a" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 45543 } }
I20250814 13:59:56.872275 18134 catalog_manager.cc:5582] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca reported cstate change: term changed from 0 to 2, leader changed from <none> to 7eb2f9678de6476c9f17aad6dc2a25ca (127.11.132.129), VOTER 274c7e6393224516bcafe58490c5987e (127.11.132.130) added, VOTER 7eb2f9678de6476c9f17aad6dc2a25ca (127.11.132.129) added, VOTER e59346ea53fe40c0b8cca6aeaefe269a (127.11.132.131) added. New cstate: current_term: 2 leader_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 35349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "274c7e6393224516bcafe58490c5987e" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59346ea53fe40c0b8cca6aeaefe269a" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 45543 } health_report { overall_health: UNKNOWN } } }
I20250814 13:59:57.418332 18413 raft_consensus.cc:1273] T 53f2621a8d3147f18044b93f74c391d3 P 274c7e6393224516bcafe58490c5987e [term 2 FOLLOWER]: Refusing update from remote peer 7eb2f9678de6476c9f17aad6dc2a25ca: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250814 13:59:57.420013 18625 consensus_queue.cc:1035] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "274c7e6393224516bcafe58490c5987e" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
W20250814 13:59:57.463884 18212 consensus_peers.cc:489] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca -> Peer e59346ea53fe40c0b8cca6aeaefe269a (127.11.132.131:45543): Couldn't send request to peer e59346ea53fe40c0b8cca6aeaefe269a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250814 13:59:57.495627 18277 consensus_queue.cc:237] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 2.206, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 35349 } } peers { permanent_uuid: "274c7e6393224516bcafe58490c5987e" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46609 } }
I20250814 13:59:57.500612 18413 raft_consensus.cc:1273] T 53f2621a8d3147f18044b93f74c391d3 P 274c7e6393224516bcafe58490c5987e [term 2 FOLLOWER]: Refusing update from remote peer 7eb2f9678de6476c9f17aad6dc2a25ca: Log matching property violated. Preceding OpId in replica: term: 2 index: 206. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
I20250814 13:59:57.502000 18628 consensus_queue.cc:1035] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "274c7e6393224516bcafe58490c5987e" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.000s
I20250814 13:59:57.508069 18628 raft_consensus.cc:2953] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [term 2 LEADER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER e59346ea53fe40c0b8cca6aeaefe269a (127.11.132.131) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 35349 } } peers { permanent_uuid: "274c7e6393224516bcafe58490c5987e" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46609 } } }
I20250814 13:59:57.510277 18413 raft_consensus.cc:2953] T 53f2621a8d3147f18044b93f74c391d3 P 274c7e6393224516bcafe58490c5987e [term 2 FOLLOWER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER e59346ea53fe40c0b8cca6aeaefe269a (127.11.132.131) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 35349 } } peers { permanent_uuid: "274c7e6393224516bcafe58490c5987e" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46609 } } }
I20250814 13:59:57.520606 18491 tablet_bootstrap.cc:492] T 53f2621a8d3147f18044b93f74c391d3 P e59346ea53fe40c0b8cca6aeaefe269a: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 13:59:57.521569 18491 tablet_bootstrap.cc:492] T 53f2621a8d3147f18044b93f74c391d3 P e59346ea53fe40c0b8cca6aeaefe269a: Bootstrap complete.
I20250814 13:59:57.523221 18491 ts_tablet_manager.cc:1397] T 53f2621a8d3147f18044b93f74c391d3 P e59346ea53fe40c0b8cca6aeaefe269a: Time spent bootstrapping tablet: real 2.225s user 2.120s sys 0.076s
I20250814 13:59:57.526427 18120 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 53f2621a8d3147f18044b93f74c391d3 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250814 13:59:57.531170 18134 catalog_manager.cc:5582] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca reported cstate change: config changed from index -1 to 207, VOTER e59346ea53fe40c0b8cca6aeaefe269a (127.11.132.131) evicted. New cstate: current_term: 2 leader_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 35349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "274c7e6393224516bcafe58490c5987e" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46609 } health_report { overall_health: HEALTHY } } }
W20250814 13:59:57.536012 11794 scanner-internal.cc:458] Time spent opening tablet: real 1.872s user 0.004s sys 0.003s
I20250814 13:59:57.536936 18491 raft_consensus.cc:357] T 53f2621a8d3147f18044b93f74c391d3 P e59346ea53fe40c0b8cca6aeaefe269a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 35349 } } peers { permanent_uuid: "274c7e6393224516bcafe58490c5987e" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46609 } } peers { permanent_uuid: "e59346ea53fe40c0b8cca6aeaefe269a" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 45543 } }
I20250814 13:59:57.539741 18491 raft_consensus.cc:738] T 53f2621a8d3147f18044b93f74c391d3 P e59346ea53fe40c0b8cca6aeaefe269a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59346ea53fe40c0b8cca6aeaefe269a, State: Initialized, Role: FOLLOWER
I20250814 13:59:57.545406 18491 consensus_queue.cc:260] T 53f2621a8d3147f18044b93f74c391d3 P e59346ea53fe40c0b8cca6aeaefe269a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 35349 } } peers { permanent_uuid: "274c7e6393224516bcafe58490c5987e" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46609 } } peers { permanent_uuid: "e59346ea53fe40c0b8cca6aeaefe269a" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 45543 } }
I20250814 13:59:57.552904 18491 ts_tablet_manager.cc:1428] T 53f2621a8d3147f18044b93f74c391d3 P e59346ea53fe40c0b8cca6aeaefe269a: Time spent starting tablet: real 0.029s user 0.014s sys 0.000s
I20250814 13:59:57.589565 18277 consensus_queue.cc:237] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 2.207, Last appended by leader: 205, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 35349 } }
I20250814 13:59:57.595580 18628 raft_consensus.cc:2953] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [term 2 LEADER]: Committing config change with OpId 2.208: config changed from index 207 to 208, VOTER 274c7e6393224516bcafe58490c5987e (127.11.132.130) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 35349 } } }
I20250814 13:59:57.607548 18120 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 53f2621a8d3147f18044b93f74c391d3 with cas_config_opid_index 207: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250814 13:59:57.611413 18132 catalog_manager.cc:5582] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca reported cstate change: config changed from index 207 to 208, VOTER 274c7e6393224516bcafe58490c5987e (127.11.132.130) evicted. New cstate: current_term: 2 leader_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 35349 } health_report { overall_health: HEALTHY } } }
I20250814 13:59:57.615142 18533 tablet_service.cc:1515] Processing DeleteTablet for tablet 53f2621a8d3147f18044b93f74c391d3 with delete_type TABLET_DATA_TOMBSTONED (TS e59346ea53fe40c0b8cca6aeaefe269a not found in new config with opid_index 207) from {username='slave'} at 127.0.0.1:41984
I20250814 13:59:57.637542 18637 tablet_replica.cc:331] stopping tablet replica
I20250814 13:59:57.638367 18637 raft_consensus.cc:2241] T 53f2621a8d3147f18044b93f74c391d3 P e59346ea53fe40c0b8cca6aeaefe269a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250814 13:59:57.638914 18637 raft_consensus.cc:2270] T 53f2621a8d3147f18044b93f74c391d3 P e59346ea53fe40c0b8cca6aeaefe269a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250814 13:59:57.645450 18393 tablet_service.cc:1515] Processing DeleteTablet for tablet 53f2621a8d3147f18044b93f74c391d3 with delete_type TABLET_DATA_TOMBSTONED (TS 274c7e6393224516bcafe58490c5987e not found in new config with opid_index 208) from {username='slave'} at 127.0.0.1:49116
I20250814 13:59:57.655272 18639 tablet_replica.cc:331] stopping tablet replica
I20250814 13:59:57.656292 18639 raft_consensus.cc:2241] T 53f2621a8d3147f18044b93f74c391d3 P 274c7e6393224516bcafe58490c5987e [term 2 FOLLOWER]: Raft consensus shutting down.
I20250814 13:59:57.656900 18639 raft_consensus.cc:2270] T 53f2621a8d3147f18044b93f74c391d3 P 274c7e6393224516bcafe58490c5987e [term 2 FOLLOWER]: Raft consensus is shut down!
I20250814 13:59:57.662925 18637 ts_tablet_manager.cc:1905] T 53f2621a8d3147f18044b93f74c391d3 P e59346ea53fe40c0b8cca6aeaefe269a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250814 13:59:57.678881 18637 ts_tablet_manager.cc:1918] T 53f2621a8d3147f18044b93f74c391d3 P e59346ea53fe40c0b8cca6aeaefe269a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.205
I20250814 13:59:57.679342 18637 log.cc:1199] T 53f2621a8d3147f18044b93f74c391d3 P e59346ea53fe40c0b8cca6aeaefe269a: Deleting WAL directory at /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-2/wal/wals/53f2621a8d3147f18044b93f74c391d3
I20250814 13:59:57.680970 18121 catalog_manager.cc:4928] TS e59346ea53fe40c0b8cca6aeaefe269a (127.11.132.131:45543): tablet 53f2621a8d3147f18044b93f74c391d3 (table pre_rebuild [id=bb010748db0d4382aed9a4b91f12f538]) successfully deleted
I20250814 13:59:57.682708 18639 ts_tablet_manager.cc:1905] T 53f2621a8d3147f18044b93f74c391d3 P 274c7e6393224516bcafe58490c5987e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250814 13:59:57.696563 18639 ts_tablet_manager.cc:1918] T 53f2621a8d3147f18044b93f74c391d3 P 274c7e6393224516bcafe58490c5987e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.207
I20250814 13:59:57.696899 18639 log.cc:1199] T 53f2621a8d3147f18044b93f74c391d3 P 274c7e6393224516bcafe58490c5987e: Deleting WAL directory at /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-1/wal/wals/53f2621a8d3147f18044b93f74c391d3
I20250814 13:59:57.698678 18120 catalog_manager.cc:4928] TS 274c7e6393224516bcafe58490c5987e (127.11.132.130:46609): tablet 53f2621a8d3147f18044b93f74c391d3 (table pre_rebuild [id=bb010748db0d4382aed9a4b91f12f538]) successfully deleted
I20250814 13:59:58.213514 18257 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250814 13:59:58.214282 18393 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250814 13:59:58.223862 18533 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
77c45e5eeed74f7e861f72e101ea9f5c | 127.11.132.190:46139 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.11.132.148:37633 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
274c7e6393224516bcafe58490c5987e | 127.11.132.130:46609 | HEALTHY | <none> | 0 | 0
7eb2f9678de6476c9f17aad6dc2a25ca | 127.11.132.129:35349 | HEALTHY | <none> | 1 | 0
e59346ea53fe40c0b8cca6aeaefe269a | 127.11.132.131:45543 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.11.132.129 | experimental | 127.11.132.129:35349
local_ip_for_outbound_sockets | 127.11.132.130 | experimental | 127.11.132.130:46609
local_ip_for_outbound_sockets | 127.11.132.131 | experimental | 127.11.132.131:45543
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-0/data/info.pb | hidden | 127.11.132.129:35349
server_dump_info_path | /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-1/data/info.pb | hidden | 127.11.132.130:46609
server_dump_info_path | /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-2/data/info.pb | hidden | 127.11.132.131:45543
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.11.132.148:37633 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------+----+---------+---------------+---------+------------+------------------+-------------
pre_rebuild | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 0
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 1
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250814 13:59:58.506937 11794 log_verifier.cc:126] Checking tablet 53f2621a8d3147f18044b93f74c391d3
I20250814 13:59:58.785651 11794 log_verifier.cc:177] Verified matching terms for 208 ops in tablet 53f2621a8d3147f18044b93f74c391d3
I20250814 13:59:58.787832 18134 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40526:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250814 13:59:58.788357 18134 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40526:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250814 13:59:58.799886 18134 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 77c45e5eeed74f7e861f72e101ea9f5c: Sending DeleteTablet for 1 replicas of tablet 53f2621a8d3147f18044b93f74c391d3
I20250814 13:59:58.801632 11794 test_util.cc:276] Using random seed: -1336381752
I20250814 13:59:58.801581 18257 tablet_service.cc:1515] Processing DeleteTablet for tablet 53f2621a8d3147f18044b93f74c391d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-08-14 13:59:58 UTC) from {username='slave'} at 127.0.0.1:33242
I20250814 13:59:58.804541 18671 tablet_replica.cc:331] stopping tablet replica
I20250814 13:59:58.805373 18671 raft_consensus.cc:2241] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [term 2 LEADER]: Raft consensus shutting down.
I20250814 13:59:58.806036 18671 raft_consensus.cc:2270] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca [term 2 FOLLOWER]: Raft consensus is shut down!
I20250814 13:59:58.836438 18134 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40564:
name: "post_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250814 13:59:58.837213 18671 ts_tablet_manager.cc:1905] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250814 13:59:58.838881 18134 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table post_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250814 13:59:58.851953 18671 ts_tablet_manager.cc:1918] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.208
I20250814 13:59:58.852289 18671 log.cc:1199] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca: Deleting WAL directory at /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-0/wal/wals/53f2621a8d3147f18044b93f74c391d3
I20250814 13:59:58.853205 18671 ts_tablet_manager.cc:1939] T 53f2621a8d3147f18044b93f74c391d3 P 7eb2f9678de6476c9f17aad6dc2a25ca: Deleting consensus metadata
I20250814 13:59:58.855934 18120 catalog_manager.cc:4928] TS 7eb2f9678de6476c9f17aad6dc2a25ca (127.11.132.129:35349): tablet 53f2621a8d3147f18044b93f74c391d3 (table pre_rebuild [id=bb010748db0d4382aed9a4b91f12f538]) successfully deleted
I20250814 13:59:58.860430 18393 tablet_service.cc:1468] Processing CreateTablet for tablet 05876933e7fe41c4b636274402511519 (DEFAULT_TABLE table=post_rebuild [id=27fd6aee80d5401792030ed6ac1bf909]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 13:59:58.861159 18533 tablet_service.cc:1468] Processing CreateTablet for tablet 05876933e7fe41c4b636274402511519 (DEFAULT_TABLE table=post_rebuild [id=27fd6aee80d5401792030ed6ac1bf909]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 13:59:58.861761 18393 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05876933e7fe41c4b636274402511519. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:59:58.862320 18533 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05876933e7fe41c4b636274402511519. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:59:58.867525 18257 tablet_service.cc:1468] Processing CreateTablet for tablet 05876933e7fe41c4b636274402511519 (DEFAULT_TABLE table=post_rebuild [id=27fd6aee80d5401792030ed6ac1bf909]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 13:59:58.868733 18257 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05876933e7fe41c4b636274402511519. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 13:59:58.880426 18679 tablet_bootstrap.cc:492] T 05876933e7fe41c4b636274402511519 P e59346ea53fe40c0b8cca6aeaefe269a: Bootstrap starting.
I20250814 13:59:58.883106 18680 tablet_bootstrap.cc:492] T 05876933e7fe41c4b636274402511519 P 274c7e6393224516bcafe58490c5987e: Bootstrap starting.
I20250814 13:59:58.886838 18679 tablet_bootstrap.cc:654] T 05876933e7fe41c4b636274402511519 P e59346ea53fe40c0b8cca6aeaefe269a: Neither blocks nor log segments found. Creating new log.
I20250814 13:59:58.887230 18680 tablet_bootstrap.cc:654] T 05876933e7fe41c4b636274402511519 P 274c7e6393224516bcafe58490c5987e: Neither blocks nor log segments found. Creating new log.
I20250814 13:59:58.893093 18679 tablet_bootstrap.cc:492] T 05876933e7fe41c4b636274402511519 P e59346ea53fe40c0b8cca6aeaefe269a: No bootstrap required, opened a new log
I20250814 13:59:58.893476 18679 ts_tablet_manager.cc:1397] T 05876933e7fe41c4b636274402511519 P e59346ea53fe40c0b8cca6aeaefe269a: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20250814 13:59:58.895131 18680 tablet_bootstrap.cc:492] T 05876933e7fe41c4b636274402511519 P 274c7e6393224516bcafe58490c5987e: No bootstrap required, opened a new log
I20250814 13:59:58.895984 18680 ts_tablet_manager.cc:1397] T 05876933e7fe41c4b636274402511519 P 274c7e6393224516bcafe58490c5987e: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.005s
I20250814 13:59:58.898866 18680 raft_consensus.cc:357] T 05876933e7fe41c4b636274402511519 P 274c7e6393224516bcafe58490c5987e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274c7e6393224516bcafe58490c5987e" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46609 } } peers { permanent_uuid: "e59346ea53fe40c0b8cca6aeaefe269a" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 45543 } } peers { permanent_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 35349 } }
I20250814 13:59:58.899571 18680 raft_consensus.cc:383] T 05876933e7fe41c4b636274402511519 P 274c7e6393224516bcafe58490c5987e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 13:59:58.899932 18680 raft_consensus.cc:738] T 05876933e7fe41c4b636274402511519 P 274c7e6393224516bcafe58490c5987e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 274c7e6393224516bcafe58490c5987e, State: Initialized, Role: FOLLOWER
I20250814 13:59:58.900738 18679 raft_consensus.cc:357] T 05876933e7fe41c4b636274402511519 P e59346ea53fe40c0b8cca6aeaefe269a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274c7e6393224516bcafe58490c5987e" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46609 } } peers { permanent_uuid: "e59346ea53fe40c0b8cca6aeaefe269a" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 45543 } } peers { permanent_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 35349 } }
I20250814 13:59:58.900792 18680 consensus_queue.cc:260] T 05876933e7fe41c4b636274402511519 P 274c7e6393224516bcafe58490c5987e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274c7e6393224516bcafe58490c5987e" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46609 } } peers { permanent_uuid: "e59346ea53fe40c0b8cca6aeaefe269a" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 45543 } } peers { permanent_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 35349 } }
I20250814 13:59:58.901448 18679 raft_consensus.cc:383] T 05876933e7fe41c4b636274402511519 P e59346ea53fe40c0b8cca6aeaefe269a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 13:59:58.901757 18679 raft_consensus.cc:738] T 05876933e7fe41c4b636274402511519 P e59346ea53fe40c0b8cca6aeaefe269a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59346ea53fe40c0b8cca6aeaefe269a, State: Initialized, Role: FOLLOWER
I20250814 13:59:58.902431 18679 consensus_queue.cc:260] T 05876933e7fe41c4b636274402511519 P e59346ea53fe40c0b8cca6aeaefe269a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274c7e6393224516bcafe58490c5987e" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46609 } } peers { permanent_uuid: "e59346ea53fe40c0b8cca6aeaefe269a" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 45543 } } peers { permanent_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 35349 } }
I20250814 13:59:58.910004 18680 ts_tablet_manager.cc:1428] T 05876933e7fe41c4b636274402511519 P 274c7e6393224516bcafe58490c5987e: Time spent starting tablet: real 0.014s user 0.005s sys 0.007s
I20250814 13:59:58.911746 18685 tablet_bootstrap.cc:492] T 05876933e7fe41c4b636274402511519 P 7eb2f9678de6476c9f17aad6dc2a25ca: Bootstrap starting.
I20250814 13:59:58.920270 18685 tablet_bootstrap.cc:654] T 05876933e7fe41c4b636274402511519 P 7eb2f9678de6476c9f17aad6dc2a25ca: Neither blocks nor log segments found. Creating new log.
I20250814 13:59:58.928297 18679 ts_tablet_manager.cc:1428] T 05876933e7fe41c4b636274402511519 P e59346ea53fe40c0b8cca6aeaefe269a: Time spent starting tablet: real 0.035s user 0.007s sys 0.019s
I20250814 13:59:58.932291 18685 tablet_bootstrap.cc:492] T 05876933e7fe41c4b636274402511519 P 7eb2f9678de6476c9f17aad6dc2a25ca: No bootstrap required, opened a new log
I20250814 13:59:58.932675 18685 ts_tablet_manager.cc:1397] T 05876933e7fe41c4b636274402511519 P 7eb2f9678de6476c9f17aad6dc2a25ca: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.011s
I20250814 13:59:58.935876 18685 raft_consensus.cc:357] T 05876933e7fe41c4b636274402511519 P 7eb2f9678de6476c9f17aad6dc2a25ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274c7e6393224516bcafe58490c5987e" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46609 } } peers { permanent_uuid: "e59346ea53fe40c0b8cca6aeaefe269a" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 45543 } } peers { permanent_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 35349 } }
I20250814 13:59:58.936836 18685 raft_consensus.cc:383] T 05876933e7fe41c4b636274402511519 P 7eb2f9678de6476c9f17aad6dc2a25ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 13:59:58.937142 18685 raft_consensus.cc:738] T 05876933e7fe41c4b636274402511519 P 7eb2f9678de6476c9f17aad6dc2a25ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb2f9678de6476c9f17aad6dc2a25ca, State: Initialized, Role: FOLLOWER
I20250814 13:59:58.937817 18685 consensus_queue.cc:260] T 05876933e7fe41c4b636274402511519 P 7eb2f9678de6476c9f17aad6dc2a25ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274c7e6393224516bcafe58490c5987e" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46609 } } peers { permanent_uuid: "e59346ea53fe40c0b8cca6aeaefe269a" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 45543 } } peers { permanent_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 35349 } }
I20250814 13:59:58.940539 18685 ts_tablet_manager.cc:1428] T 05876933e7fe41c4b636274402511519 P 7eb2f9678de6476c9f17aad6dc2a25ca: Time spent starting tablet: real 0.007s user 0.000s sys 0.006s
W20250814 13:59:59.006767 18323 tablet.cc:2378] T 05876933e7fe41c4b636274402511519 P 7eb2f9678de6476c9f17aad6dc2a25ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250814 13:59:59.102351 18683 raft_consensus.cc:491] T 05876933e7fe41c4b636274402511519 P 274c7e6393224516bcafe58490c5987e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250814 13:59:59.102733 18683 raft_consensus.cc:513] T 05876933e7fe41c4b636274402511519 P 274c7e6393224516bcafe58490c5987e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274c7e6393224516bcafe58490c5987e" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46609 } } peers { permanent_uuid: "e59346ea53fe40c0b8cca6aeaefe269a" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 45543 } } peers { permanent_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 35349 } }
I20250814 13:59:59.105207 18683 leader_election.cc:290] T 05876933e7fe41c4b636274402511519 P 274c7e6393224516bcafe58490c5987e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e59346ea53fe40c0b8cca6aeaefe269a (127.11.132.131:45543), 7eb2f9678de6476c9f17aad6dc2a25ca (127.11.132.129:35349)
W20250814 13:59:59.106241 18603 tablet.cc:2378] T 05876933e7fe41c4b636274402511519 P e59346ea53fe40c0b8cca6aeaefe269a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250814 13:59:59.119494 18459 tablet.cc:2378] T 05876933e7fe41c4b636274402511519 P 274c7e6393224516bcafe58490c5987e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250814 13:59:59.120260 18553 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05876933e7fe41c4b636274402511519" candidate_uuid: "274c7e6393224516bcafe58490c5987e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59346ea53fe40c0b8cca6aeaefe269a" is_pre_election: true
I20250814 13:59:59.120922 18553 raft_consensus.cc:2466] T 05876933e7fe41c4b636274402511519 P e59346ea53fe40c0b8cca6aeaefe269a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 274c7e6393224516bcafe58490c5987e in term 0.
I20250814 13:59:59.122071 18348 leader_election.cc:304] T 05876933e7fe41c4b636274402511519 P 274c7e6393224516bcafe58490c5987e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 274c7e6393224516bcafe58490c5987e, e59346ea53fe40c0b8cca6aeaefe269a; no voters:
I20250814 13:59:59.122694 18683 raft_consensus.cc:2802] T 05876933e7fe41c4b636274402511519 P 274c7e6393224516bcafe58490c5987e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250814 13:59:59.122917 18683 raft_consensus.cc:491] T 05876933e7fe41c4b636274402511519 P 274c7e6393224516bcafe58490c5987e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250814 13:59:59.123142 18683 raft_consensus.cc:3058] T 05876933e7fe41c4b636274402511519 P 274c7e6393224516bcafe58490c5987e [term 0 FOLLOWER]: Advancing to term 1
I20250814 13:59:59.124605 18277 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05876933e7fe41c4b636274402511519" candidate_uuid: "274c7e6393224516bcafe58490c5987e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" is_pre_election: true
I20250814 13:59:59.125604 18277 raft_consensus.cc:2466] T 05876933e7fe41c4b636274402511519 P 7eb2f9678de6476c9f17aad6dc2a25ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 274c7e6393224516bcafe58490c5987e in term 0.
I20250814 13:59:59.128604 18683 raft_consensus.cc:513] T 05876933e7fe41c4b636274402511519 P 274c7e6393224516bcafe58490c5987e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274c7e6393224516bcafe58490c5987e" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46609 } } peers { permanent_uuid: "e59346ea53fe40c0b8cca6aeaefe269a" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 45543 } } peers { permanent_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 35349 } }
I20250814 13:59:59.130003 18683 leader_election.cc:290] T 05876933e7fe41c4b636274402511519 P 274c7e6393224516bcafe58490c5987e [CANDIDATE]: Term 1 election: Requested vote from peers e59346ea53fe40c0b8cca6aeaefe269a (127.11.132.131:45543), 7eb2f9678de6476c9f17aad6dc2a25ca (127.11.132.129:35349)
I20250814 13:59:59.130647 18553 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05876933e7fe41c4b636274402511519" candidate_uuid: "274c7e6393224516bcafe58490c5987e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59346ea53fe40c0b8cca6aeaefe269a"
I20250814 13:59:59.131043 18553 raft_consensus.cc:3058] T 05876933e7fe41c4b636274402511519 P e59346ea53fe40c0b8cca6aeaefe269a [term 0 FOLLOWER]: Advancing to term 1
I20250814 13:59:59.130901 18277 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05876933e7fe41c4b636274402511519" candidate_uuid: "274c7e6393224516bcafe58490c5987e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca"
I20250814 13:59:59.131347 18277 raft_consensus.cc:3058] T 05876933e7fe41c4b636274402511519 P 7eb2f9678de6476c9f17aad6dc2a25ca [term 0 FOLLOWER]: Advancing to term 1
I20250814 13:59:59.135210 18553 raft_consensus.cc:2466] T 05876933e7fe41c4b636274402511519 P e59346ea53fe40c0b8cca6aeaefe269a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 274c7e6393224516bcafe58490c5987e in term 1.
I20250814 13:59:59.135854 18348 leader_election.cc:304] T 05876933e7fe41c4b636274402511519 P 274c7e6393224516bcafe58490c5987e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 274c7e6393224516bcafe58490c5987e, e59346ea53fe40c0b8cca6aeaefe269a; no voters:
I20250814 13:59:59.136323 18683 raft_consensus.cc:2802] T 05876933e7fe41c4b636274402511519 P 274c7e6393224516bcafe58490c5987e [term 1 FOLLOWER]: Leader election won for term 1
I20250814 13:59:59.137468 18277 raft_consensus.cc:2466] T 05876933e7fe41c4b636274402511519 P 7eb2f9678de6476c9f17aad6dc2a25ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 274c7e6393224516bcafe58490c5987e in term 1.
I20250814 13:59:59.138144 18683 raft_consensus.cc:695] T 05876933e7fe41c4b636274402511519 P 274c7e6393224516bcafe58490c5987e [term 1 LEADER]: Becoming Leader. State: Replica: 274c7e6393224516bcafe58490c5987e, State: Running, Role: LEADER
I20250814 13:59:59.139267 18683 consensus_queue.cc:237] T 05876933e7fe41c4b636274402511519 P 274c7e6393224516bcafe58490c5987e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274c7e6393224516bcafe58490c5987e" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46609 } } peers { permanent_uuid: "e59346ea53fe40c0b8cca6aeaefe269a" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 45543 } } peers { permanent_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 35349 } }
I20250814 13:59:59.146917 18134 catalog_manager.cc:5582] T 05876933e7fe41c4b636274402511519 P 274c7e6393224516bcafe58490c5987e reported cstate change: term changed from 0 to 1, leader changed from <none> to 274c7e6393224516bcafe58490c5987e (127.11.132.130). New cstate: current_term: 1 leader_uuid: "274c7e6393224516bcafe58490c5987e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274c7e6393224516bcafe58490c5987e" member_type: VOTER last_known_addr { host: "127.11.132.130" port: 46609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59346ea53fe40c0b8cca6aeaefe269a" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 45543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 35349 } health_report { overall_health: UNKNOWN } } }
I20250814 13:59:59.314654 18553 raft_consensus.cc:1273] T 05876933e7fe41c4b636274402511519 P e59346ea53fe40c0b8cca6aeaefe269a [term 1 FOLLOWER]: Refusing update from remote peer 274c7e6393224516bcafe58490c5987e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250814 13:59:59.316744 18277 raft_consensus.cc:1273] T 05876933e7fe41c4b636274402511519 P 7eb2f9678de6476c9f17aad6dc2a25ca [term 1 FOLLOWER]: Refusing update from remote peer 274c7e6393224516bcafe58490c5987e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250814 13:59:59.318209 18692 consensus_queue.cc:1035] T 05876933e7fe41c4b636274402511519 P 274c7e6393224516bcafe58490c5987e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59346ea53fe40c0b8cca6aeaefe269a" member_type: VOTER last_known_addr { host: "127.11.132.131" port: 45543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250814 13:59:59.321002 18683 consensus_queue.cc:1035] T 05876933e7fe41c4b636274402511519 P 274c7e6393224516bcafe58490c5987e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb2f9678de6476c9f17aad6dc2a25ca" member_type: VOTER last_known_addr { host: "127.11.132.129" port: 35349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250814 13:59:59.353996 18700 mvcc.cc:204] Tried to move back new op lower bound from 7189217277176619008 to 7189217276481110016. Current Snapshot: MvccSnapshot[applied={T|T < 7189217277176619008}]
I20250814 14:00:03.917179 18393 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250814 14:00:03.927927 18257 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250814 14:00:03.928870 18533 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
77c45e5eeed74f7e861f72e101ea9f5c | 127.11.132.190:46139 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.11.132.148:37633 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
274c7e6393224516bcafe58490c5987e | 127.11.132.130:46609 | HEALTHY | <none> | 1 | 0
7eb2f9678de6476c9f17aad6dc2a25ca | 127.11.132.129:35349 | HEALTHY | <none> | 0 | 0
e59346ea53fe40c0b8cca6aeaefe269a | 127.11.132.131:45543 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.11.132.129 | experimental | 127.11.132.129:35349
local_ip_for_outbound_sockets | 127.11.132.130 | experimental | 127.11.132.130:46609
local_ip_for_outbound_sockets | 127.11.132.131 | experimental | 127.11.132.131:45543
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-0/data/info.pb | hidden | 127.11.132.129:35349
server_dump_info_path | /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-1/data/info.pb | hidden | 127.11.132.130:46609
server_dump_info_path | /tmp/dist-test-taskUXIPqC/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755179819681315-11794-0/minicluster-data/ts-2/data/info.pb | hidden | 127.11.132.131:45543
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.11.132.148:37633 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
--------------+----+---------+---------------+---------+------------+------------------+-------------
post_rebuild | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250814 14:00:04.141395 11794 log_verifier.cc:126] Checking tablet 05876933e7fe41c4b636274402511519
I20250814 14:00:04.839533 11794 log_verifier.cc:177] Verified matching terms for 205 ops in tablet 05876933e7fe41c4b636274402511519
I20250814 14:00:04.840415 11794 log_verifier.cc:126] Checking tablet 53f2621a8d3147f18044b93f74c391d3
I20250814 14:00:04.840695 11794 log_verifier.cc:177] Verified matching terms for 0 ops in tablet 53f2621a8d3147f18044b93f74c391d3
I20250814 14:00:04.864336 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 18171
I20250814 14:00:04.909696 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 18326
I20250814 14:00:04.953856 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 18462
I20250814 14:00:04.986428 11794 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUXIPqC/build/tsan/bin/kudu with pid 18101
2025-08-14T14:00:05Z chronyd exiting
[ OK ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0 (34821 ms)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest (34821 ms total)
[----------] Global test environment tear-down
[==========] 9 tests from 5 test suites ran. (185291 ms total)
[ PASSED ] 8 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AdminCliTest.TestRebuildTables
1 FAILED TEST