Note: This is test shard 6 of 8.
[==========] Running 9 tests from 5 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from AdminCliTest
[ RUN ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250815 13:52:42.228206 426 test_util.cc:276] Using random seed: -1272301229
W20250815 13:52:43.403993 426 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.127s user 0.477s sys 0.649s
W20250815 13:52:43.404470 426 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.127s user 0.477s sys 0.649s
I20250815 13:52:43.407497 426 ts_itest-base.cc:115] Starting cluster with:
I20250815 13:52:43.407769 426 ts_itest-base.cc:116] --------------
I20250815 13:52:43.407996 426 ts_itest-base.cc:117] 4 tablet servers
I20250815 13:52:43.408219 426 ts_itest-base.cc:118] 3 replicas per TS
I20250815 13:52:43.408409 426 ts_itest-base.cc:119] --------------
2025-08-15T13:52:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-15T13:52:43Z Disabled control of system clock
I20250815 13:52:43.464772 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.190:44703
--webserver_interface=127.0.106.190
--webserver_port=0
--builtin_ntp_servers=127.0.106.148:44757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.106.190:44703 with env {}
W20250815 13:52:43.813657 440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:52:43.814267 440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:52:43.814836 440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:52:43.847787 440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 13:52:43.848093 440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:52:43.848363 440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 13:52:43.848590 440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 13:52:43.884294 440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:44757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.106.190:44703
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.190:44703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.106.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:52:43.885771 440 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:52:43.887563 440 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:52:43.900545 446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:52:43.905929 449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:52:43.902565 447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:52:45.071424 448 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1165 milliseconds
I20250815 13:52:45.071549 440 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 13:52:45.072892 440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:52:45.076149 440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:52:45.077584 440 hybrid_clock.cc:648] HybridClock initialized: now 1755265965077531 us; error 89 us; skew 500 ppm
I20250815 13:52:45.078539 440 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:52:45.085951 440 webserver.cc:480] Webserver started at http://127.0.106.190:41297/ using document root <none> and password file <none>
I20250815 13:52:45.086979 440 fs_manager.cc:362] Metadata directory not provided
I20250815 13:52:45.087266 440 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:52:45.087762 440 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 13:52:45.092404 440 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "17c68a6894774ffcb876c4ba45e7e7c9"
format_stamp: "Formatted at 2025-08-15 13:52:45 on dist-test-slave-kt02"
I20250815 13:52:45.093910 440 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "17c68a6894774ffcb876c4ba45e7e7c9"
format_stamp: "Formatted at 2025-08-15 13:52:45 on dist-test-slave-kt02"
I20250815 13:52:45.101902 440 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20250815 13:52:45.108480 456 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:52:45.109930 440 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.003s
I20250815 13:52:45.110369 440 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
uuid: "17c68a6894774ffcb876c4ba45e7e7c9"
format_stamp: "Formatted at 2025-08-15 13:52:45 on dist-test-slave-kt02"
I20250815 13:52:45.110756 440 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:52:45.163795 440 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:52:45.165319 440 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:52:45.165789 440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:52:45.258419 440 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.190:44703
I20250815 13:52:45.258566 507 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.190:44703 every 8 connection(s)
I20250815 13:52:45.261274 440 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250815 13:52:45.268510 508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:52:45.272126 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 440
I20250815 13:52:45.272541 426 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250815 13:52:45.293568 508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9: Bootstrap starting.
I20250815 13:52:45.300597 508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9: Neither blocks nor log segments found. Creating new log.
I20250815 13:52:45.302680 508 log.cc:826] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9: Log is configured to *not* fsync() on all Append() calls
I20250815 13:52:45.308760 508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9: No bootstrap required, opened a new log
I20250815 13:52:45.330878 508 raft_consensus.cc:357] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c68a6894774ffcb876c4ba45e7e7c9" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 44703 } }
I20250815 13:52:45.331661 508 raft_consensus.cc:383] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 13:52:45.331871 508 raft_consensus.cc:738] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17c68a6894774ffcb876c4ba45e7e7c9, State: Initialized, Role: FOLLOWER
I20250815 13:52:45.332553 508 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c68a6894774ffcb876c4ba45e7e7c9" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 44703 } }
I20250815 13:52:45.333065 508 raft_consensus.cc:397] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 13:52:45.333297 508 raft_consensus.cc:491] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 13:52:45.333580 508 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9 [term 0 FOLLOWER]: Advancing to term 1
I20250815 13:52:45.338603 508 raft_consensus.cc:513] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c68a6894774ffcb876c4ba45e7e7c9" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 44703 } }
I20250815 13:52:45.339443 508 leader_election.cc:304] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17c68a6894774ffcb876c4ba45e7e7c9; no voters:
I20250815 13:52:45.341135 508 leader_election.cc:290] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 13:52:45.342727 513 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 13:52:45.346149 513 raft_consensus.cc:695] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9 [term 1 LEADER]: Becoming Leader. State: Replica: 17c68a6894774ffcb876c4ba45e7e7c9, State: Running, Role: LEADER
I20250815 13:52:45.347131 513 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c68a6894774ffcb876c4ba45e7e7c9" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 44703 } }
I20250815 13:52:45.348132 508 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9 [sys.catalog]: configured and running, proceeding with master startup.
I20250815 13:52:45.364554 514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17c68a6894774ffcb876c4ba45e7e7c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c68a6894774ffcb876c4ba45e7e7c9" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 44703 } } }
I20250815 13:52:45.364881 515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17c68a6894774ffcb876c4ba45e7e7c9. Latest consensus state: current_term: 1 leader_uuid: "17c68a6894774ffcb876c4ba45e7e7c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c68a6894774ffcb876c4ba45e7e7c9" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 44703 } } }
I20250815 13:52:45.366298 514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9 [sys.catalog]: This master's current role is: LEADER
I20250815 13:52:45.366545 515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9 [sys.catalog]: This master's current role is: LEADER
I20250815 13:52:45.369499 523 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 13:52:45.387069 523 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 13:52:45.404418 523 catalog_manager.cc:1349] Generated new cluster ID: dbce9e72c486460095a75d353e87d5a9
I20250815 13:52:45.404786 523 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 13:52:45.432581 523 catalog_manager.cc:1372] Generated new certificate authority record
I20250815 13:52:45.434116 523 catalog_manager.cc:1506] Loading token signing keys...
I20250815 13:52:45.456871 523 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9: Generated new TSK 0
I20250815 13:52:45.457881 523 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250815 13:52:45.485880 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.129:0
--local_ip_for_outbound_sockets=127.0.106.129
--webserver_interface=127.0.106.129
--webserver_port=0
--tserver_master_addrs=127.0.106.190:44703
--builtin_ntp_servers=127.0.106.148:44757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250815 13:52:45.860823 532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:52:45.861423 532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:52:45.861954 532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:52:45.897827 532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:52:45.898803 532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.129
I20250815 13:52:45.942679 532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:44757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.106.129
--webserver_port=0
--tserver_master_addrs=127.0.106.190:44703
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.129
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:52:45.944708 532 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:52:45.946936 532 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:52:45.964884 538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:52:45.966004 539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:52:45.976047 541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 13:52:45.973464 532 server_base.cc:1047] running on GCE node
I20250815 13:52:47.103955 532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:52:47.106741 532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:52:47.108270 532 hybrid_clock.cc:648] HybridClock initialized: now 1755265967108246 us; error 73 us; skew 500 ppm
I20250815 13:52:47.109323 532 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:52:47.117213 532 webserver.cc:480] Webserver started at http://127.0.106.129:41153/ using document root <none> and password file <none>
I20250815 13:52:47.118495 532 fs_manager.cc:362] Metadata directory not provided
I20250815 13:52:47.118774 532 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:52:47.119411 532 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 13:52:47.126024 532 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "87db2b33cd35440fae9f366e99a1b6e2"
format_stamp: "Formatted at 2025-08-15 13:52:47 on dist-test-slave-kt02"
I20250815 13:52:47.127571 532 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "87db2b33cd35440fae9f366e99a1b6e2"
format_stamp: "Formatted at 2025-08-15 13:52:47 on dist-test-slave-kt02"
I20250815 13:52:47.137245 532 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.005s sys 0.003s
I20250815 13:52:47.145249 548 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:52:47.146807 532 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.004s
I20250815 13:52:47.147310 532 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "87db2b33cd35440fae9f366e99a1b6e2"
format_stamp: "Formatted at 2025-08-15 13:52:47 on dist-test-slave-kt02"
I20250815 13:52:47.147758 532 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:52:47.214816 532 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:52:47.216933 532 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:52:47.217525 532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:52:47.221362 532 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 13:52:47.227111 532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 13:52:47.227396 532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20250815 13:52:47.227706 532 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 13:52:47.227919 532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 13:52:47.415639 532 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.129:38637
I20250815 13:52:47.415798 660 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.129:38637 every 8 connection(s)
I20250815 13:52:47.418484 532 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250815 13:52:47.421188 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 532
I20250815 13:52:47.421730 426 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250815 13:52:47.429963 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.130:0
--local_ip_for_outbound_sockets=127.0.106.130
--webserver_interface=127.0.106.130
--webserver_port=0
--tserver_master_addrs=127.0.106.190:44703
--builtin_ntp_servers=127.0.106.148:44757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 13:52:47.443795 661 heartbeater.cc:344] Connected to a master server at 127.0.106.190:44703
I20250815 13:52:47.444522 661 heartbeater.cc:461] Registering TS with master...
I20250815 13:52:47.446142 661 heartbeater.cc:507] Master 127.0.106.190:44703 requested a full tablet report, sending...
I20250815 13:52:47.449566 473 ts_manager.cc:194] Registered new tserver with Master: 87db2b33cd35440fae9f366e99a1b6e2 (127.0.106.129:38637)
I20250815 13:52:47.452194 473 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.129:55115
W20250815 13:52:47.781760 665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:52:47.782562 665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:52:47.783174 665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:52:47.817711 665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:52:47.818615 665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.130
I20250815 13:52:47.856778 665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:44757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.106.130
--webserver_port=0
--tserver_master_addrs=127.0.106.190:44703
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.130
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:52:47.858326 665 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:52:47.860105 665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:52:47.874429 671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 13:52:48.456863 661 heartbeater.cc:499] Master 127.0.106.190:44703 was elected leader, sending a full tablet report...
W20250815 13:52:47.875890 672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:52:49.059638 674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:52:49.061378 673 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1185 milliseconds
W20250815 13:52:49.062554 665 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.187s user 0.435s sys 0.736s
W20250815 13:52:49.062970 665 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.188s user 0.435s sys 0.736s
I20250815 13:52:49.063287 665 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 13:52:49.065083 665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:52:49.069496 665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:52:49.073308 665 hybrid_clock.cc:648] HybridClock initialized: now 1755265969073016 us; error 252 us; skew 500 ppm
I20250815 13:52:49.074618 665 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:52:49.086951 665 webserver.cc:480] Webserver started at http://127.0.106.130:38135/ using document root <none> and password file <none>
I20250815 13:52:49.088544 665 fs_manager.cc:362] Metadata directory not provided
I20250815 13:52:49.088865 665 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:52:49.089543 665 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 13:52:49.097033 665 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "fe93b76dfb244e45bfc87202827f6acd"
format_stamp: "Formatted at 2025-08-15 13:52:49 on dist-test-slave-kt02"
I20250815 13:52:49.098870 665 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "fe93b76dfb244e45bfc87202827f6acd"
format_stamp: "Formatted at 2025-08-15 13:52:49 on dist-test-slave-kt02"
I20250815 13:52:49.110620 665 fs_manager.cc:696] Time spent creating directory manager: real 0.011s user 0.005s sys 0.005s
I20250815 13:52:49.120169 681 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:52:49.121497 665 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.002s sys 0.003s
I20250815 13:52:49.121876 665 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "fe93b76dfb244e45bfc87202827f6acd"
format_stamp: "Formatted at 2025-08-15 13:52:49 on dist-test-slave-kt02"
I20250815 13:52:49.122236 665 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:52:49.224504 665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:52:49.226207 665 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:52:49.226683 665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:52:49.230289 665 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 13:52:49.235035 665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 13:52:49.235303 665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250815 13:52:49.235586 665 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 13:52:49.235738 665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 13:52:49.422122 665 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.130:36813
I20250815 13:52:49.422356 793 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.130:36813 every 8 connection(s)
I20250815 13:52:49.425073 665 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250815 13:52:49.432762 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 665
I20250815 13:52:49.433241 426 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250815 13:52:49.440202 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.131:0
--local_ip_for_outbound_sockets=127.0.106.131
--webserver_interface=127.0.106.131
--webserver_port=0
--tserver_master_addrs=127.0.106.190:44703
--builtin_ntp_servers=127.0.106.148:44757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 13:52:49.456707 794 heartbeater.cc:344] Connected to a master server at 127.0.106.190:44703
I20250815 13:52:49.457473 794 heartbeater.cc:461] Registering TS with master...
I20250815 13:52:49.459148 794 heartbeater.cc:507] Master 127.0.106.190:44703 requested a full tablet report, sending...
I20250815 13:52:49.462158 473 ts_manager.cc:194] Registered new tserver with Master: fe93b76dfb244e45bfc87202827f6acd (127.0.106.130:36813)
I20250815 13:52:49.464280 473 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.130:52711
W20250815 13:52:49.802497 798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:52:49.803174 798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:52:49.803706 798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:52:49.837682 798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:52:49.838704 798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.131
I20250815 13:52:49.879112 798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:44757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.106.131
--webserver_port=0
--tserver_master_addrs=127.0.106.190:44703
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.131
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:52:49.880756 798 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:52:49.882674 798 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:52:49.899228 804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 13:52:50.469980 794 heartbeater.cc:499] Master 127.0.106.190:44703 was elected leader, sending a full tablet report...
W20250815 13:52:49.900343 805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:52:51.279170 807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:52:51.282073 806 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1381 milliseconds
W20250815 13:52:51.283407 798 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.384s user 0.507s sys 0.849s
W20250815 13:52:51.283838 798 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.384s user 0.507s sys 0.849s
I20250815 13:52:51.284214 798 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 13:52:51.285817 798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:52:51.289896 798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:52:51.291409 798 hybrid_clock.cc:648] HybridClock initialized: now 1755265971291356 us; error 43 us; skew 500 ppm
I20250815 13:52:51.292642 798 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:52:51.303761 798 webserver.cc:480] Webserver started at http://127.0.106.131:33495/ using document root <none> and password file <none>
I20250815 13:52:51.305450 798 fs_manager.cc:362] Metadata directory not provided
I20250815 13:52:51.305816 798 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:52:51.306499 798 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 13:52:51.313958 798 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "ed679beae3fd46d0b830a37ec01a3e29"
format_stamp: "Formatted at 2025-08-15 13:52:51 on dist-test-slave-kt02"
I20250815 13:52:51.315790 798 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "ed679beae3fd46d0b830a37ec01a3e29"
format_stamp: "Formatted at 2025-08-15 13:52:51 on dist-test-slave-kt02"
I20250815 13:52:51.327818 798 fs_manager.cc:696] Time spent creating directory manager: real 0.011s user 0.001s sys 0.009s
I20250815 13:52:51.336865 814 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:52:51.338426 798 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.002s sys 0.002s
I20250815 13:52:51.338908 798 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ed679beae3fd46d0b830a37ec01a3e29"
format_stamp: "Formatted at 2025-08-15 13:52:51 on dist-test-slave-kt02"
I20250815 13:52:51.339440 798 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:52:51.441424 798 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:52:51.443171 798 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:52:51.443653 798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:52:51.446543 798 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 13:52:51.452077 798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 13:52:51.452337 798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 13:52:51.452595 798 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 13:52:51.452757 798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 13:52:51.654834 798 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.131:37049
I20250815 13:52:51.654932 926 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.131:37049 every 8 connection(s)
I20250815 13:52:51.657821 798 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250815 13:52:51.667945 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 798
I20250815 13:52:51.668324 426 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250815 13:52:51.674085 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.132:0
--local_ip_for_outbound_sockets=127.0.106.132
--webserver_interface=127.0.106.132
--webserver_port=0
--tserver_master_addrs=127.0.106.190:44703
--builtin_ntp_servers=127.0.106.148:44757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 13:52:51.684092 927 heartbeater.cc:344] Connected to a master server at 127.0.106.190:44703
I20250815 13:52:51.684746 927 heartbeater.cc:461] Registering TS with master...
I20250815 13:52:51.686190 927 heartbeater.cc:507] Master 127.0.106.190:44703 requested a full tablet report, sending...
I20250815 13:52:51.689079 473 ts_manager.cc:194] Registered new tserver with Master: ed679beae3fd46d0b830a37ec01a3e29 (127.0.106.131:37049)
I20250815 13:52:51.690541 473 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.131:53127
W20250815 13:52:52.020762 931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:52:52.021379 931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:52:52.021895 931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:52:52.056208 931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:52:52.057178 931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.132
I20250815 13:52:52.097643 931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:44757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.106.132
--webserver_port=0
--tserver_master_addrs=127.0.106.190:44703
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.132
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:52:52.099330 931 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:52:52.101264 931 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:52:52.118795 937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 13:52:52.695693 927 heartbeater.cc:499] Master 127.0.106.190:44703 was elected leader, sending a full tablet report...
W20250815 13:52:52.121222 938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:52:53.520068 936 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 931
W20250815 13:52:53.663316 931 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.546s user 0.627s sys 0.891s
W20250815 13:52:53.664888 931 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.548s user 0.627s sys 0.891s
W20250815 13:52:53.666177 940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:52:53.670745 939 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1545 milliseconds
I20250815 13:52:53.670835 931 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 13:52:53.672055 931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:52:53.674288 931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:52:53.675649 931 hybrid_clock.cc:648] HybridClock initialized: now 1755265973675577 us; error 84 us; skew 500 ppm
I20250815 13:52:53.676478 931 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:52:53.683194 931 webserver.cc:480] Webserver started at http://127.0.106.132:39015/ using document root <none> and password file <none>
I20250815 13:52:53.684156 931 fs_manager.cc:362] Metadata directory not provided
I20250815 13:52:53.684373 931 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:52:53.684921 931 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 13:52:53.689908 931 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "2476a8890caa4741b9d6a49abda486e2"
format_stamp: "Formatted at 2025-08-15 13:52:53 on dist-test-slave-kt02"
I20250815 13:52:53.691089 931 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "2476a8890caa4741b9d6a49abda486e2"
format_stamp: "Formatted at 2025-08-15 13:52:53 on dist-test-slave-kt02"
I20250815 13:52:53.698719 931 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20250815 13:52:53.705866 948 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:52:53.707216 931 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20250815 13:52:53.707548 931 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "2476a8890caa4741b9d6a49abda486e2"
format_stamp: "Formatted at 2025-08-15 13:52:53 on dist-test-slave-kt02"
I20250815 13:52:53.707854 931 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:52:53.755679 931 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:52:53.757193 931 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:52:53.757727 931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:52:53.760365 931 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 13:52:53.764627 931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 13:52:53.764843 931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 13:52:53.765079 931 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 13:52:53.765244 931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 13:52:53.925400 931 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.132:38593
I20250815 13:52:53.925529 1061 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.132:38593 every 8 connection(s)
I20250815 13:52:53.928419 931 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250815 13:52:53.931406 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 931
I20250815 13:52:53.932085 426 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250815 13:52:53.955440 1062 heartbeater.cc:344] Connected to a master server at 127.0.106.190:44703
I20250815 13:52:53.956017 1062 heartbeater.cc:461] Registering TS with master...
I20250815 13:52:53.957443 1062 heartbeater.cc:507] Master 127.0.106.190:44703 requested a full tablet report, sending...
I20250815 13:52:53.960511 473 ts_manager.cc:194] Registered new tserver with Master: 2476a8890caa4741b9d6a49abda486e2 (127.0.106.132:38593)
I20250815 13:52:53.962002 473 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.132:57245
I20250815 13:52:53.972189 426 external_mini_cluster.cc:949] 4 TS(s) registered with all masters
I20250815 13:52:54.023499 473 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48272:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250815 13:52:54.109165 729 tablet_service.cc:1468] Processing CreateTablet for tablet 506e1def1e7f4fad82e88d07af8c5f51 (DEFAULT_TABLE table=TestTable [id=580f63afd698468c95da5702005badd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 13:52:54.112470 729 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 506e1def1e7f4fad82e88d07af8c5f51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:52:54.113330 997 tablet_service.cc:1468] Processing CreateTablet for tablet 506e1def1e7f4fad82e88d07af8c5f51 (DEFAULT_TABLE table=TestTable [id=580f63afd698468c95da5702005badd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 13:52:54.113922 862 tablet_service.cc:1468] Processing CreateTablet for tablet 506e1def1e7f4fad82e88d07af8c5f51 (DEFAULT_TABLE table=TestTable [id=580f63afd698468c95da5702005badd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 13:52:54.115285 997 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 506e1def1e7f4fad82e88d07af8c5f51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:52:54.115703 862 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 506e1def1e7f4fad82e88d07af8c5f51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:52:54.143146 1081 tablet_bootstrap.cc:492] T 506e1def1e7f4fad82e88d07af8c5f51 P ed679beae3fd46d0b830a37ec01a3e29: Bootstrap starting.
I20250815 13:52:54.146600 1083 tablet_bootstrap.cc:492] T 506e1def1e7f4fad82e88d07af8c5f51 P 2476a8890caa4741b9d6a49abda486e2: Bootstrap starting.
I20250815 13:52:54.149389 1082 tablet_bootstrap.cc:492] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd: Bootstrap starting.
I20250815 13:52:54.151566 1081 tablet_bootstrap.cc:654] T 506e1def1e7f4fad82e88d07af8c5f51 P ed679beae3fd46d0b830a37ec01a3e29: Neither blocks nor log segments found. Creating new log.
I20250815 13:52:54.154428 1081 log.cc:826] T 506e1def1e7f4fad82e88d07af8c5f51 P ed679beae3fd46d0b830a37ec01a3e29: Log is configured to *not* fsync() on all Append() calls
I20250815 13:52:54.156988 1083 tablet_bootstrap.cc:654] T 506e1def1e7f4fad82e88d07af8c5f51 P 2476a8890caa4741b9d6a49abda486e2: Neither blocks nor log segments found. Creating new log.
I20250815 13:52:54.159986 1083 log.cc:826] T 506e1def1e7f4fad82e88d07af8c5f51 P 2476a8890caa4741b9d6a49abda486e2: Log is configured to *not* fsync() on all Append() calls
I20250815 13:52:54.160056 1082 tablet_bootstrap.cc:654] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd: Neither blocks nor log segments found. Creating new log.
I20250815 13:52:54.162108 1081 tablet_bootstrap.cc:492] T 506e1def1e7f4fad82e88d07af8c5f51 P ed679beae3fd46d0b830a37ec01a3e29: No bootstrap required, opened a new log
I20250815 13:52:54.162775 1081 ts_tablet_manager.cc:1397] T 506e1def1e7f4fad82e88d07af8c5f51 P ed679beae3fd46d0b830a37ec01a3e29: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.016s
I20250815 13:52:54.163197 1082 log.cc:826] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd: Log is configured to *not* fsync() on all Append() calls
I20250815 13:52:54.167089 1083 tablet_bootstrap.cc:492] T 506e1def1e7f4fad82e88d07af8c5f51 P 2476a8890caa4741b9d6a49abda486e2: No bootstrap required, opened a new log
I20250815 13:52:54.167733 1083 ts_tablet_manager.cc:1397] T 506e1def1e7f4fad82e88d07af8c5f51 P 2476a8890caa4741b9d6a49abda486e2: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.008s
I20250815 13:52:54.169953 1082 tablet_bootstrap.cc:492] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd: No bootstrap required, opened a new log
I20250815 13:52:54.170516 1082 ts_tablet_manager.cc:1397] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd: Time spent bootstrapping tablet: real 0.022s user 0.014s sys 0.005s
I20250815 13:52:54.186034 1081 raft_consensus.cc:357] T 506e1def1e7f4fad82e88d07af8c5f51 P ed679beae3fd46d0b830a37ec01a3e29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 36813 } } peers { permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 } } peers { permanent_uuid: "ed679beae3fd46d0b830a37ec01a3e29" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 37049 } }
I20250815 13:52:54.186739 1081 raft_consensus.cc:383] T 506e1def1e7f4fad82e88d07af8c5f51 P ed679beae3fd46d0b830a37ec01a3e29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 13:52:54.187004 1081 raft_consensus.cc:738] T 506e1def1e7f4fad82e88d07af8c5f51 P ed679beae3fd46d0b830a37ec01a3e29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed679beae3fd46d0b830a37ec01a3e29, State: Initialized, Role: FOLLOWER
I20250815 13:52:54.187824 1081 consensus_queue.cc:260] T 506e1def1e7f4fad82e88d07af8c5f51 P ed679beae3fd46d0b830a37ec01a3e29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 36813 } } peers { permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 } } peers { permanent_uuid: "ed679beae3fd46d0b830a37ec01a3e29" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 37049 } }
I20250815 13:52:54.192842 1081 ts_tablet_manager.cc:1428] T 506e1def1e7f4fad82e88d07af8c5f51 P ed679beae3fd46d0b830a37ec01a3e29: Time spent starting tablet: real 0.030s user 0.025s sys 0.001s
I20250815 13:52:54.198278 1083 raft_consensus.cc:357] T 506e1def1e7f4fad82e88d07af8c5f51 P 2476a8890caa4741b9d6a49abda486e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 36813 } } peers { permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 } } peers { permanent_uuid: "ed679beae3fd46d0b830a37ec01a3e29" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 37049 } }
I20250815 13:52:54.199385 1083 raft_consensus.cc:383] T 506e1def1e7f4fad82e88d07af8c5f51 P 2476a8890caa4741b9d6a49abda486e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 13:52:54.199697 1083 raft_consensus.cc:738] T 506e1def1e7f4fad82e88d07af8c5f51 P 2476a8890caa4741b9d6a49abda486e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2476a8890caa4741b9d6a49abda486e2, State: Initialized, Role: FOLLOWER
I20250815 13:52:54.200685 1083 consensus_queue.cc:260] T 506e1def1e7f4fad82e88d07af8c5f51 P 2476a8890caa4741b9d6a49abda486e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 36813 } } peers { permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 } } peers { permanent_uuid: "ed679beae3fd46d0b830a37ec01a3e29" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 37049 } }
I20250815 13:52:54.201242 1082 raft_consensus.cc:357] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 36813 } } peers { permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 } } peers { permanent_uuid: "ed679beae3fd46d0b830a37ec01a3e29" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 37049 } }
I20250815 13:52:54.202219 1082 raft_consensus.cc:383] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 13:52:54.202561 1082 raft_consensus.cc:738] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe93b76dfb244e45bfc87202827f6acd, State: Initialized, Role: FOLLOWER
I20250815 13:52:54.203958 1082 consensus_queue.cc:260] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 36813 } } peers { permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 } } peers { permanent_uuid: "ed679beae3fd46d0b830a37ec01a3e29" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 37049 } }
I20250815 13:52:54.206256 1062 heartbeater.cc:499] Master 127.0.106.190:44703 was elected leader, sending a full tablet report...
I20250815 13:52:54.209146 1082 ts_tablet_manager.cc:1428] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd: Time spent starting tablet: real 0.038s user 0.032s sys 0.006s
I20250815 13:52:54.210160 1083 ts_tablet_manager.cc:1428] T 506e1def1e7f4fad82e88d07af8c5f51 P 2476a8890caa4741b9d6a49abda486e2: Time spent starting tablet: real 0.042s user 0.030s sys 0.008s
I20250815 13:52:54.332185 1087 raft_consensus.cc:491] T 506e1def1e7f4fad82e88d07af8c5f51 P ed679beae3fd46d0b830a37ec01a3e29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 13:52:54.332772 1087 raft_consensus.cc:513] T 506e1def1e7f4fad82e88d07af8c5f51 P ed679beae3fd46d0b830a37ec01a3e29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 36813 } } peers { permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 } } peers { permanent_uuid: "ed679beae3fd46d0b830a37ec01a3e29" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 37049 } }
I20250815 13:52:54.335595 1087 leader_election.cc:290] T 506e1def1e7f4fad82e88d07af8c5f51 P ed679beae3fd46d0b830a37ec01a3e29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe93b76dfb244e45bfc87202827f6acd (127.0.106.130:36813), 2476a8890caa4741b9d6a49abda486e2 (127.0.106.132:38593)
I20250815 13:52:54.351830 1017 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "506e1def1e7f4fad82e88d07af8c5f51" candidate_uuid: "ed679beae3fd46d0b830a37ec01a3e29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2476a8890caa4741b9d6a49abda486e2" is_pre_election: true
I20250815 13:52:54.351887 749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "506e1def1e7f4fad82e88d07af8c5f51" candidate_uuid: "ed679beae3fd46d0b830a37ec01a3e29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe93b76dfb244e45bfc87202827f6acd" is_pre_election: true
I20250815 13:52:54.352581 1017 raft_consensus.cc:2466] T 506e1def1e7f4fad82e88d07af8c5f51 P 2476a8890caa4741b9d6a49abda486e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed679beae3fd46d0b830a37ec01a3e29 in term 0.
I20250815 13:52:54.352717 749 raft_consensus.cc:2466] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed679beae3fd46d0b830a37ec01a3e29 in term 0.
I20250815 13:52:54.353961 816 leader_election.cc:304] T 506e1def1e7f4fad82e88d07af8c5f51 P ed679beae3fd46d0b830a37ec01a3e29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2476a8890caa4741b9d6a49abda486e2, ed679beae3fd46d0b830a37ec01a3e29; no voters:
I20250815 13:52:54.355420 1087 raft_consensus.cc:2802] T 506e1def1e7f4fad82e88d07af8c5f51 P ed679beae3fd46d0b830a37ec01a3e29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250815 13:52:54.355842 1087 raft_consensus.cc:491] T 506e1def1e7f4fad82e88d07af8c5f51 P ed679beae3fd46d0b830a37ec01a3e29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 13:52:54.356230 1087 raft_consensus.cc:3058] T 506e1def1e7f4fad82e88d07af8c5f51 P ed679beae3fd46d0b830a37ec01a3e29 [term 0 FOLLOWER]: Advancing to term 1
I20250815 13:52:54.363328 1087 raft_consensus.cc:513] T 506e1def1e7f4fad82e88d07af8c5f51 P ed679beae3fd46d0b830a37ec01a3e29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 36813 } } peers { permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 } } peers { permanent_uuid: "ed679beae3fd46d0b830a37ec01a3e29" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 37049 } }
I20250815 13:52:54.365020 1087 leader_election.cc:290] T 506e1def1e7f4fad82e88d07af8c5f51 P ed679beae3fd46d0b830a37ec01a3e29 [CANDIDATE]: Term 1 election: Requested vote from peers fe93b76dfb244e45bfc87202827f6acd (127.0.106.130:36813), 2476a8890caa4741b9d6a49abda486e2 (127.0.106.132:38593)
I20250815 13:52:54.366096 1017 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "506e1def1e7f4fad82e88d07af8c5f51" candidate_uuid: "ed679beae3fd46d0b830a37ec01a3e29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2476a8890caa4741b9d6a49abda486e2"
I20250815 13:52:54.366036 749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "506e1def1e7f4fad82e88d07af8c5f51" candidate_uuid: "ed679beae3fd46d0b830a37ec01a3e29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe93b76dfb244e45bfc87202827f6acd"
I20250815 13:52:54.366564 1017 raft_consensus.cc:3058] T 506e1def1e7f4fad82e88d07af8c5f51 P 2476a8890caa4741b9d6a49abda486e2 [term 0 FOLLOWER]: Advancing to term 1
I20250815 13:52:54.366598 749 raft_consensus.cc:3058] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [term 0 FOLLOWER]: Advancing to term 1
I20250815 13:52:54.371753 1017 raft_consensus.cc:2466] T 506e1def1e7f4fad82e88d07af8c5f51 P 2476a8890caa4741b9d6a49abda486e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed679beae3fd46d0b830a37ec01a3e29 in term 1.
I20250815 13:52:54.371882 749 raft_consensus.cc:2466] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed679beae3fd46d0b830a37ec01a3e29 in term 1.
I20250815 13:52:54.372869 816 leader_election.cc:304] T 506e1def1e7f4fad82e88d07af8c5f51 P ed679beae3fd46d0b830a37ec01a3e29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2476a8890caa4741b9d6a49abda486e2, ed679beae3fd46d0b830a37ec01a3e29; no voters:
I20250815 13:52:54.373565 1087 raft_consensus.cc:2802] T 506e1def1e7f4fad82e88d07af8c5f51 P ed679beae3fd46d0b830a37ec01a3e29 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 13:52:54.375255 1087 raft_consensus.cc:695] T 506e1def1e7f4fad82e88d07af8c5f51 P ed679beae3fd46d0b830a37ec01a3e29 [term 1 LEADER]: Becoming Leader. State: Replica: ed679beae3fd46d0b830a37ec01a3e29, State: Running, Role: LEADER
I20250815 13:52:54.376194 1087 consensus_queue.cc:237] T 506e1def1e7f4fad82e88d07af8c5f51 P ed679beae3fd46d0b830a37ec01a3e29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 36813 } } peers { permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 } } peers { permanent_uuid: "ed679beae3fd46d0b830a37ec01a3e29" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 37049 } }
I20250815 13:52:54.387355 472 catalog_manager.cc:5582] T 506e1def1e7f4fad82e88d07af8c5f51 P ed679beae3fd46d0b830a37ec01a3e29 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed679beae3fd46d0b830a37ec01a3e29 (127.0.106.131). New cstate: current_term: 1 leader_uuid: "ed679beae3fd46d0b830a37ec01a3e29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 36813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed679beae3fd46d0b830a37ec01a3e29" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 37049 } health_report { overall_health: HEALTHY } } }
W20250815 13:52:54.416607 928 tablet.cc:2378] T 506e1def1e7f4fad82e88d07af8c5f51 P ed679beae3fd46d0b830a37ec01a3e29: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 13:52:54.418242 426 external_mini_cluster.cc:949] 4 TS(s) registered with all masters
I20250815 13:52:54.423393 426 ts_itest-base.cc:246] Waiting for 1 tablets on tserver fe93b76dfb244e45bfc87202827f6acd to finish bootstrapping
W20250815 13:52:54.434253 1063 tablet.cc:2378] T 506e1def1e7f4fad82e88d07af8c5f51 P 2476a8890caa4741b9d6a49abda486e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250815 13:52:54.436318 795 tablet.cc:2378] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 13:52:54.442127 426 ts_itest-base.cc:246] Waiting for 1 tablets on tserver ed679beae3fd46d0b830a37ec01a3e29 to finish bootstrapping
I20250815 13:52:54.456708 426 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 2476a8890caa4741b9d6a49abda486e2 to finish bootstrapping
I20250815 13:52:54.474745 426 kudu-admin-test.cc:709] Waiting for Master to see the current replicas...
I20250815 13:52:54.478739 426 kudu-admin-test.cc:716] Tablet locations:
tablet_locations {
tablet_id: "506e1def1e7f4fad82e88d07af8c5f51"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 2
role: LEADER
}
}
ts_infos {
permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd"
rpc_addresses {
host: "127.0.106.130"
port: 36813
}
}
ts_infos {
permanent_uuid: "2476a8890caa4741b9d6a49abda486e2"
rpc_addresses {
host: "127.0.106.132"
port: 38593
}
}
ts_infos {
permanent_uuid: "ed679beae3fd46d0b830a37ec01a3e29"
rpc_addresses {
host: "127.0.106.131"
port: 37049
}
}
I20250815 13:52:54.817112 1087 consensus_queue.cc:1035] T 506e1def1e7f4fad82e88d07af8c5f51 P ed679beae3fd46d0b830a37ec01a3e29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 36813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250815 13:52:54.836407 1092 consensus_queue.cc:1035] T 506e1def1e7f4fad82e88d07af8c5f51 P ed679beae3fd46d0b830a37ec01a3e29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250815 13:52:54.837584 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 798
W20250815 13:52:54.866837 952 connection.cc:537] server connection from 127.0.106.131:57183 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250815 13:52:54.867638 460 connection.cc:537] server connection from 127.0.106.131:53127 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250815 13:52:54.868702 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 440
I20250815 13:52:54.895295 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.190:44703
--webserver_interface=127.0.106.190
--webserver_port=41297
--builtin_ntp_servers=127.0.106.148:44757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.106.190:44703 with env {}
W20250815 13:52:55.233099 1103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:52:55.233728 1103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:52:55.234176 1103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:52:55.267879 1103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 13:52:55.268230 1103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:52:55.268462 1103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 13:52:55.268662 1103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 13:52:55.310051 1103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:44757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.106.190:44703
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.190:44703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.106.190
--webserver_port=41297
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:52:55.311765 1103 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:52:55.313710 1103 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:52:55.326926 1109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:52:55.502923 661 heartbeater.cc:646] Failed to heartbeat to 127.0.106.190:44703 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.106.190:44703: connect: Connection refused (error 111)
W20250815 13:52:55.861069 794 heartbeater.cc:646] Failed to heartbeat to 127.0.106.190:44703 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.106.190:44703: connect: Connection refused (error 111)
W20250815 13:52:55.889616 1062 heartbeater.cc:646] Failed to heartbeat to 127.0.106.190:44703 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.106.190:44703: connect: Connection refused (error 111)
I20250815 13:52:56.352015 1120 raft_consensus.cc:491] T 506e1def1e7f4fad82e88d07af8c5f51 P 2476a8890caa4741b9d6a49abda486e2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ed679beae3fd46d0b830a37ec01a3e29)
I20250815 13:52:56.352355 1119 raft_consensus.cc:491] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ed679beae3fd46d0b830a37ec01a3e29)
I20250815 13:52:56.352711 1120 raft_consensus.cc:513] T 506e1def1e7f4fad82e88d07af8c5f51 P 2476a8890caa4741b9d6a49abda486e2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 36813 } } peers { permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 } } peers { permanent_uuid: "ed679beae3fd46d0b830a37ec01a3e29" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 37049 } }
I20250815 13:52:56.353107 1119 raft_consensus.cc:513] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 36813 } } peers { permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 } } peers { permanent_uuid: "ed679beae3fd46d0b830a37ec01a3e29" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 37049 } }
I20250815 13:52:56.355779 1120 leader_election.cc:290] T 506e1def1e7f4fad82e88d07af8c5f51 P 2476a8890caa4741b9d6a49abda486e2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fe93b76dfb244e45bfc87202827f6acd (127.0.106.130:36813), ed679beae3fd46d0b830a37ec01a3e29 (127.0.106.131:37049)
W20250815 13:52:56.366082 950 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.106.131:37049: connect: Connection refused (error 111)
I20250815 13:52:56.378001 1017 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "506e1def1e7f4fad82e88d07af8c5f51" candidate_uuid: "fe93b76dfb244e45bfc87202827f6acd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2476a8890caa4741b9d6a49abda486e2" is_pre_election: true
I20250815 13:52:56.378830 1017 raft_consensus.cc:2466] T 506e1def1e7f4fad82e88d07af8c5f51 P 2476a8890caa4741b9d6a49abda486e2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe93b76dfb244e45bfc87202827f6acd in term 1.
I20250815 13:52:56.380770 683 leader_election.cc:304] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2476a8890caa4741b9d6a49abda486e2, fe93b76dfb244e45bfc87202827f6acd; no voters:
I20250815 13:52:56.394989 1119 leader_election.cc:290] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2476a8890caa4741b9d6a49abda486e2 (127.0.106.132:38593), ed679beae3fd46d0b830a37ec01a3e29 (127.0.106.131:37049)
I20250815 13:52:56.395943 1119 raft_consensus.cc:2802] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250815 13:52:56.396385 1119 raft_consensus.cc:491] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [term 1 FOLLOWER]: Starting leader election (detected failure of leader ed679beae3fd46d0b830a37ec01a3e29)
I20250815 13:52:56.396823 1119 raft_consensus.cc:3058] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [term 1 FOLLOWER]: Advancing to term 2
W20250815 13:52:56.414739 950 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.131:37049, user_credentials={real_user=slave}} blocked reactor thread for 48967.6us
W20250815 13:52:56.438798 950 leader_election.cc:336] T 506e1def1e7f4fad82e88d07af8c5f51 P 2476a8890caa4741b9d6a49abda486e2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ed679beae3fd46d0b830a37ec01a3e29 (127.0.106.131:37049): Network error: Client connection negotiation failed: client connection to 127.0.106.131:37049: connect: Connection refused (error 111)
W20250815 13:52:56.444386 683 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.106.132:38593, user_credentials={real_user=slave}} blocked reactor thread for 63989.1us
W20250815 13:52:56.447925 683 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.106.131:37049: connect: Connection refused (error 111)
I20250815 13:52:56.452342 749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "506e1def1e7f4fad82e88d07af8c5f51" candidate_uuid: "2476a8890caa4741b9d6a49abda486e2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe93b76dfb244e45bfc87202827f6acd" is_pre_election: true
W20250815 13:52:56.463079 683 leader_election.cc:336] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ed679beae3fd46d0b830a37ec01a3e29 (127.0.106.131:37049): Network error: Client connection negotiation failed: client connection to 127.0.106.131:37049: connect: Connection refused (error 111)
I20250815 13:52:56.464869 1119 raft_consensus.cc:513] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 36813 } } peers { permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 } } peers { permanent_uuid: "ed679beae3fd46d0b830a37ec01a3e29" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 37049 } }
I20250815 13:52:56.466244 749 raft_consensus.cc:2391] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2476a8890caa4741b9d6a49abda486e2 in current term 2: Already voted for candidate fe93b76dfb244e45bfc87202827f6acd in this term.
I20250815 13:52:56.470278 951 leader_election.cc:304] T 506e1def1e7f4fad82e88d07af8c5f51 P 2476a8890caa4741b9d6a49abda486e2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2476a8890caa4741b9d6a49abda486e2; no voters: ed679beae3fd46d0b830a37ec01a3e29, fe93b76dfb244e45bfc87202827f6acd
I20250815 13:52:56.471287 1120 raft_consensus.cc:3058] T 506e1def1e7f4fad82e88d07af8c5f51 P 2476a8890caa4741b9d6a49abda486e2 [term 1 FOLLOWER]: Advancing to term 2
I20250815 13:52:56.473119 1119 leader_election.cc:290] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [CANDIDATE]: Term 2 election: Requested vote from peers 2476a8890caa4741b9d6a49abda486e2 (127.0.106.132:38593), ed679beae3fd46d0b830a37ec01a3e29 (127.0.106.131:37049)
I20250815 13:52:56.474244 1017 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "506e1def1e7f4fad82e88d07af8c5f51" candidate_uuid: "fe93b76dfb244e45bfc87202827f6acd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2476a8890caa4741b9d6a49abda486e2"
I20250815 13:52:56.478449 1120 raft_consensus.cc:2747] T 506e1def1e7f4fad82e88d07af8c5f51 P 2476a8890caa4741b9d6a49abda486e2 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250815 13:52:56.480185 683 leader_election.cc:336] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ed679beae3fd46d0b830a37ec01a3e29 (127.0.106.131:37049): Network error: Client connection negotiation failed: client connection to 127.0.106.131:37049: connect: Connection refused (error 111)
I20250815 13:52:56.485252 1017 raft_consensus.cc:2466] T 506e1def1e7f4fad82e88d07af8c5f51 P 2476a8890caa4741b9d6a49abda486e2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe93b76dfb244e45bfc87202827f6acd in term 2.
I20250815 13:52:56.488792 683 leader_election.cc:304] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2476a8890caa4741b9d6a49abda486e2, fe93b76dfb244e45bfc87202827f6acd; no voters: ed679beae3fd46d0b830a37ec01a3e29
I20250815 13:52:56.489943 1119 raft_consensus.cc:2802] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [term 2 FOLLOWER]: Leader election won for term 2
I20250815 13:52:56.490424 1119 raft_consensus.cc:695] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [term 2 LEADER]: Becoming Leader. State: Replica: fe93b76dfb244e45bfc87202827f6acd, State: Running, Role: LEADER
I20250815 13:52:56.491442 1119 consensus_queue.cc:237] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 36813 } } peers { permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 } } peers { permanent_uuid: "ed679beae3fd46d0b830a37ec01a3e29" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 37049 } }
W20250815 13:52:56.730382 1108 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 1103
W20250815 13:52:55.328110 1110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:52:56.748535 1103 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.421s user 0.584s sys 0.712s
W20250815 13:52:56.749557 1103 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.422s user 0.585s sys 0.712s
W20250815 13:52:56.750859 1112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:52:56.759801 1111 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1431 milliseconds
I20250815 13:52:56.760486 1103 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 13:52:56.762158 1103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:52:56.765391 1103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:52:56.766824 1103 hybrid_clock.cc:648] HybridClock initialized: now 1755265976766769 us; error 56 us; skew 500 ppm
I20250815 13:52:56.767793 1103 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:52:56.774890 1103 webserver.cc:480] Webserver started at http://127.0.106.190:41297/ using document root <none> and password file <none>
I20250815 13:52:56.776024 1103 fs_manager.cc:362] Metadata directory not provided
I20250815 13:52:56.776278 1103 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:52:56.785737 1103 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.001s sys 0.004s
I20250815 13:52:56.791533 1133 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:52:56.793078 1103 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20250815 13:52:56.793483 1103 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
uuid: "17c68a6894774ffcb876c4ba45e7e7c9"
format_stamp: "Formatted at 2025-08-15 13:52:45 on dist-test-slave-kt02"
I20250815 13:52:56.795847 1103 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:52:56.854058 1103 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:52:56.855708 1103 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:52:56.856153 1103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:52:56.872371 1017 raft_consensus.cc:1273] T 506e1def1e7f4fad82e88d07af8c5f51 P 2476a8890caa4741b9d6a49abda486e2 [term 2 FOLLOWER]: Refusing update from remote peer fe93b76dfb244e45bfc87202827f6acd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250815 13:52:56.873664 1119 consensus_queue.cc:1035] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250815 13:52:56.897868 683 consensus_peers.cc:489] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd -> Peer ed679beae3fd46d0b830a37ec01a3e29 (127.0.106.131:37049): Couldn't send request to peer ed679beae3fd46d0b830a37ec01a3e29. Status: Network error: Client connection negotiation failed: client connection to 127.0.106.131:37049: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250815 13:52:56.957933 1103 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.190:44703
I20250815 13:52:56.957975 1189 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.190:44703 every 8 connection(s)
I20250815 13:52:56.960959 1103 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250815 13:52:56.965612 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 1103
I20250815 13:52:56.966115 426 kudu-admin-test.cc:735] Forcing unsafe config change on tserver fe93b76dfb244e45bfc87202827f6acd
I20250815 13:52:56.972337 1190 sys_catalog.cc:263] Verifying existing consensus state
I20250815 13:52:56.979795 1190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9: Bootstrap starting.
I20250815 13:52:57.038118 1190 log.cc:826] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9: Log is configured to *not* fsync() on all Append() calls
I20250815 13:52:57.071144 1190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 13:52:57.072337 1190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9: Bootstrap complete.
I20250815 13:52:57.097059 1190 raft_consensus.cc:357] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c68a6894774ffcb876c4ba45e7e7c9" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 44703 } }
I20250815 13:52:57.099257 1190 raft_consensus.cc:738] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17c68a6894774ffcb876c4ba45e7e7c9, State: Initialized, Role: FOLLOWER
I20250815 13:52:57.100137 1190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c68a6894774ffcb876c4ba45e7e7c9" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 44703 } }
I20250815 13:52:57.100652 1190 raft_consensus.cc:397] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 13:52:57.100919 1190 raft_consensus.cc:491] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 13:52:57.101271 1190 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9 [term 1 FOLLOWER]: Advancing to term 2
I20250815 13:52:57.108779 1190 raft_consensus.cc:513] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c68a6894774ffcb876c4ba45e7e7c9" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 44703 } }
I20250815 13:52:57.109540 1190 leader_election.cc:304] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17c68a6894774ffcb876c4ba45e7e7c9; no voters:
I20250815 13:52:57.111634 1190 leader_election.cc:290] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250815 13:52:57.112380 1194 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9 [term 2 FOLLOWER]: Leader election won for term 2
I20250815 13:52:57.115801 1190 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9 [sys.catalog]: configured and running, proceeding with master startup.
I20250815 13:52:57.120530 1194 raft_consensus.cc:695] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9 [term 2 LEADER]: Becoming Leader. State: Replica: 17c68a6894774ffcb876c4ba45e7e7c9, State: Running, Role: LEADER
I20250815 13:52:57.121850 1194 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c68a6894774ffcb876c4ba45e7e7c9" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 44703 } }
I20250815 13:52:57.132774 1196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17c68a6894774ffcb876c4ba45e7e7c9. Latest consensus state: current_term: 2 leader_uuid: "17c68a6894774ffcb876c4ba45e7e7c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c68a6894774ffcb876c4ba45e7e7c9" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 44703 } } }
I20250815 13:52:57.134792 1196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9 [sys.catalog]: This master's current role is: LEADER
I20250815 13:52:57.134148 1195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "17c68a6894774ffcb876c4ba45e7e7c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c68a6894774ffcb876c4ba45e7e7c9" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 44703 } } }
I20250815 13:52:57.136874 1195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9 [sys.catalog]: This master's current role is: LEADER
I20250815 13:52:57.142354 1205 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 13:52:57.158063 1205 catalog_manager.cc:671] Loaded metadata for table TestTable [id=580f63afd698468c95da5702005badd3]
I20250815 13:52:57.165946 1205 tablet_loader.cc:96] loaded metadata for tablet 506e1def1e7f4fad82e88d07af8c5f51 (table TestTable [id=580f63afd698468c95da5702005badd3])
I20250815 13:52:57.167554 1205 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 13:52:57.172686 1205 catalog_manager.cc:1261] Loaded cluster ID: dbce9e72c486460095a75d353e87d5a9
I20250815 13:52:57.173000 1205 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 13:52:57.181051 1205 catalog_manager.cc:1506] Loading token signing keys...
I20250815 13:52:57.186357 1205 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 17c68a6894774ffcb876c4ba45e7e7c9: Loaded TSK: 0
I20250815 13:52:57.188223 1205 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250815 13:52:57.353338 1192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:52:57.353935 1192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:52:57.385375 1192 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250815 13:52:57.575027 661 heartbeater.cc:344] Connected to a master server at 127.0.106.190:44703
I20250815 13:52:57.581908 1155 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "87db2b33cd35440fae9f366e99a1b6e2" instance_seqno: 1755265967372597) as {username='slave'} at 127.0.106.129:34197; Asking this server to re-register.
I20250815 13:52:57.583966 661 heartbeater.cc:461] Registering TS with master...
I20250815 13:52:57.584762 661 heartbeater.cc:507] Master 127.0.106.190:44703 requested a full tablet report, sending...
I20250815 13:52:57.588198 1154 ts_manager.cc:194] Registered new tserver with Master: 87db2b33cd35440fae9f366e99a1b6e2 (127.0.106.129:38637)
W20250815 13:52:57.625702 790 debug-util.cc:398] Leaking SignalData structure 0x7b08000bbe80 after lost signal to thread 666
W20250815 13:52:57.627007 790 debug-util.cc:398] Leaking SignalData structure 0x7b0800083980 after lost signal to thread 793
I20250815 13:52:57.907155 794 heartbeater.cc:344] Connected to a master server at 127.0.106.190:44703
I20250815 13:52:57.912505 1155 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd" instance_seqno: 1755265969381137) as {username='slave'} at 127.0.106.130:48663; Asking this server to re-register.
I20250815 13:52:57.914577 794 heartbeater.cc:461] Registering TS with master...
I20250815 13:52:57.915498 794 heartbeater.cc:507] Master 127.0.106.190:44703 requested a full tablet report, sending...
I20250815 13:52:57.918970 1155 ts_manager.cc:194] Registered new tserver with Master: fe93b76dfb244e45bfc87202827f6acd (127.0.106.130:36813)
I20250815 13:52:57.923818 1155 catalog_manager.cc:5582] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd reported cstate change: term changed from 1 to 2, leader changed from ed679beae3fd46d0b830a37ec01a3e29 (127.0.106.131) to fe93b76dfb244e45bfc87202827f6acd (127.0.106.130). New cstate: current_term: 2 leader_uuid: "fe93b76dfb244e45bfc87202827f6acd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 36813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed679beae3fd46d0b830a37ec01a3e29" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 37049 } health_report { overall_health: UNKNOWN } } }
I20250815 13:52:57.961562 1062 heartbeater.cc:344] Connected to a master server at 127.0.106.190:44703
I20250815 13:52:57.965049 1154 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" instance_seqno: 1755265973883384) as {username='slave'} at 127.0.106.132:42145; Asking this server to re-register.
I20250815 13:52:57.966758 1062 heartbeater.cc:461] Registering TS with master...
I20250815 13:52:57.967473 1062 heartbeater.cc:507] Master 127.0.106.190:44703 requested a full tablet report, sending...
I20250815 13:52:57.970698 1154 ts_manager.cc:194] Registered new tserver with Master: 2476a8890caa4741b9d6a49abda486e2 (127.0.106.132:38593)
W20250815 13:52:58.829581 1216 debug-util.cc:398] Leaking SignalData structure 0x7b08000347a0 after lost signal to thread 1192
W20250815 13:52:58.830258 1216 kernel_stack_watchdog.cc:198] Thread 1192 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 399ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250815 13:52:58.954459 1192 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.526s user 0.545s sys 0.907s
W20250815 13:52:59.113330 1192 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.685s user 0.546s sys 0.916s
I20250815 13:52:59.172042 749 tablet_service.cc:1905] Received UnsafeChangeConfig RPC: dest_uuid: "fe93b76dfb244e45bfc87202827f6acd"
tablet_id: "506e1def1e7f4fad82e88d07af8c5f51"
caller_id: "kudu-tools"
new_config {
peers {
permanent_uuid: "2476a8890caa4741b9d6a49abda486e2"
}
peers {
permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd"
}
}
from {username='slave'} at 127.0.0.1:46954
W20250815 13:52:59.173475 749 raft_consensus.cc:2216] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [term 2 LEADER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 36813 } } peers { permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 } } peers { permanent_uuid: "ed679beae3fd46d0b830a37ec01a3e29" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 37049 } }NEW CONFIG: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 36813 } } peers { permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 } } unsafe_config_change: true
I20250815 13:52:59.174558 749 raft_consensus.cc:3053] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [term 2 LEADER]: Stepping down as leader of term 2
I20250815 13:52:59.174769 749 raft_consensus.cc:738] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fe93b76dfb244e45bfc87202827f6acd, State: Running, Role: LEADER
I20250815 13:52:59.175459 749 consensus_queue.cc:260] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 36813 } } peers { permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 } } peers { permanent_uuid: "ed679beae3fd46d0b830a37ec01a3e29" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 37049 } }
I20250815 13:52:59.176527 749 raft_consensus.cc:3058] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [term 2 FOLLOWER]: Advancing to term 3
I20250815 13:53:00.340004 1242 raft_consensus.cc:491] T 506e1def1e7f4fad82e88d07af8c5f51 P 2476a8890caa4741b9d6a49abda486e2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fe93b76dfb244e45bfc87202827f6acd)
I20250815 13:53:00.340472 1242 raft_consensus.cc:513] T 506e1def1e7f4fad82e88d07af8c5f51 P 2476a8890caa4741b9d6a49abda486e2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 36813 } } peers { permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 } } peers { permanent_uuid: "ed679beae3fd46d0b830a37ec01a3e29" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 37049 } }
I20250815 13:53:00.341960 1242 leader_election.cc:290] T 506e1def1e7f4fad82e88d07af8c5f51 P 2476a8890caa4741b9d6a49abda486e2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fe93b76dfb244e45bfc87202827f6acd (127.0.106.130:36813), ed679beae3fd46d0b830a37ec01a3e29 (127.0.106.131:37049)
I20250815 13:53:00.343143 749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "506e1def1e7f4fad82e88d07af8c5f51" candidate_uuid: "2476a8890caa4741b9d6a49abda486e2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe93b76dfb244e45bfc87202827f6acd" is_pre_election: true
W20250815 13:53:00.346446 950 leader_election.cc:336] T 506e1def1e7f4fad82e88d07af8c5f51 P 2476a8890caa4741b9d6a49abda486e2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ed679beae3fd46d0b830a37ec01a3e29 (127.0.106.131:37049): Network error: Client connection negotiation failed: client connection to 127.0.106.131:37049: connect: Connection refused (error 111)
I20250815 13:53:00.346771 950 leader_election.cc:304] T 506e1def1e7f4fad82e88d07af8c5f51 P 2476a8890caa4741b9d6a49abda486e2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2476a8890caa4741b9d6a49abda486e2; no voters: ed679beae3fd46d0b830a37ec01a3e29, fe93b76dfb244e45bfc87202827f6acd
I20250815 13:53:00.347440 1242 raft_consensus.cc:2747] T 506e1def1e7f4fad82e88d07af8c5f51 P 2476a8890caa4741b9d6a49abda486e2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250815 13:53:00.684134 1245 raft_consensus.cc:491] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools)
I20250815 13:53:00.684640 1245 raft_consensus.cc:513] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 36813 } } peers { permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 } } unsafe_config_change: true
I20250815 13:53:00.686058 1245 leader_election.cc:290] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2476a8890caa4741b9d6a49abda486e2 (127.0.106.132:38593)
I20250815 13:53:00.691702 1017 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "506e1def1e7f4fad82e88d07af8c5f51" candidate_uuid: "fe93b76dfb244e45bfc87202827f6acd" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2476a8890caa4741b9d6a49abda486e2" is_pre_election: true
I20250815 13:53:00.692193 1017 raft_consensus.cc:2466] T 506e1def1e7f4fad82e88d07af8c5f51 P 2476a8890caa4741b9d6a49abda486e2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe93b76dfb244e45bfc87202827f6acd in term 2.
I20250815 13:53:00.693159 683 leader_election.cc:304] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2476a8890caa4741b9d6a49abda486e2, fe93b76dfb244e45bfc87202827f6acd; no voters:
I20250815 13:53:00.693744 1245 raft_consensus.cc:2802] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250815 13:53:00.694011 1245 raft_consensus.cc:491] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [term 3 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools)
I20250815 13:53:00.694264 1245 raft_consensus.cc:3058] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [term 3 FOLLOWER]: Advancing to term 4
I20250815 13:53:00.698376 1245 raft_consensus.cc:513] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 36813 } } peers { permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 } } unsafe_config_change: true
I20250815 13:53:00.699374 1245 leader_election.cc:290] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [CANDIDATE]: Term 4 election: Requested vote from peers 2476a8890caa4741b9d6a49abda486e2 (127.0.106.132:38593)
I20250815 13:53:00.700572 1017 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "506e1def1e7f4fad82e88d07af8c5f51" candidate_uuid: "fe93b76dfb244e45bfc87202827f6acd" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2476a8890caa4741b9d6a49abda486e2"
I20250815 13:53:00.700955 1017 raft_consensus.cc:3058] T 506e1def1e7f4fad82e88d07af8c5f51 P 2476a8890caa4741b9d6a49abda486e2 [term 2 FOLLOWER]: Advancing to term 4
I20250815 13:53:00.705386 1017 raft_consensus.cc:2466] T 506e1def1e7f4fad82e88d07af8c5f51 P 2476a8890caa4741b9d6a49abda486e2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe93b76dfb244e45bfc87202827f6acd in term 4.
I20250815 13:53:00.706308 683 leader_election.cc:304] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2476a8890caa4741b9d6a49abda486e2, fe93b76dfb244e45bfc87202827f6acd; no voters:
I20250815 13:53:00.706844 1245 raft_consensus.cc:2802] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [term 4 FOLLOWER]: Leader election won for term 4
I20250815 13:53:00.707770 1245 raft_consensus.cc:695] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [term 4 LEADER]: Becoming Leader. State: Replica: fe93b76dfb244e45bfc87202827f6acd, State: Running, Role: LEADER
I20250815 13:53:00.708405 1245 consensus_queue.cc:237] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 36813 } } peers { permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 } } unsafe_config_change: true
I20250815 13:53:00.713816 1154 catalog_manager.cc:5582] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd reported cstate change: term changed from 2 to 4, now has a pending config: VOTER fe93b76dfb244e45bfc87202827f6acd (127.0.106.130), VOTER 2476a8890caa4741b9d6a49abda486e2 (127.0.106.132). New cstate: current_term: 4 leader_uuid: "fe93b76dfb244e45bfc87202827f6acd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 36813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed679beae3fd46d0b830a37ec01a3e29" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 37049 } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 36813 } } peers { permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 } } unsafe_config_change: true }
I20250815 13:53:01.332252 1017 raft_consensus.cc:1273] T 506e1def1e7f4fad82e88d07af8c5f51 P 2476a8890caa4741b9d6a49abda486e2 [term 4 FOLLOWER]: Refusing update from remote peer fe93b76dfb244e45bfc87202827f6acd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250815 13:53:01.333473 1252 consensus_queue.cc:1035] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250815 13:53:01.342761 1255 raft_consensus.cc:2953] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [term 4 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER ed679beae3fd46d0b830a37ec01a3e29 (127.0.106.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 36813 } } peers { permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 } } unsafe_config_change: true }
I20250815 13:53:01.344205 1017 raft_consensus.cc:2953] T 506e1def1e7f4fad82e88d07af8c5f51 P 2476a8890caa4741b9d6a49abda486e2 [term 4 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER ed679beae3fd46d0b830a37ec01a3e29 (127.0.106.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 36813 } } peers { permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 } } unsafe_config_change: true }
I20250815 13:53:01.353164 1154 catalog_manager.cc:5582] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd reported cstate change: config changed from index -1 to 3, VOTER ed679beae3fd46d0b830a37ec01a3e29 (127.0.106.131) evicted, no longer has a pending config: VOTER fe93b76dfb244e45bfc87202827f6acd (127.0.106.130), VOTER 2476a8890caa4741b9d6a49abda486e2 (127.0.106.132). New cstate: current_term: 4 leader_uuid: "fe93b76dfb244e45bfc87202827f6acd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 36813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
W20250815 13:53:01.362886 1154 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet 506e1def1e7f4fad82e88d07af8c5f51 on TS ed679beae3fd46d0b830a37ec01a3e29: Not found: failed to reset TS proxy: Could not find TS for UUID ed679beae3fd46d0b830a37ec01a3e29
I20250815 13:53:01.379531 749 consensus_queue.cc:237] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 36813 } } peers { permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 } } peers { permanent_uuid: "87db2b33cd35440fae9f366e99a1b6e2" member_type: NON_VOTER last_known_addr { host: "127.0.106.129" port: 38637 } attrs { promote: true } } unsafe_config_change: true
I20250815 13:53:01.384277 1017 raft_consensus.cc:1273] T 506e1def1e7f4fad82e88d07af8c5f51 P 2476a8890caa4741b9d6a49abda486e2 [term 4 FOLLOWER]: Refusing update from remote peer fe93b76dfb244e45bfc87202827f6acd: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250815 13:53:01.385892 1252 consensus_queue.cc:1035] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250815 13:53:01.391851 1255 raft_consensus.cc:2953] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER 87db2b33cd35440fae9f366e99a1b6e2 (127.0.106.129) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 36813 } } peers { permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 } } peers { permanent_uuid: "87db2b33cd35440fae9f366e99a1b6e2" member_type: NON_VOTER last_known_addr { host: "127.0.106.129" port: 38637 } attrs { promote: true } } unsafe_config_change: true }
I20250815 13:53:01.393375 1017 raft_consensus.cc:2953] T 506e1def1e7f4fad82e88d07af8c5f51 P 2476a8890caa4741b9d6a49abda486e2 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER 87db2b33cd35440fae9f366e99a1b6e2 (127.0.106.129) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 36813 } } peers { permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 } } peers { permanent_uuid: "87db2b33cd35440fae9f366e99a1b6e2" member_type: NON_VOTER last_known_addr { host: "127.0.106.129" port: 38637 } attrs { promote: true } } unsafe_config_change: true }
W20250815 13:53:01.396507 684 consensus_peers.cc:489] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd -> Peer 87db2b33cd35440fae9f366e99a1b6e2 (127.0.106.129:38637): Couldn't send request to peer 87db2b33cd35440fae9f366e99a1b6e2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 506e1def1e7f4fad82e88d07af8c5f51. This is attempt 1: this message will repeat every 5th retry.
I20250815 13:53:01.399554 1136 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 506e1def1e7f4fad82e88d07af8c5f51 with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250815 13:53:01.402854 1155 catalog_manager.cc:5582] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd reported cstate change: config changed from index 3 to 5, NON_VOTER 87db2b33cd35440fae9f366e99a1b6e2 (127.0.106.129) added. New cstate: current_term: 4 leader_uuid: "fe93b76dfb244e45bfc87202827f6acd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 36813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87db2b33cd35440fae9f366e99a1b6e2" member_type: NON_VOTER last_known_addr { host: "127.0.106.129" port: 38637 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true }
W20250815 13:53:01.415364 1135 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 506e1def1e7f4fad82e88d07af8c5f51 on TS ed679beae3fd46d0b830a37ec01a3e29 failed: Not found: failed to reset TS proxy: Could not find TS for UUID ed679beae3fd46d0b830a37ec01a3e29
W20250815 13:53:01.637884 1058 debug-util.cc:398] Leaking SignalData structure 0x7b080003c260 after lost signal to thread 932
W20250815 13:53:01.638819 1058 debug-util.cc:398] Leaking SignalData structure 0x7b08000c8d40 after lost signal to thread 1061
I20250815 13:53:01.968283 1265 ts_tablet_manager.cc:927] T 506e1def1e7f4fad82e88d07af8c5f51 P 87db2b33cd35440fae9f366e99a1b6e2: Initiating tablet copy from peer fe93b76dfb244e45bfc87202827f6acd (127.0.106.130:36813)
I20250815 13:53:01.971170 1265 tablet_copy_client.cc:323] T 506e1def1e7f4fad82e88d07af8c5f51 P 87db2b33cd35440fae9f366e99a1b6e2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.106.130:36813
I20250815 13:53:02.004186 769 tablet_copy_service.cc:140] P fe93b76dfb244e45bfc87202827f6acd: Received BeginTabletCopySession request for tablet 506e1def1e7f4fad82e88d07af8c5f51 from peer 87db2b33cd35440fae9f366e99a1b6e2 ({username='slave'} at 127.0.106.129:35399)
I20250815 13:53:02.004704 769 tablet_copy_service.cc:161] P fe93b76dfb244e45bfc87202827f6acd: Beginning new tablet copy session on tablet 506e1def1e7f4fad82e88d07af8c5f51 from peer 87db2b33cd35440fae9f366e99a1b6e2 at {username='slave'} at 127.0.106.129:35399: session id = 87db2b33cd35440fae9f366e99a1b6e2-506e1def1e7f4fad82e88d07af8c5f51
I20250815 13:53:02.010470 769 tablet_copy_source_session.cc:215] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd: Tablet Copy: opened 0 blocks and 1 log segments
I20250815 13:53:02.016645 1265 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 506e1def1e7f4fad82e88d07af8c5f51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:53:02.036492 1265 tablet_copy_client.cc:806] T 506e1def1e7f4fad82e88d07af8c5f51 P 87db2b33cd35440fae9f366e99a1b6e2: tablet copy: Starting download of 0 data blocks...
I20250815 13:53:02.037268 1265 tablet_copy_client.cc:670] T 506e1def1e7f4fad82e88d07af8c5f51 P 87db2b33cd35440fae9f366e99a1b6e2: tablet copy: Starting download of 1 WAL segments...
I20250815 13:53:02.042600 1265 tablet_copy_client.cc:538] T 506e1def1e7f4fad82e88d07af8c5f51 P 87db2b33cd35440fae9f366e99a1b6e2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250815 13:53:02.049503 1265 tablet_bootstrap.cc:492] T 506e1def1e7f4fad82e88d07af8c5f51 P 87db2b33cd35440fae9f366e99a1b6e2: Bootstrap starting.
I20250815 13:53:02.064523 1265 log.cc:826] T 506e1def1e7f4fad82e88d07af8c5f51 P 87db2b33cd35440fae9f366e99a1b6e2: Log is configured to *not* fsync() on all Append() calls
I20250815 13:53:02.077471 1265 tablet_bootstrap.cc:492] T 506e1def1e7f4fad82e88d07af8c5f51 P 87db2b33cd35440fae9f366e99a1b6e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 13:53:02.078344 1265 tablet_bootstrap.cc:492] T 506e1def1e7f4fad82e88d07af8c5f51 P 87db2b33cd35440fae9f366e99a1b6e2: Bootstrap complete.
I20250815 13:53:02.079113 1265 ts_tablet_manager.cc:1397] T 506e1def1e7f4fad82e88d07af8c5f51 P 87db2b33cd35440fae9f366e99a1b6e2: Time spent bootstrapping tablet: real 0.030s user 0.020s sys 0.012s
I20250815 13:53:02.098090 1265 raft_consensus.cc:357] T 506e1def1e7f4fad82e88d07af8c5f51 P 87db2b33cd35440fae9f366e99a1b6e2 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 36813 } } peers { permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 } } peers { permanent_uuid: "87db2b33cd35440fae9f366e99a1b6e2" member_type: NON_VOTER last_known_addr { host: "127.0.106.129" port: 38637 } attrs { promote: true } } unsafe_config_change: true
I20250815 13:53:02.099121 1265 raft_consensus.cc:738] T 506e1def1e7f4fad82e88d07af8c5f51 P 87db2b33cd35440fae9f366e99a1b6e2 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 87db2b33cd35440fae9f366e99a1b6e2, State: Initialized, Role: LEARNER
I20250815 13:53:02.099839 1265 consensus_queue.cc:260] T 506e1def1e7f4fad82e88d07af8c5f51 P 87db2b33cd35440fae9f366e99a1b6e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 36813 } } peers { permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 } } peers { permanent_uuid: "87db2b33cd35440fae9f366e99a1b6e2" member_type: NON_VOTER last_known_addr { host: "127.0.106.129" port: 38637 } attrs { promote: true } } unsafe_config_change: true
I20250815 13:53:02.104255 1265 ts_tablet_manager.cc:1428] T 506e1def1e7f4fad82e88d07af8c5f51 P 87db2b33cd35440fae9f366e99a1b6e2: Time spent starting tablet: real 0.025s user 0.023s sys 0.001s
I20250815 13:53:02.106273 769 tablet_copy_service.cc:342] P fe93b76dfb244e45bfc87202827f6acd: Request end of tablet copy session 87db2b33cd35440fae9f366e99a1b6e2-506e1def1e7f4fad82e88d07af8c5f51 received from {username='slave'} at 127.0.106.129:35399
I20250815 13:53:02.106737 769 tablet_copy_service.cc:434] P fe93b76dfb244e45bfc87202827f6acd: ending tablet copy session 87db2b33cd35440fae9f366e99a1b6e2-506e1def1e7f4fad82e88d07af8c5f51 on tablet 506e1def1e7f4fad82e88d07af8c5f51 with peer 87db2b33cd35440fae9f366e99a1b6e2
I20250815 13:53:02.546820 616 raft_consensus.cc:1215] T 506e1def1e7f4fad82e88d07af8c5f51 P 87db2b33cd35440fae9f366e99a1b6e2 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.4->[4.5-4.5] Dedup: 4.5->[]
W20250815 13:53:02.582067 1135 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 506e1def1e7f4fad82e88d07af8c5f51 on TS ed679beae3fd46d0b830a37ec01a3e29 failed: Not found: failed to reset TS proxy: Could not find TS for UUID ed679beae3fd46d0b830a37ec01a3e29
I20250815 13:53:03.019718 1274 raft_consensus.cc:1062] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd: attempting to promote NON_VOTER 87db2b33cd35440fae9f366e99a1b6e2 to VOTER
I20250815 13:53:03.021481 1274 consensus_queue.cc:237] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 36813 } } peers { permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 } } peers { permanent_uuid: "87db2b33cd35440fae9f366e99a1b6e2" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 38637 } attrs { promote: false } } unsafe_config_change: true
I20250815 13:53:03.027731 616 raft_consensus.cc:1273] T 506e1def1e7f4fad82e88d07af8c5f51 P 87db2b33cd35440fae9f366e99a1b6e2 [term 4 LEARNER]: Refusing update from remote peer fe93b76dfb244e45bfc87202827f6acd: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250815 13:53:03.027550 1017 raft_consensus.cc:1273] T 506e1def1e7f4fad82e88d07af8c5f51 P 2476a8890caa4741b9d6a49abda486e2 [term 4 FOLLOWER]: Refusing update from remote peer fe93b76dfb244e45bfc87202827f6acd: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250815 13:53:03.029206 1274 consensus_queue.cc:1035] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [LEADER]: Connected to new peer: Peer: permanent_uuid: "87db2b33cd35440fae9f366e99a1b6e2" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 38637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250815 13:53:03.030189 1273 consensus_queue.cc:1035] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250815 13:53:03.040509 1273 raft_consensus.cc:2953] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 87db2b33cd35440fae9f366e99a1b6e2 (127.0.106.129) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 36813 } } peers { permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 } } peers { permanent_uuid: "87db2b33cd35440fae9f366e99a1b6e2" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 38637 } attrs { promote: false } } unsafe_config_change: true }
I20250815 13:53:03.042408 616 raft_consensus.cc:2953] T 506e1def1e7f4fad82e88d07af8c5f51 P 87db2b33cd35440fae9f366e99a1b6e2 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 87db2b33cd35440fae9f366e99a1b6e2 (127.0.106.129) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 36813 } } peers { permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 } } peers { permanent_uuid: "87db2b33cd35440fae9f366e99a1b6e2" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 38637 } attrs { promote: false } } unsafe_config_change: true }
I20250815 13:53:03.044605 1017 raft_consensus.cc:2953] T 506e1def1e7f4fad82e88d07af8c5f51 P 2476a8890caa4741b9d6a49abda486e2 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 87db2b33cd35440fae9f366e99a1b6e2 (127.0.106.129) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 36813 } } peers { permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 } } peers { permanent_uuid: "87db2b33cd35440fae9f366e99a1b6e2" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 38637 } attrs { promote: false } } unsafe_config_change: true }
I20250815 13:53:03.054936 1155 catalog_manager.cc:5582] T 506e1def1e7f4fad82e88d07af8c5f51 P fe93b76dfb244e45bfc87202827f6acd reported cstate change: config changed from index 5 to 6, 87db2b33cd35440fae9f366e99a1b6e2 (127.0.106.129) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "fe93b76dfb244e45bfc87202827f6acd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 36813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2476a8890caa4741b9d6a49abda486e2" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 38593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87db2b33cd35440fae9f366e99a1b6e2" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 38637 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
I20250815 13:53:03.173416 426 kudu-admin-test.cc:751] Waiting for Master to see new config...
I20250815 13:53:03.190610 426 kudu-admin-test.cc:756] Tablet locations:
tablet_locations {
tablet_id: "506e1def1e7f4fad82e88d07af8c5f51"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: LEADER
}
interned_replicas {
ts_info_idx: 1
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "fe93b76dfb244e45bfc87202827f6acd"
rpc_addresses {
host: "127.0.106.130"
port: 36813
}
}
ts_infos {
permanent_uuid: "2476a8890caa4741b9d6a49abda486e2"
rpc_addresses {
host: "127.0.106.132"
port: 38593
}
}
ts_infos {
permanent_uuid: "87db2b33cd35440fae9f366e99a1b6e2"
rpc_addresses {
host: "127.0.106.129"
port: 38637
}
}
I20250815 13:53:03.194662 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 532
I20250815 13:53:03.235734 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 665
I20250815 13:53:03.260823 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 931
I20250815 13:53:03.286041 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 1103
2025-08-15T13:53:03Z chronyd exiting
[ OK ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes (21121 ms)
[ RUN ] AdminCliTest.TestGracefulSpecificLeaderStepDown
I20250815 13:53:03.346544 426 test_util.cc:276] Using random seed: -1251182762
I20250815 13:53:03.353027 426 ts_itest-base.cc:115] Starting cluster with:
I20250815 13:53:03.353204 426 ts_itest-base.cc:116] --------------
I20250815 13:53:03.353332 426 ts_itest-base.cc:117] 3 tablet servers
I20250815 13:53:03.353444 426 ts_itest-base.cc:118] 3 replicas per TS
I20250815 13:53:03.353552 426 ts_itest-base.cc:119] --------------
2025-08-15T13:53:03Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-15T13:53:03Z Disabled control of system clock
I20250815 13:53:03.390522 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.190:44303
--webserver_interface=127.0.106.190
--webserver_port=0
--builtin_ntp_servers=127.0.106.148:41597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.106.190:44303
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250815 13:53:03.746201 1293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:53:03.746934 1293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:53:03.747459 1293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:53:03.784057 1293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 13:53:03.784435 1293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:53:03.784678 1293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 13:53:03.784888 1293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 13:53:03.825979 1293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:41597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.0.106.190:44303
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.190:44303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.106.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:53:03.827790 1293 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:53:03.829746 1293 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:53:03.842782 1299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:53:03.844724 1300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:53:04.955983 1302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:53:04.958784 1301 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1109 milliseconds
I20250815 13:53:04.958899 1293 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 13:53:04.960300 1293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:53:04.962947 1293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:53:04.964388 1293 hybrid_clock.cc:648] HybridClock initialized: now 1755265984964366 us; error 55 us; skew 500 ppm
I20250815 13:53:04.965268 1293 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:53:04.972119 1293 webserver.cc:480] Webserver started at http://127.0.106.190:44239/ using document root <none> and password file <none>
I20250815 13:53:04.973152 1293 fs_manager.cc:362] Metadata directory not provided
I20250815 13:53:04.973382 1293 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:53:04.973850 1293 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 13:53:04.978732 1293 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d70ac68081804ca993da49ffc94ce968"
format_stamp: "Formatted at 2025-08-15 13:53:04 on dist-test-slave-kt02"
I20250815 13:53:04.979871 1293 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d70ac68081804ca993da49ffc94ce968"
format_stamp: "Formatted at 2025-08-15 13:53:04 on dist-test-slave-kt02"
I20250815 13:53:04.987461 1293 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250815 13:53:04.993347 1309 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:53:04.994670 1293 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250815 13:53:04.995111 1293 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d70ac68081804ca993da49ffc94ce968"
format_stamp: "Formatted at 2025-08-15 13:53:04 on dist-test-slave-kt02"
I20250815 13:53:04.995464 1293 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:53:05.054255 1293 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:53:05.055775 1293 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:53:05.056210 1293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:53:05.129317 1293 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.190:44303
I20250815 13:53:05.129388 1360 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.190:44303 every 8 connection(s)
I20250815 13:53:05.132375 1293 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250815 13:53:05.138094 1361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:53:05.142640 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 1293
I20250815 13:53:05.143141 426 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250815 13:53:05.163842 1361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d70ac68081804ca993da49ffc94ce968: Bootstrap starting.
I20250815 13:53:05.169766 1361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d70ac68081804ca993da49ffc94ce968: Neither blocks nor log segments found. Creating new log.
I20250815 13:53:05.171922 1361 log.cc:826] T 00000000000000000000000000000000 P d70ac68081804ca993da49ffc94ce968: Log is configured to *not* fsync() on all Append() calls
I20250815 13:53:05.176748 1361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d70ac68081804ca993da49ffc94ce968: No bootstrap required, opened a new log
I20250815 13:53:05.193425 1361 raft_consensus.cc:357] T 00000000000000000000000000000000 P d70ac68081804ca993da49ffc94ce968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70ac68081804ca993da49ffc94ce968" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 44303 } }
I20250815 13:53:05.194147 1361 raft_consensus.cc:383] T 00000000000000000000000000000000 P d70ac68081804ca993da49ffc94ce968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 13:53:05.194401 1361 raft_consensus.cc:738] T 00000000000000000000000000000000 P d70ac68081804ca993da49ffc94ce968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d70ac68081804ca993da49ffc94ce968, State: Initialized, Role: FOLLOWER
I20250815 13:53:05.195147 1361 consensus_queue.cc:260] T 00000000000000000000000000000000 P d70ac68081804ca993da49ffc94ce968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70ac68081804ca993da49ffc94ce968" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 44303 } }
I20250815 13:53:05.195679 1361 raft_consensus.cc:397] T 00000000000000000000000000000000 P d70ac68081804ca993da49ffc94ce968 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 13:53:05.195933 1361 raft_consensus.cc:491] T 00000000000000000000000000000000 P d70ac68081804ca993da49ffc94ce968 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 13:53:05.196254 1361 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d70ac68081804ca993da49ffc94ce968 [term 0 FOLLOWER]: Advancing to term 1
I20250815 13:53:05.200316 1361 raft_consensus.cc:513] T 00000000000000000000000000000000 P d70ac68081804ca993da49ffc94ce968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70ac68081804ca993da49ffc94ce968" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 44303 } }
I20250815 13:53:05.201045 1361 leader_election.cc:304] T 00000000000000000000000000000000 P d70ac68081804ca993da49ffc94ce968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d70ac68081804ca993da49ffc94ce968; no voters:
I20250815 13:53:05.202713 1361 leader_election.cc:290] T 00000000000000000000000000000000 P d70ac68081804ca993da49ffc94ce968 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 13:53:05.203658 1366 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d70ac68081804ca993da49ffc94ce968 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 13:53:05.206162 1366 raft_consensus.cc:695] T 00000000000000000000000000000000 P d70ac68081804ca993da49ffc94ce968 [term 1 LEADER]: Becoming Leader. State: Replica: d70ac68081804ca993da49ffc94ce968, State: Running, Role: LEADER
I20250815 13:53:05.207072 1361 sys_catalog.cc:564] T 00000000000000000000000000000000 P d70ac68081804ca993da49ffc94ce968 [sys.catalog]: configured and running, proceeding with master startup.
I20250815 13:53:05.207115 1366 consensus_queue.cc:237] T 00000000000000000000000000000000 P d70ac68081804ca993da49ffc94ce968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70ac68081804ca993da49ffc94ce968" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 44303 } }
I20250815 13:53:05.227880 1367 sys_catalog.cc:455] T 00000000000000000000000000000000 P d70ac68081804ca993da49ffc94ce968 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d70ac68081804ca993da49ffc94ce968" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70ac68081804ca993da49ffc94ce968" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 44303 } } }
I20250815 13:53:05.228691 1367 sys_catalog.cc:458] T 00000000000000000000000000000000 P d70ac68081804ca993da49ffc94ce968 [sys.catalog]: This master's current role is: LEADER
I20250815 13:53:05.231487 1368 sys_catalog.cc:455] T 00000000000000000000000000000000 P d70ac68081804ca993da49ffc94ce968 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d70ac68081804ca993da49ffc94ce968. Latest consensus state: current_term: 1 leader_uuid: "d70ac68081804ca993da49ffc94ce968" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70ac68081804ca993da49ffc94ce968" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 44303 } } }
I20250815 13:53:05.232271 1368 sys_catalog.cc:458] T 00000000000000000000000000000000 P d70ac68081804ca993da49ffc94ce968 [sys.catalog]: This master's current role is: LEADER
I20250815 13:53:05.237561 1374 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 13:53:05.249061 1374 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 13:53:05.266438 1374 catalog_manager.cc:1349] Generated new cluster ID: be53cbc6364d433ca71d159a619a93be
I20250815 13:53:05.266831 1374 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 13:53:05.307682 1374 catalog_manager.cc:1372] Generated new certificate authority record
I20250815 13:53:05.310086 1374 catalog_manager.cc:1506] Loading token signing keys...
I20250815 13:53:05.328920 1374 catalog_manager.cc:5955] T 00000000000000000000000000000000 P d70ac68081804ca993da49ffc94ce968: Generated new TSK 0
I20250815 13:53:05.330237 1374 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250815 13:53:05.342962 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.129:0
--local_ip_for_outbound_sockets=127.0.106.129
--webserver_interface=127.0.106.129
--webserver_port=0
--tserver_master_addrs=127.0.106.190:44303
--builtin_ntp_servers=127.0.106.148:41597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250815 13:53:05.653950 1385 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250815 13:53:05.654572 1385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:53:05.654857 1385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:53:05.655432 1385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:53:05.687363 1385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:53:05.688189 1385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.129
I20250815 13:53:05.723964 1385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:41597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.106.129
--webserver_port=0
--tserver_master_addrs=127.0.106.190:44303
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.129
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:53:05.725278 1385 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:53:05.727119 1385 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:53:05.740443 1391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:53:07.143193 1390 debug-util.cc:398] Leaking SignalData structure 0x7b08000068a0 after lost signal to thread 1385
W20250815 13:53:05.747464 1392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:53:07.283295 1385 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.543s user 0.616s sys 0.847s
W20250815 13:53:07.283816 1385 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.544s user 0.616s sys 0.847s
W20250815 13:53:07.287892 1393 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1536 milliseconds
W20250815 13:53:07.289403 1394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 13:53:07.289496 1385 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 13:53:07.290830 1385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:53:07.293411 1385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:53:07.294839 1385 hybrid_clock.cc:648] HybridClock initialized: now 1755265987294778 us; error 73 us; skew 500 ppm
I20250815 13:53:07.295795 1385 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:53:07.303526 1385 webserver.cc:480] Webserver started at http://127.0.106.129:38001/ using document root <none> and password file <none>
I20250815 13:53:07.304634 1385 fs_manager.cc:362] Metadata directory not provided
I20250815 13:53:07.304864 1385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:53:07.305354 1385 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 13:53:07.310374 1385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "31d9f7ba4dcd46feb4cdc8acf04e3fce"
format_stamp: "Formatted at 2025-08-15 13:53:07 on dist-test-slave-kt02"
I20250815 13:53:07.311949 1385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "31d9f7ba4dcd46feb4cdc8acf04e3fce"
format_stamp: "Formatted at 2025-08-15 13:53:07 on dist-test-slave-kt02"
I20250815 13:53:07.321080 1385 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s
I20250815 13:53:07.327852 1401 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:53:07.329295 1385 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20250815 13:53:07.329685 1385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "31d9f7ba4dcd46feb4cdc8acf04e3fce"
format_stamp: "Formatted at 2025-08-15 13:53:07 on dist-test-slave-kt02"
I20250815 13:53:07.330072 1385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:53:07.392874 1385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:53:07.394567 1385 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:53:07.395074 1385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:53:07.398473 1385 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 13:53:07.403280 1385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 13:53:07.403519 1385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 13:53:07.403770 1385 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 13:53:07.403932 1385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 13:53:07.588932 1385 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.129:36063
I20250815 13:53:07.589129 1513 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.129:36063 every 8 connection(s)
I20250815 13:53:07.591917 1385 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250815 13:53:07.601333 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 1385
I20250815 13:53:07.601893 426 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250815 13:53:07.610625 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.130:0
--local_ip_for_outbound_sockets=127.0.106.130
--webserver_interface=127.0.106.130
--webserver_port=0
--tserver_master_addrs=127.0.106.190:44303
--builtin_ntp_servers=127.0.106.148:41597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250815 13:53:07.624531 1514 heartbeater.cc:344] Connected to a master server at 127.0.106.190:44303
I20250815 13:53:07.625200 1514 heartbeater.cc:461] Registering TS with master...
I20250815 13:53:07.626863 1514 heartbeater.cc:507] Master 127.0.106.190:44303 requested a full tablet report, sending...
I20250815 13:53:07.630709 1326 ts_manager.cc:194] Registered new tserver with Master: 31d9f7ba4dcd46feb4cdc8acf04e3fce (127.0.106.129:36063)
I20250815 13:53:07.633611 1326 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.129:34079
W20250815 13:53:07.958379 1518 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250815 13:53:07.959205 1518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:53:07.959487 1518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:53:07.959990 1518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:53:07.994201 1518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:53:07.995204 1518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.130
I20250815 13:53:08.033324 1518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:41597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.106.130
--webserver_port=0
--tserver_master_addrs=127.0.106.190:44303
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.130
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:53:08.034956 1518 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:53:08.036841 1518 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:53:08.050124 1524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 13:53:08.638974 1514 heartbeater.cc:499] Master 127.0.106.190:44303 was elected leader, sending a full tablet report...
W20250815 13:53:08.051554 1525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:53:09.191167 1527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:53:09.195614 1526 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1138 milliseconds
I20250815 13:53:09.195751 1518 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 13:53:09.197121 1518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:53:09.199777 1518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:53:09.201268 1518 hybrid_clock.cc:648] HybridClock initialized: now 1755265989201216 us; error 69 us; skew 500 ppm
I20250815 13:53:09.202118 1518 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:53:09.209676 1518 webserver.cc:480] Webserver started at http://127.0.106.130:34735/ using document root <none> and password file <none>
I20250815 13:53:09.210783 1518 fs_manager.cc:362] Metadata directory not provided
I20250815 13:53:09.211009 1518 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:53:09.211594 1518 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 13:53:09.216609 1518 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "478cc56543fe4e9685812617d9375373"
format_stamp: "Formatted at 2025-08-15 13:53:09 on dist-test-slave-kt02"
I20250815 13:53:09.217916 1518 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "478cc56543fe4e9685812617d9375373"
format_stamp: "Formatted at 2025-08-15 13:53:09 on dist-test-slave-kt02"
I20250815 13:53:09.226997 1518 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.010s sys 0.001s
I20250815 13:53:09.233713 1534 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:53:09.235430 1518 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.001s
I20250815 13:53:09.235829 1518 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "478cc56543fe4e9685812617d9375373"
format_stamp: "Formatted at 2025-08-15 13:53:09 on dist-test-slave-kt02"
I20250815 13:53:09.236214 1518 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:53:09.282361 1518 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:53:09.283849 1518 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:53:09.284299 1518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:53:09.286857 1518 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 13:53:09.291422 1518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 13:53:09.291656 1518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 13:53:09.291898 1518 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 13:53:09.292111 1518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 13:53:09.435434 1518 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.130:37063
I20250815 13:53:09.435576 1646 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.130:37063 every 8 connection(s)
I20250815 13:53:09.438124 1518 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250815 13:53:09.446516 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 1518
I20250815 13:53:09.447017 426 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250815 13:53:09.453634 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.131:0
--local_ip_for_outbound_sockets=127.0.106.131
--webserver_interface=127.0.106.131
--webserver_port=0
--tserver_master_addrs=127.0.106.190:44303
--builtin_ntp_servers=127.0.106.148:41597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250815 13:53:09.460784 1647 heartbeater.cc:344] Connected to a master server at 127.0.106.190:44303
I20250815 13:53:09.461369 1647 heartbeater.cc:461] Registering TS with master...
I20250815 13:53:09.462981 1647 heartbeater.cc:507] Master 127.0.106.190:44303 requested a full tablet report, sending...
I20250815 13:53:09.465961 1326 ts_manager.cc:194] Registered new tserver with Master: 478cc56543fe4e9685812617d9375373 (127.0.106.130:37063)
I20250815 13:53:09.467521 1326 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.130:59899
W20250815 13:53:09.780788 1651 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250815 13:53:09.781432 1651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:53:09.781690 1651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:53:09.782182 1651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:53:09.814249 1651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:53:09.815126 1651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.131
I20250815 13:53:09.851368 1651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:41597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.106.131
--webserver_port=0
--tserver_master_addrs=127.0.106.190:44303
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.131
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:53:09.852741 1651 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:53:09.854310 1651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:53:09.867904 1657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 13:53:10.472189 1647 heartbeater.cc:499] Master 127.0.106.190:44303 was elected leader, sending a full tablet report...
W20250815 13:53:09.870733 1658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:53:09.872351 1660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 13:53:09.873075 1651 server_base.cc:1047] running on GCE node
I20250815 13:53:10.961270 1651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:53:10.963569 1651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:53:10.964964 1651 hybrid_clock.cc:648] HybridClock initialized: now 1755265990964904 us; error 70 us; skew 500 ppm
I20250815 13:53:10.965766 1651 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:53:10.972107 1651 webserver.cc:480] Webserver started at http://127.0.106.131:40335/ using document root <none> and password file <none>
I20250815 13:53:10.973016 1651 fs_manager.cc:362] Metadata directory not provided
I20250815 13:53:10.973222 1651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:53:10.973695 1651 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 13:53:10.977972 1651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "728d6aea5e7e4b35b87e2439f6a27d51"
format_stamp: "Formatted at 2025-08-15 13:53:10 on dist-test-slave-kt02"
I20250815 13:53:10.979029 1651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "728d6aea5e7e4b35b87e2439f6a27d51"
format_stamp: "Formatted at 2025-08-15 13:53:10 on dist-test-slave-kt02"
I20250815 13:53:10.986394 1651 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20250815 13:53:10.992714 1667 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:53:10.993906 1651 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250815 13:53:10.994305 1651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "728d6aea5e7e4b35b87e2439f6a27d51"
format_stamp: "Formatted at 2025-08-15 13:53:10 on dist-test-slave-kt02"
I20250815 13:53:10.994647 1651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:53:11.040719 1651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:53:11.042104 1651 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:53:11.042532 1651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:53:11.045008 1651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 13:53:11.049134 1651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 13:53:11.049337 1651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 13:53:11.049589 1651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 13:53:11.049741 1651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 13:53:11.192530 1651 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.131:40519
I20250815 13:53:11.192693 1779 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.131:40519 every 8 connection(s)
I20250815 13:53:11.194977 1651 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250815 13:53:11.197389 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 1651
I20250815 13:53:11.197875 426 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250815 13:53:11.217821 1780 heartbeater.cc:344] Connected to a master server at 127.0.106.190:44303
I20250815 13:53:11.218457 1780 heartbeater.cc:461] Registering TS with master...
I20250815 13:53:11.219713 1780 heartbeater.cc:507] Master 127.0.106.190:44303 requested a full tablet report, sending...
I20250815 13:53:11.221938 1326 ts_manager.cc:194] Registered new tserver with Master: 728d6aea5e7e4b35b87e2439f6a27d51 (127.0.106.131:40519)
I20250815 13:53:11.223353 1326 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.131:37425
I20250815 13:53:11.234242 426 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 13:53:11.270720 1326 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57910:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250815 13:53:11.289245 1326 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250815 13:53:11.344085 1449 tablet_service.cc:1468] Processing CreateTablet for tablet 7f50e5209a374463bfb85e53ffe33927 (DEFAULT_TABLE table=TestTable [id=fc4c4d4292cc45a0bbc264a5de95179c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 13:53:11.345567 1449 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f50e5209a374463bfb85e53ffe33927. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:53:11.344906 1715 tablet_service.cc:1468] Processing CreateTablet for tablet 7f50e5209a374463bfb85e53ffe33927 (DEFAULT_TABLE table=TestTable [id=fc4c4d4292cc45a0bbc264a5de95179c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 13:53:11.344906 1582 tablet_service.cc:1468] Processing CreateTablet for tablet 7f50e5209a374463bfb85e53ffe33927 (DEFAULT_TABLE table=TestTable [id=fc4c4d4292cc45a0bbc264a5de95179c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 13:53:11.347005 1582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f50e5209a374463bfb85e53ffe33927. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:53:11.347021 1715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f50e5209a374463bfb85e53ffe33927. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:53:11.367481 1799 tablet_bootstrap.cc:492] T 7f50e5209a374463bfb85e53ffe33927 P 31d9f7ba4dcd46feb4cdc8acf04e3fce: Bootstrap starting.
I20250815 13:53:11.373283 1799 tablet_bootstrap.cc:654] T 7f50e5209a374463bfb85e53ffe33927 P 31d9f7ba4dcd46feb4cdc8acf04e3fce: Neither blocks nor log segments found. Creating new log.
I20250815 13:53:11.375638 1799 log.cc:826] T 7f50e5209a374463bfb85e53ffe33927 P 31d9f7ba4dcd46feb4cdc8acf04e3fce: Log is configured to *not* fsync() on all Append() calls
I20250815 13:53:11.377737 1800 tablet_bootstrap.cc:492] T 7f50e5209a374463bfb85e53ffe33927 P 478cc56543fe4e9685812617d9375373: Bootstrap starting.
I20250815 13:53:11.381224 1801 tablet_bootstrap.cc:492] T 7f50e5209a374463bfb85e53ffe33927 P 728d6aea5e7e4b35b87e2439f6a27d51: Bootstrap starting.
I20250815 13:53:11.386190 1800 tablet_bootstrap.cc:654] T 7f50e5209a374463bfb85e53ffe33927 P 478cc56543fe4e9685812617d9375373: Neither blocks nor log segments found. Creating new log.
I20250815 13:53:11.387645 1799 tablet_bootstrap.cc:492] T 7f50e5209a374463bfb85e53ffe33927 P 31d9f7ba4dcd46feb4cdc8acf04e3fce: No bootstrap required, opened a new log
I20250815 13:53:11.388262 1799 ts_tablet_manager.cc:1397] T 7f50e5209a374463bfb85e53ffe33927 P 31d9f7ba4dcd46feb4cdc8acf04e3fce: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.011s
I20250815 13:53:11.388439 1800 log.cc:826] T 7f50e5209a374463bfb85e53ffe33927 P 478cc56543fe4e9685812617d9375373: Log is configured to *not* fsync() on all Append() calls
I20250815 13:53:11.388818 1801 tablet_bootstrap.cc:654] T 7f50e5209a374463bfb85e53ffe33927 P 728d6aea5e7e4b35b87e2439f6a27d51: Neither blocks nor log segments found. Creating new log.
I20250815 13:53:11.391423 1801 log.cc:826] T 7f50e5209a374463bfb85e53ffe33927 P 728d6aea5e7e4b35b87e2439f6a27d51: Log is configured to *not* fsync() on all Append() calls
I20250815 13:53:11.393405 1800 tablet_bootstrap.cc:492] T 7f50e5209a374463bfb85e53ffe33927 P 478cc56543fe4e9685812617d9375373: No bootstrap required, opened a new log
I20250815 13:53:11.394008 1800 ts_tablet_manager.cc:1397] T 7f50e5209a374463bfb85e53ffe33927 P 478cc56543fe4e9685812617d9375373: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.011s
I20250815 13:53:11.397672 1801 tablet_bootstrap.cc:492] T 7f50e5209a374463bfb85e53ffe33927 P 728d6aea5e7e4b35b87e2439f6a27d51: No bootstrap required, opened a new log
I20250815 13:53:11.398224 1801 ts_tablet_manager.cc:1397] T 7f50e5209a374463bfb85e53ffe33927 P 728d6aea5e7e4b35b87e2439f6a27d51: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.005s
I20250815 13:53:11.411983 1800 raft_consensus.cc:357] T 7f50e5209a374463bfb85e53ffe33927 P 478cc56543fe4e9685812617d9375373 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "728d6aea5e7e4b35b87e2439f6a27d51" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 40519 } } peers { permanent_uuid: "31d9f7ba4dcd46feb4cdc8acf04e3fce" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 36063 } } peers { permanent_uuid: "478cc56543fe4e9685812617d9375373" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 37063 } }
I20250815 13:53:11.412704 1800 raft_consensus.cc:738] T 7f50e5209a374463bfb85e53ffe33927 P 478cc56543fe4e9685812617d9375373 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 478cc56543fe4e9685812617d9375373, State: Initialized, Role: FOLLOWER
I20250815 13:53:11.413290 1800 consensus_queue.cc:260] T 7f50e5209a374463bfb85e53ffe33927 P 478cc56543fe4e9685812617d9375373 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "728d6aea5e7e4b35b87e2439f6a27d51" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 40519 } } peers { permanent_uuid: "31d9f7ba4dcd46feb4cdc8acf04e3fce" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 36063 } } peers { permanent_uuid: "478cc56543fe4e9685812617d9375373" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 37063 } }
I20250815 13:53:11.415740 1799 raft_consensus.cc:357] T 7f50e5209a374463bfb85e53ffe33927 P 31d9f7ba4dcd46feb4cdc8acf04e3fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "728d6aea5e7e4b35b87e2439f6a27d51" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 40519 } } peers { permanent_uuid: "31d9f7ba4dcd46feb4cdc8acf04e3fce" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 36063 } } peers { permanent_uuid: "478cc56543fe4e9685812617d9375373" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 37063 } }
I20250815 13:53:11.416783 1799 raft_consensus.cc:738] T 7f50e5209a374463bfb85e53ffe33927 P 31d9f7ba4dcd46feb4cdc8acf04e3fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31d9f7ba4dcd46feb4cdc8acf04e3fce, State: Initialized, Role: FOLLOWER
I20250815 13:53:11.416957 1800 ts_tablet_manager.cc:1428] T 7f50e5209a374463bfb85e53ffe33927 P 478cc56543fe4e9685812617d9375373: Time spent starting tablet: real 0.023s user 0.020s sys 0.001s
I20250815 13:53:11.417575 1799 consensus_queue.cc:260] T 7f50e5209a374463bfb85e53ffe33927 P 31d9f7ba4dcd46feb4cdc8acf04e3fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "728d6aea5e7e4b35b87e2439f6a27d51" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 40519 } } peers { permanent_uuid: "31d9f7ba4dcd46feb4cdc8acf04e3fce" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 36063 } } peers { permanent_uuid: "478cc56543fe4e9685812617d9375373" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 37063 } }
I20250815 13:53:11.422358 1799 ts_tablet_manager.cc:1428] T 7f50e5209a374463bfb85e53ffe33927 P 31d9f7ba4dcd46feb4cdc8acf04e3fce: Time spent starting tablet: real 0.034s user 0.024s sys 0.009s
I20250815 13:53:11.426841 1801 raft_consensus.cc:357] T 7f50e5209a374463bfb85e53ffe33927 P 728d6aea5e7e4b35b87e2439f6a27d51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "728d6aea5e7e4b35b87e2439f6a27d51" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 40519 } } peers { permanent_uuid: "31d9f7ba4dcd46feb4cdc8acf04e3fce" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 36063 } } peers { permanent_uuid: "478cc56543fe4e9685812617d9375373" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 37063 } }
I20250815 13:53:11.427609 1801 raft_consensus.cc:738] T 7f50e5209a374463bfb85e53ffe33927 P 728d6aea5e7e4b35b87e2439f6a27d51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 728d6aea5e7e4b35b87e2439f6a27d51, State: Initialized, Role: FOLLOWER
I20250815 13:53:11.428328 1801 consensus_queue.cc:260] T 7f50e5209a374463bfb85e53ffe33927 P 728d6aea5e7e4b35b87e2439f6a27d51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "728d6aea5e7e4b35b87e2439f6a27d51" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 40519 } } peers { permanent_uuid: "31d9f7ba4dcd46feb4cdc8acf04e3fce" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 36063 } } peers { permanent_uuid: "478cc56543fe4e9685812617d9375373" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 37063 } }
I20250815 13:53:11.431573 1780 heartbeater.cc:499] Master 127.0.106.190:44303 was elected leader, sending a full tablet report...
I20250815 13:53:11.432653 1801 ts_tablet_manager.cc:1428] T 7f50e5209a374463bfb85e53ffe33927 P 728d6aea5e7e4b35b87e2439f6a27d51: Time spent starting tablet: real 0.034s user 0.031s sys 0.003s
W20250815 13:53:11.444782 1648 tablet.cc:2378] T 7f50e5209a374463bfb85e53ffe33927 P 478cc56543fe4e9685812617d9375373: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250815 13:53:11.450031 1781 tablet.cc:2378] T 7f50e5209a374463bfb85e53ffe33927 P 728d6aea5e7e4b35b87e2439f6a27d51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 13:53:11.451390 426 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 13:53:11.455860 426 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 31d9f7ba4dcd46feb4cdc8acf04e3fce to finish bootstrapping
I20250815 13:53:11.474078 426 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 478cc56543fe4e9685812617d9375373 to finish bootstrapping
I20250815 13:53:11.486424 426 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 728d6aea5e7e4b35b87e2439f6a27d51 to finish bootstrapping
I20250815 13:53:11.535185 1469 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "7f50e5209a374463bfb85e53ffe33927"
dest_uuid: "31d9f7ba4dcd46feb4cdc8acf04e3fce"
from {username='slave'} at 127.0.0.1:33138
I20250815 13:53:11.535830 1469 raft_consensus.cc:491] T 7f50e5209a374463bfb85e53ffe33927 P 31d9f7ba4dcd46feb4cdc8acf04e3fce [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250815 13:53:11.536134 1469 raft_consensus.cc:3058] T 7f50e5209a374463bfb85e53ffe33927 P 31d9f7ba4dcd46feb4cdc8acf04e3fce [term 0 FOLLOWER]: Advancing to term 1
I20250815 13:53:11.540709 1469 raft_consensus.cc:513] T 7f50e5209a374463bfb85e53ffe33927 P 31d9f7ba4dcd46feb4cdc8acf04e3fce [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "728d6aea5e7e4b35b87e2439f6a27d51" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 40519 } } peers { permanent_uuid: "31d9f7ba4dcd46feb4cdc8acf04e3fce" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 36063 } } peers { permanent_uuid: "478cc56543fe4e9685812617d9375373" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 37063 } }
I20250815 13:53:11.543099 1469 leader_election.cc:290] T 7f50e5209a374463bfb85e53ffe33927 P 31d9f7ba4dcd46feb4cdc8acf04e3fce [CANDIDATE]: Term 1 election: Requested vote from peers 728d6aea5e7e4b35b87e2439f6a27d51 (127.0.106.131:40519), 478cc56543fe4e9685812617d9375373 (127.0.106.130:37063)
I20250815 13:53:11.553493 426 cluster_itest_util.cc:257] Not converged past 1 yet: 0.0 0.0 0.0
I20250815 13:53:11.558260 1735 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f50e5209a374463bfb85e53ffe33927" candidate_uuid: "31d9f7ba4dcd46feb4cdc8acf04e3fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "728d6aea5e7e4b35b87e2439f6a27d51"
I20250815 13:53:11.558933 1735 raft_consensus.cc:3058] T 7f50e5209a374463bfb85e53ffe33927 P 728d6aea5e7e4b35b87e2439f6a27d51 [term 0 FOLLOWER]: Advancing to term 1
I20250815 13:53:11.558890 1602 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f50e5209a374463bfb85e53ffe33927" candidate_uuid: "31d9f7ba4dcd46feb4cdc8acf04e3fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "478cc56543fe4e9685812617d9375373"
I20250815 13:53:11.559676 1602 raft_consensus.cc:3058] T 7f50e5209a374463bfb85e53ffe33927 P 478cc56543fe4e9685812617d9375373 [term 0 FOLLOWER]: Advancing to term 1
I20250815 13:53:11.563795 1735 raft_consensus.cc:2466] T 7f50e5209a374463bfb85e53ffe33927 P 728d6aea5e7e4b35b87e2439f6a27d51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31d9f7ba4dcd46feb4cdc8acf04e3fce in term 1.
I20250815 13:53:11.565218 1602 raft_consensus.cc:2466] T 7f50e5209a374463bfb85e53ffe33927 P 478cc56543fe4e9685812617d9375373 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31d9f7ba4dcd46feb4cdc8acf04e3fce in term 1.
I20250815 13:53:11.565225 1403 leader_election.cc:304] T 7f50e5209a374463bfb85e53ffe33927 P 31d9f7ba4dcd46feb4cdc8acf04e3fce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31d9f7ba4dcd46feb4cdc8acf04e3fce, 728d6aea5e7e4b35b87e2439f6a27d51; no voters:
I20250815 13:53:11.566592 1806 raft_consensus.cc:2802] T 7f50e5209a374463bfb85e53ffe33927 P 31d9f7ba4dcd46feb4cdc8acf04e3fce [term 1 FOLLOWER]: Leader election won for term 1
I20250815 13:53:11.568321 1806 raft_consensus.cc:695] T 7f50e5209a374463bfb85e53ffe33927 P 31d9f7ba4dcd46feb4cdc8acf04e3fce [term 1 LEADER]: Becoming Leader. State: Replica: 31d9f7ba4dcd46feb4cdc8acf04e3fce, State: Running, Role: LEADER
I20250815 13:53:11.569278 1806 consensus_queue.cc:237] T 7f50e5209a374463bfb85e53ffe33927 P 31d9f7ba4dcd46feb4cdc8acf04e3fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "728d6aea5e7e4b35b87e2439f6a27d51" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 40519 } } peers { permanent_uuid: "31d9f7ba4dcd46feb4cdc8acf04e3fce" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 36063 } } peers { permanent_uuid: "478cc56543fe4e9685812617d9375373" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 37063 } }
I20250815 13:53:11.580782 1325 catalog_manager.cc:5582] T 7f50e5209a374463bfb85e53ffe33927 P 31d9f7ba4dcd46feb4cdc8acf04e3fce reported cstate change: term changed from 0 to 1, leader changed from <none> to 31d9f7ba4dcd46feb4cdc8acf04e3fce (127.0.106.129). New cstate: current_term: 1 leader_uuid: "31d9f7ba4dcd46feb4cdc8acf04e3fce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "728d6aea5e7e4b35b87e2439f6a27d51" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 40519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31d9f7ba4dcd46feb4cdc8acf04e3fce" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 36063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "478cc56543fe4e9685812617d9375373" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 37063 } health_report { overall_health: UNKNOWN } } }
W20250815 13:53:11.601631 1515 tablet.cc:2378] T 7f50e5209a374463bfb85e53ffe33927 P 31d9f7ba4dcd46feb4cdc8acf04e3fce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 13:53:11.660403 426 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250815 13:53:11.867025 426 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250815 13:53:11.988684 1806 consensus_queue.cc:1035] T 7f50e5209a374463bfb85e53ffe33927 P 31d9f7ba4dcd46feb4cdc8acf04e3fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "728d6aea5e7e4b35b87e2439f6a27d51" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 40519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250815 13:53:12.007777 1806 consensus_queue.cc:1035] T 7f50e5209a374463bfb85e53ffe33927 P 31d9f7ba4dcd46feb4cdc8acf04e3fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "478cc56543fe4e9685812617d9375373" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250815 13:53:13.839222 1469 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "7f50e5209a374463bfb85e53ffe33927"
dest_uuid: "31d9f7ba4dcd46feb4cdc8acf04e3fce"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33144
I20250815 13:53:13.839915 1469 raft_consensus.cc:604] T 7f50e5209a374463bfb85e53ffe33927 P 31d9f7ba4dcd46feb4cdc8acf04e3fce [term 1 LEADER]: Received request to transfer leadership
I20250815 13:53:13.946544 1829 raft_consensus.cc:991] T 7f50e5209a374463bfb85e53ffe33927 P 31d9f7ba4dcd46feb4cdc8acf04e3fce: : Instructing follower 728d6aea5e7e4b35b87e2439f6a27d51 to start an election
I20250815 13:53:13.946931 1829 raft_consensus.cc:1079] T 7f50e5209a374463bfb85e53ffe33927 P 31d9f7ba4dcd46feb4cdc8acf04e3fce [term 1 LEADER]: Signalling peer 728d6aea5e7e4b35b87e2439f6a27d51 to start an election
I20250815 13:53:13.948469 1735 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "7f50e5209a374463bfb85e53ffe33927"
dest_uuid: "728d6aea5e7e4b35b87e2439f6a27d51"
from {username='slave'} at 127.0.106.129:43195
I20250815 13:53:13.948959 1735 raft_consensus.cc:491] T 7f50e5209a374463bfb85e53ffe33927 P 728d6aea5e7e4b35b87e2439f6a27d51 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250815 13:53:13.949203 1735 raft_consensus.cc:3058] T 7f50e5209a374463bfb85e53ffe33927 P 728d6aea5e7e4b35b87e2439f6a27d51 [term 1 FOLLOWER]: Advancing to term 2
I20250815 13:53:13.953383 1735 raft_consensus.cc:513] T 7f50e5209a374463bfb85e53ffe33927 P 728d6aea5e7e4b35b87e2439f6a27d51 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "728d6aea5e7e4b35b87e2439f6a27d51" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 40519 } } peers { permanent_uuid: "31d9f7ba4dcd46feb4cdc8acf04e3fce" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 36063 } } peers { permanent_uuid: "478cc56543fe4e9685812617d9375373" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 37063 } }
I20250815 13:53:13.955502 1735 leader_election.cc:290] T 7f50e5209a374463bfb85e53ffe33927 P 728d6aea5e7e4b35b87e2439f6a27d51 [CANDIDATE]: Term 2 election: Requested vote from peers 31d9f7ba4dcd46feb4cdc8acf04e3fce (127.0.106.129:36063), 478cc56543fe4e9685812617d9375373 (127.0.106.130:37063)
I20250815 13:53:13.966799 1469 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f50e5209a374463bfb85e53ffe33927" candidate_uuid: "728d6aea5e7e4b35b87e2439f6a27d51" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "31d9f7ba4dcd46feb4cdc8acf04e3fce"
I20250815 13:53:13.966912 1602 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f50e5209a374463bfb85e53ffe33927" candidate_uuid: "728d6aea5e7e4b35b87e2439f6a27d51" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "478cc56543fe4e9685812617d9375373"
I20250815 13:53:13.967362 1469 raft_consensus.cc:3053] T 7f50e5209a374463bfb85e53ffe33927 P 31d9f7ba4dcd46feb4cdc8acf04e3fce [term 1 LEADER]: Stepping down as leader of term 1
I20250815 13:53:13.967481 1602 raft_consensus.cc:3058] T 7f50e5209a374463bfb85e53ffe33927 P 478cc56543fe4e9685812617d9375373 [term 1 FOLLOWER]: Advancing to term 2
I20250815 13:53:13.967618 1469 raft_consensus.cc:738] T 7f50e5209a374463bfb85e53ffe33927 P 31d9f7ba4dcd46feb4cdc8acf04e3fce [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 31d9f7ba4dcd46feb4cdc8acf04e3fce, State: Running, Role: LEADER
I20250815 13:53:13.968129 1469 consensus_queue.cc:260] T 7f50e5209a374463bfb85e53ffe33927 P 31d9f7ba4dcd46feb4cdc8acf04e3fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "728d6aea5e7e4b35b87e2439f6a27d51" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 40519 } } peers { permanent_uuid: "31d9f7ba4dcd46feb4cdc8acf04e3fce" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 36063 } } peers { permanent_uuid: "478cc56543fe4e9685812617d9375373" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 37063 } }
I20250815 13:53:13.969027 1469 raft_consensus.cc:3058] T 7f50e5209a374463bfb85e53ffe33927 P 31d9f7ba4dcd46feb4cdc8acf04e3fce [term 1 FOLLOWER]: Advancing to term 2
I20250815 13:53:13.972254 1602 raft_consensus.cc:2466] T 7f50e5209a374463bfb85e53ffe33927 P 478cc56543fe4e9685812617d9375373 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 728d6aea5e7e4b35b87e2439f6a27d51 in term 2.
I20250815 13:53:13.973706 1469 raft_consensus.cc:2466] T 7f50e5209a374463bfb85e53ffe33927 P 31d9f7ba4dcd46feb4cdc8acf04e3fce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 728d6aea5e7e4b35b87e2439f6a27d51 in term 2.
I20250815 13:53:13.973587 1671 leader_election.cc:304] T 7f50e5209a374463bfb85e53ffe33927 P 728d6aea5e7e4b35b87e2439f6a27d51 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 478cc56543fe4e9685812617d9375373, 728d6aea5e7e4b35b87e2439f6a27d51; no voters:
I20250815 13:53:13.975726 1849 raft_consensus.cc:2802] T 7f50e5209a374463bfb85e53ffe33927 P 728d6aea5e7e4b35b87e2439f6a27d51 [term 2 FOLLOWER]: Leader election won for term 2
I20250815 13:53:13.977047 1849 raft_consensus.cc:695] T 7f50e5209a374463bfb85e53ffe33927 P 728d6aea5e7e4b35b87e2439f6a27d51 [term 2 LEADER]: Becoming Leader. State: Replica: 728d6aea5e7e4b35b87e2439f6a27d51, State: Running, Role: LEADER
I20250815 13:53:13.977844 1849 consensus_queue.cc:237] T 7f50e5209a374463bfb85e53ffe33927 P 728d6aea5e7e4b35b87e2439f6a27d51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "728d6aea5e7e4b35b87e2439f6a27d51" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 40519 } } peers { permanent_uuid: "31d9f7ba4dcd46feb4cdc8acf04e3fce" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 36063 } } peers { permanent_uuid: "478cc56543fe4e9685812617d9375373" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 37063 } }
I20250815 13:53:13.986091 1322 catalog_manager.cc:5582] T 7f50e5209a374463bfb85e53ffe33927 P 728d6aea5e7e4b35b87e2439f6a27d51 reported cstate change: term changed from 1 to 2, leader changed from 31d9f7ba4dcd46feb4cdc8acf04e3fce (127.0.106.129) to 728d6aea5e7e4b35b87e2439f6a27d51 (127.0.106.131). New cstate: current_term: 2 leader_uuid: "728d6aea5e7e4b35b87e2439f6a27d51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "728d6aea5e7e4b35b87e2439f6a27d51" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 40519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31d9f7ba4dcd46feb4cdc8acf04e3fce" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 36063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "478cc56543fe4e9685812617d9375373" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 37063 } health_report { overall_health: UNKNOWN } } }
I20250815 13:53:14.416692 1469 raft_consensus.cc:1273] T 7f50e5209a374463bfb85e53ffe33927 P 31d9f7ba4dcd46feb4cdc8acf04e3fce [term 2 FOLLOWER]: Refusing update from remote peer 728d6aea5e7e4b35b87e2439f6a27d51: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250815 13:53:14.418109 1849 consensus_queue.cc:1035] T 7f50e5209a374463bfb85e53ffe33927 P 728d6aea5e7e4b35b87e2439f6a27d51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d9f7ba4dcd46feb4cdc8acf04e3fce" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 36063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250815 13:53:14.425709 1602 raft_consensus.cc:1273] T 7f50e5209a374463bfb85e53ffe33927 P 478cc56543fe4e9685812617d9375373 [term 2 FOLLOWER]: Refusing update from remote peer 728d6aea5e7e4b35b87e2439f6a27d51: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250815 13:53:14.427661 1850 consensus_queue.cc:1035] T 7f50e5209a374463bfb85e53ffe33927 P 728d6aea5e7e4b35b87e2439f6a27d51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "478cc56543fe4e9685812617d9375373" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250815 13:53:16.572911 1469 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "7f50e5209a374463bfb85e53ffe33927"
dest_uuid: "31d9f7ba4dcd46feb4cdc8acf04e3fce"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33152
I20250815 13:53:16.573803 1469 raft_consensus.cc:604] T 7f50e5209a374463bfb85e53ffe33927 P 31d9f7ba4dcd46feb4cdc8acf04e3fce [term 2 FOLLOWER]: Received request to transfer leadership
I20250815 13:53:16.574239 1469 raft_consensus.cc:612] T 7f50e5209a374463bfb85e53ffe33927 P 31d9f7ba4dcd46feb4cdc8acf04e3fce [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250815 13:53:17.621086 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 1385
I20250815 13:53:17.645818 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 1518
I20250815 13:53:17.673048 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 1651
I20250815 13:53:17.699177 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 1293
2025-08-15T13:53:17Z chronyd exiting
[ OK ] AdminCliTest.TestGracefulSpecificLeaderStepDown (14410 ms)
[ RUN ] AdminCliTest.TestDescribeTableColumnFlags
I20250815 13:53:17.758288 426 test_util.cc:276] Using random seed: -1236771027
I20250815 13:53:17.763232 426 ts_itest-base.cc:115] Starting cluster with:
I20250815 13:53:17.763438 426 ts_itest-base.cc:116] --------------
I20250815 13:53:17.763592 426 ts_itest-base.cc:117] 3 tablet servers
I20250815 13:53:17.763732 426 ts_itest-base.cc:118] 3 replicas per TS
I20250815 13:53:17.763868 426 ts_itest-base.cc:119] --------------
2025-08-15T13:53:17Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-15T13:53:17Z Disabled control of system clock
I20250815 13:53:17.807214 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.190:33945
--webserver_interface=127.0.106.190
--webserver_port=0
--builtin_ntp_servers=127.0.106.148:35329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.106.190:33945 with env {}
W20250815 13:53:18.107252 1890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:53:18.107878 1890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:53:18.108345 1890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:53:18.139434 1890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 13:53:18.139778 1890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:53:18.140028 1890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 13:53:18.140275 1890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 13:53:18.176383 1890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:35329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.106.190:33945
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.190:33945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.106.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:53:18.177696 1890 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:53:18.179315 1890 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:53:18.190718 1896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:53:18.191426 1897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:53:19.386374 1899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:53:19.389585 1898 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1192 milliseconds
W20250815 13:53:19.390197 1890 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.200s user 0.401s sys 0.793s
W20250815 13:53:19.390628 1890 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.200s user 0.401s sys 0.793s
I20250815 13:53:19.390936 1890 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 13:53:19.392665 1890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:53:19.396544 1890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:53:19.398046 1890 hybrid_clock.cc:648] HybridClock initialized: now 1755265999397965 us; error 81 us; skew 500 ppm
I20250815 13:53:19.399302 1890 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:53:19.409284 1890 webserver.cc:480] Webserver started at http://127.0.106.190:46811/ using document root <none> and password file <none>
I20250815 13:53:19.410773 1890 fs_manager.cc:362] Metadata directory not provided
I20250815 13:53:19.411110 1890 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:53:19.411818 1890 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 13:53:19.418956 1890 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "4bf8c68e29fb464083b90007799a98f9"
format_stamp: "Formatted at 2025-08-15 13:53:19 on dist-test-slave-kt02"
I20250815 13:53:19.420701 1890 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "4bf8c68e29fb464083b90007799a98f9"
format_stamp: "Formatted at 2025-08-15 13:53:19 on dist-test-slave-kt02"
I20250815 13:53:19.432020 1890 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.009s sys 0.002s
I20250815 13:53:19.440855 1906 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:53:19.442441 1890 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.004s
I20250815 13:53:19.442945 1890 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
uuid: "4bf8c68e29fb464083b90007799a98f9"
format_stamp: "Formatted at 2025-08-15 13:53:19 on dist-test-slave-kt02"
I20250815 13:53:19.443508 1890 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:53:19.523715 1890 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:53:19.525138 1890 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:53:19.525556 1890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:53:19.605510 1890 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.190:33945
I20250815 13:53:19.605690 1957 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.190:33945 every 8 connection(s)
I20250815 13:53:19.608394 1890 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250815 13:53:19.613016 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 1890
I20250815 13:53:19.613757 426 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250815 13:53:19.615917 1958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:53:19.646673 1958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bf8c68e29fb464083b90007799a98f9: Bootstrap starting.
I20250815 13:53:19.653752 1958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bf8c68e29fb464083b90007799a98f9: Neither blocks nor log segments found. Creating new log.
I20250815 13:53:19.655869 1958 log.cc:826] T 00000000000000000000000000000000 P 4bf8c68e29fb464083b90007799a98f9: Log is configured to *not* fsync() on all Append() calls
I20250815 13:53:19.661326 1958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bf8c68e29fb464083b90007799a98f9: No bootstrap required, opened a new log
I20250815 13:53:19.681044 1958 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4bf8c68e29fb464083b90007799a98f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf8c68e29fb464083b90007799a98f9" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 33945 } }
I20250815 13:53:19.681839 1958 raft_consensus.cc:383] T 00000000000000000000000000000000 P 4bf8c68e29fb464083b90007799a98f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 13:53:19.682089 1958 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4bf8c68e29fb464083b90007799a98f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bf8c68e29fb464083b90007799a98f9, State: Initialized, Role: FOLLOWER
I20250815 13:53:19.682895 1958 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bf8c68e29fb464083b90007799a98f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf8c68e29fb464083b90007799a98f9" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 33945 } }
I20250815 13:53:19.683476 1958 raft_consensus.cc:397] T 00000000000000000000000000000000 P 4bf8c68e29fb464083b90007799a98f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 13:53:19.683758 1958 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4bf8c68e29fb464083b90007799a98f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 13:53:19.684068 1958 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4bf8c68e29fb464083b90007799a98f9 [term 0 FOLLOWER]: Advancing to term 1
I20250815 13:53:19.688616 1958 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4bf8c68e29fb464083b90007799a98f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf8c68e29fb464083b90007799a98f9" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 33945 } }
I20250815 13:53:19.689522 1958 leader_election.cc:304] T 00000000000000000000000000000000 P 4bf8c68e29fb464083b90007799a98f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bf8c68e29fb464083b90007799a98f9; no voters:
I20250815 13:53:19.691576 1958 leader_election.cc:290] T 00000000000000000000000000000000 P 4bf8c68e29fb464083b90007799a98f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 13:53:19.692392 1963 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4bf8c68e29fb464083b90007799a98f9 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 13:53:19.694705 1963 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4bf8c68e29fb464083b90007799a98f9 [term 1 LEADER]: Becoming Leader. State: Replica: 4bf8c68e29fb464083b90007799a98f9, State: Running, Role: LEADER
I20250815 13:53:19.695616 1963 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bf8c68e29fb464083b90007799a98f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf8c68e29fb464083b90007799a98f9" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 33945 } }
I20250815 13:53:19.696769 1958 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bf8c68e29fb464083b90007799a98f9 [sys.catalog]: configured and running, proceeding with master startup.
I20250815 13:53:19.703154 1965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bf8c68e29fb464083b90007799a98f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bf8c68e29fb464083b90007799a98f9. Latest consensus state: current_term: 1 leader_uuid: "4bf8c68e29fb464083b90007799a98f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf8c68e29fb464083b90007799a98f9" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 33945 } } }
I20250815 13:53:19.704120 1965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bf8c68e29fb464083b90007799a98f9 [sys.catalog]: This master's current role is: LEADER
I20250815 13:53:19.704521 1964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bf8c68e29fb464083b90007799a98f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bf8c68e29fb464083b90007799a98f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf8c68e29fb464083b90007799a98f9" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 33945 } } }
I20250815 13:53:19.705368 1964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bf8c68e29fb464083b90007799a98f9 [sys.catalog]: This master's current role is: LEADER
I20250815 13:53:19.709390 1970 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 13:53:19.723757 1970 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 13:53:19.742022 1970 catalog_manager.cc:1349] Generated new cluster ID: ea126d456ca54c7b85abfe15eaf227b8
I20250815 13:53:19.742384 1970 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 13:53:19.756176 1970 catalog_manager.cc:1372] Generated new certificate authority record
I20250815 13:53:19.757774 1970 catalog_manager.cc:1506] Loading token signing keys...
I20250815 13:53:19.776103 1970 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 4bf8c68e29fb464083b90007799a98f9: Generated new TSK 0
I20250815 13:53:19.777072 1970 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250815 13:53:19.793956 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.129:0
--local_ip_for_outbound_sockets=127.0.106.129
--webserver_interface=127.0.106.129
--webserver_port=0
--tserver_master_addrs=127.0.106.190:33945
--builtin_ntp_servers=127.0.106.148:35329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250815 13:53:20.098677 1982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:53:20.099195 1982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:53:20.099697 1982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:53:20.131222 1982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:53:20.132069 1982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.129
I20250815 13:53:20.167369 1982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:35329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.106.129
--webserver_port=0
--tserver_master_addrs=127.0.106.190:33945
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.129
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:53:20.168654 1982 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:53:20.170223 1982 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:53:20.183398 1988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:53:21.586115 1987 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 1982
W20250815 13:53:21.735708 1982 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.551s user 0.570s sys 0.919s
W20250815 13:53:20.184744 1989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:53:21.737344 1982 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.553s user 0.570s sys 0.919s
W20250815 13:53:21.738173 1990 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1551 milliseconds
W20250815 13:53:21.738248 1991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 13:53:21.738672 1982 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 13:53:21.742667 1982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:53:21.744896 1982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:53:21.746287 1982 hybrid_clock.cc:648] HybridClock initialized: now 1755266001746236 us; error 83 us; skew 500 ppm
I20250815 13:53:21.747155 1982 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:53:21.753718 1982 webserver.cc:480] Webserver started at http://127.0.106.129:37425/ using document root <none> and password file <none>
I20250815 13:53:21.754662 1982 fs_manager.cc:362] Metadata directory not provided
I20250815 13:53:21.754879 1982 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:53:21.755386 1982 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 13:53:21.760027 1982 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "f395daa183804551a30a8b85c95d579d"
format_stamp: "Formatted at 2025-08-15 13:53:21 on dist-test-slave-kt02"
I20250815 13:53:21.761195 1982 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "f395daa183804551a30a8b85c95d579d"
format_stamp: "Formatted at 2025-08-15 13:53:21 on dist-test-slave-kt02"
I20250815 13:53:21.769047 1982 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20250815 13:53:21.775758 1998 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:53:21.777235 1982 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20250815 13:53:21.777647 1982 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f395daa183804551a30a8b85c95d579d"
format_stamp: "Formatted at 2025-08-15 13:53:21 on dist-test-slave-kt02"
I20250815 13:53:21.777992 1982 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:53:21.838945 1982 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:53:21.840489 1982 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:53:21.840925 1982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:53:21.843724 1982 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 13:53:21.847931 1982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 13:53:21.848137 1982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 13:53:21.848389 1982 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 13:53:21.848544 1982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 13:53:22.012960 1982 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.129:45197
I20250815 13:53:22.013298 2110 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.129:45197 every 8 connection(s)
I20250815 13:53:22.016005 1982 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250815 13:53:22.025120 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 1982
I20250815 13:53:22.025588 426 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250815 13:53:22.033169 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.130:0
--local_ip_for_outbound_sockets=127.0.106.130
--webserver_interface=127.0.106.130
--webserver_port=0
--tserver_master_addrs=127.0.106.190:33945
--builtin_ntp_servers=127.0.106.148:35329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 13:53:22.045030 2111 heartbeater.cc:344] Connected to a master server at 127.0.106.190:33945
I20250815 13:53:22.045682 2111 heartbeater.cc:461] Registering TS with master...
I20250815 13:53:22.047266 2111 heartbeater.cc:507] Master 127.0.106.190:33945 requested a full tablet report, sending...
I20250815 13:53:22.051312 1923 ts_manager.cc:194] Registered new tserver with Master: f395daa183804551a30a8b85c95d579d (127.0.106.129:45197)
I20250815 13:53:22.054497 1923 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.129:47887
W20250815 13:53:22.340878 2115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:53:22.341334 2115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:53:22.341784 2115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:53:22.373773 2115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:53:22.374620 2115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.130
I20250815 13:53:22.410056 2115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:35329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.106.130
--webserver_port=0
--tserver_master_addrs=127.0.106.190:33945
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.130
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:53:22.411427 2115 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:53:22.413061 2115 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:53:22.425544 2122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 13:53:23.060997 2111 heartbeater.cc:499] Master 127.0.106.190:33945 was elected leader, sending a full tablet report...
W20250815 13:53:22.425843 2123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:53:23.786134 2125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:53:23.788226 2115 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.363s user 0.460s sys 0.896s
W20250815 13:53:23.788642 2115 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.364s user 0.460s sys 0.896s
W20250815 13:53:23.790230 2124 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1356 milliseconds
I20250815 13:53:23.790241 2115 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 13:53:23.791734 2115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:53:23.794163 2115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:53:23.795534 2115 hybrid_clock.cc:648] HybridClock initialized: now 1755266003795499 us; error 70 us; skew 500 ppm
I20250815 13:53:23.796360 2115 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:53:23.804206 2115 webserver.cc:480] Webserver started at http://127.0.106.130:46369/ using document root <none> and password file <none>
I20250815 13:53:23.805330 2115 fs_manager.cc:362] Metadata directory not provided
I20250815 13:53:23.805548 2115 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:53:23.806005 2115 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 13:53:23.810575 2115 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "b3ed9c90b00b4a989da29172e5a58438"
format_stamp: "Formatted at 2025-08-15 13:53:23 on dist-test-slave-kt02"
I20250815 13:53:23.811709 2115 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "b3ed9c90b00b4a989da29172e5a58438"
format_stamp: "Formatted at 2025-08-15 13:53:23 on dist-test-slave-kt02"
I20250815 13:53:23.819883 2115 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.001s sys 0.006s
I20250815 13:53:23.826443 2132 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:53:23.827857 2115 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20250815 13:53:23.828218 2115 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "b3ed9c90b00b4a989da29172e5a58438"
format_stamp: "Formatted at 2025-08-15 13:53:23 on dist-test-slave-kt02"
I20250815 13:53:23.828570 2115 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:53:23.908581 2115 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:53:23.910053 2115 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:53:23.910531 2115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:53:23.913116 2115 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 13:53:23.917352 2115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 13:53:23.917585 2115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 13:53:23.917793 2115 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 13:53:23.917929 2115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 13:53:24.059015 2115 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.130:40785
I20250815 13:53:24.059268 2244 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.130:40785 every 8 connection(s)
I20250815 13:53:24.061753 2115 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250815 13:53:24.072445 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 2115
I20250815 13:53:24.073177 426 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250815 13:53:24.081802 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.131:0
--local_ip_for_outbound_sockets=127.0.106.131
--webserver_interface=127.0.106.131
--webserver_port=0
--tserver_master_addrs=127.0.106.190:33945
--builtin_ntp_servers=127.0.106.148:35329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 13:53:24.089762 2245 heartbeater.cc:344] Connected to a master server at 127.0.106.190:33945
I20250815 13:53:24.090507 2245 heartbeater.cc:461] Registering TS with master...
I20250815 13:53:24.092240 2245 heartbeater.cc:507] Master 127.0.106.190:33945 requested a full tablet report, sending...
I20250815 13:53:24.095242 1923 ts_manager.cc:194] Registered new tserver with Master: b3ed9c90b00b4a989da29172e5a58438 (127.0.106.130:40785)
I20250815 13:53:24.096658 1923 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.130:43049
W20250815 13:53:24.386839 2249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:53:24.387401 2249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:53:24.387890 2249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:53:24.418978 2249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:53:24.419857 2249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.131
I20250815 13:53:24.454559 2249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:35329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.106.131
--webserver_port=0
--tserver_master_addrs=127.0.106.190:33945
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.131
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:53:24.455932 2249 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:53:24.457504 2249 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:53:24.470702 2255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 13:53:25.101146 2245 heartbeater.cc:499] Master 127.0.106.190:33945 was elected leader, sending a full tablet report...
W20250815 13:53:24.472167 2256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:53:24.475301 2258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:53:25.582394 2257 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1106 milliseconds
I20250815 13:53:25.582530 2249 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 13:53:25.583860 2249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:53:25.586154 2249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:53:25.587567 2249 hybrid_clock.cc:648] HybridClock initialized: now 1755266005587517 us; error 67 us; skew 500 ppm
I20250815 13:53:25.588394 2249 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:53:25.594729 2249 webserver.cc:480] Webserver started at http://127.0.106.131:38009/ using document root <none> and password file <none>
I20250815 13:53:25.595829 2249 fs_manager.cc:362] Metadata directory not provided
I20250815 13:53:25.596072 2249 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:53:25.596549 2249 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 13:53:25.600970 2249 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "dc14e23fc1a442369d076203388e8238"
format_stamp: "Formatted at 2025-08-15 13:53:25 on dist-test-slave-kt02"
I20250815 13:53:25.602046 2249 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "dc14e23fc1a442369d076203388e8238"
format_stamp: "Formatted at 2025-08-15 13:53:25 on dist-test-slave-kt02"
I20250815 13:53:25.611078 2249 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.010s sys 0.001s
I20250815 13:53:25.618744 2265 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:53:25.620178 2249 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20250815 13:53:25.620635 2249 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "dc14e23fc1a442369d076203388e8238"
format_stamp: "Formatted at 2025-08-15 13:53:25 on dist-test-slave-kt02"
I20250815 13:53:25.621059 2249 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:53:25.670797 2249 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:53:25.672322 2249 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:53:25.672767 2249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:53:25.675631 2249 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 13:53:25.679839 2249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 13:53:25.680056 2249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 13:53:25.680301 2249 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 13:53:25.680459 2249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 13:53:25.818580 2249 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.131:38805
I20250815 13:53:25.818746 2377 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.131:38805 every 8 connection(s)
I20250815 13:53:25.821118 2249 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250815 13:53:25.824785 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 2249
I20250815 13:53:25.825330 426 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250815 13:53:25.848074 2378 heartbeater.cc:344] Connected to a master server at 127.0.106.190:33945
I20250815 13:53:25.848584 2378 heartbeater.cc:461] Registering TS with master...
I20250815 13:53:25.849671 2378 heartbeater.cc:507] Master 127.0.106.190:33945 requested a full tablet report, sending...
I20250815 13:53:25.851866 1923 ts_manager.cc:194] Registered new tserver with Master: dc14e23fc1a442369d076203388e8238 (127.0.106.131:38805)
I20250815 13:53:25.853097 1923 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.131:52191
I20250815 13:53:25.861297 426 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 13:53:25.899963 1923 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56788:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250815 13:53:25.919090 1923 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250815 13:53:25.985072 2046 tablet_service.cc:1468] Processing CreateTablet for tablet 7ca6f9a1c69b47fb808152918b9801e6 (DEFAULT_TABLE table=TestTable [id=07c9bd2583c84edb8d1e4cfbf3279245]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 13:53:25.987493 2046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ca6f9a1c69b47fb808152918b9801e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:53:25.987283 2313 tablet_service.cc:1468] Processing CreateTablet for tablet 7ca6f9a1c69b47fb808152918b9801e6 (DEFAULT_TABLE table=TestTable [id=07c9bd2583c84edb8d1e4cfbf3279245]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 13:53:25.988427 2180 tablet_service.cc:1468] Processing CreateTablet for tablet 7ca6f9a1c69b47fb808152918b9801e6 (DEFAULT_TABLE table=TestTable [id=07c9bd2583c84edb8d1e4cfbf3279245]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 13:53:25.989372 2313 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ca6f9a1c69b47fb808152918b9801e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:53:25.990003 2180 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ca6f9a1c69b47fb808152918b9801e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:53:26.018503 2397 tablet_bootstrap.cc:492] T 7ca6f9a1c69b47fb808152918b9801e6 P b3ed9c90b00b4a989da29172e5a58438: Bootstrap starting.
I20250815 13:53:26.020035 2398 tablet_bootstrap.cc:492] T 7ca6f9a1c69b47fb808152918b9801e6 P f395daa183804551a30a8b85c95d579d: Bootstrap starting.
I20250815 13:53:26.025628 2399 tablet_bootstrap.cc:492] T 7ca6f9a1c69b47fb808152918b9801e6 P dc14e23fc1a442369d076203388e8238: Bootstrap starting.
I20250815 13:53:26.027812 2397 tablet_bootstrap.cc:654] T 7ca6f9a1c69b47fb808152918b9801e6 P b3ed9c90b00b4a989da29172e5a58438: Neither blocks nor log segments found. Creating new log.
I20250815 13:53:26.030557 2397 log.cc:826] T 7ca6f9a1c69b47fb808152918b9801e6 P b3ed9c90b00b4a989da29172e5a58438: Log is configured to *not* fsync() on all Append() calls
I20250815 13:53:26.031590 2398 tablet_bootstrap.cc:654] T 7ca6f9a1c69b47fb808152918b9801e6 P f395daa183804551a30a8b85c95d579d: Neither blocks nor log segments found. Creating new log.
I20250815 13:53:26.033105 2399 tablet_bootstrap.cc:654] T 7ca6f9a1c69b47fb808152918b9801e6 P dc14e23fc1a442369d076203388e8238: Neither blocks nor log segments found. Creating new log.
I20250815 13:53:26.034638 2398 log.cc:826] T 7ca6f9a1c69b47fb808152918b9801e6 P f395daa183804551a30a8b85c95d579d: Log is configured to *not* fsync() on all Append() calls
I20250815 13:53:26.035552 2399 log.cc:826] T 7ca6f9a1c69b47fb808152918b9801e6 P dc14e23fc1a442369d076203388e8238: Log is configured to *not* fsync() on all Append() calls
I20250815 13:53:26.042786 2397 tablet_bootstrap.cc:492] T 7ca6f9a1c69b47fb808152918b9801e6 P b3ed9c90b00b4a989da29172e5a58438: No bootstrap required, opened a new log
I20250815 13:53:26.044122 2397 ts_tablet_manager.cc:1397] T 7ca6f9a1c69b47fb808152918b9801e6 P b3ed9c90b00b4a989da29172e5a58438: Time spent bootstrapping tablet: real 0.026s user 0.014s sys 0.010s
I20250815 13:53:26.047530 2398 tablet_bootstrap.cc:492] T 7ca6f9a1c69b47fb808152918b9801e6 P f395daa183804551a30a8b85c95d579d: No bootstrap required, opened a new log
I20250815 13:53:26.048223 2398 ts_tablet_manager.cc:1397] T 7ca6f9a1c69b47fb808152918b9801e6 P f395daa183804551a30a8b85c95d579d: Time spent bootstrapping tablet: real 0.029s user 0.013s sys 0.009s
I20250815 13:53:26.049778 2399 tablet_bootstrap.cc:492] T 7ca6f9a1c69b47fb808152918b9801e6 P dc14e23fc1a442369d076203388e8238: No bootstrap required, opened a new log
I20250815 13:53:26.050390 2399 ts_tablet_manager.cc:1397] T 7ca6f9a1c69b47fb808152918b9801e6 P dc14e23fc1a442369d076203388e8238: Time spent bootstrapping tablet: real 0.025s user 0.013s sys 0.009s
I20250815 13:53:26.073321 2397 raft_consensus.cc:357] T 7ca6f9a1c69b47fb808152918b9801e6 P b3ed9c90b00b4a989da29172e5a58438 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3ed9c90b00b4a989da29172e5a58438" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 40785 } } peers { permanent_uuid: "f395daa183804551a30a8b85c95d579d" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 45197 } } peers { permanent_uuid: "dc14e23fc1a442369d076203388e8238" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 38805 } }
I20250815 13:53:26.074681 2397 raft_consensus.cc:383] T 7ca6f9a1c69b47fb808152918b9801e6 P b3ed9c90b00b4a989da29172e5a58438 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 13:53:26.075096 2397 raft_consensus.cc:738] T 7ca6f9a1c69b47fb808152918b9801e6 P b3ed9c90b00b4a989da29172e5a58438 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3ed9c90b00b4a989da29172e5a58438, State: Initialized, Role: FOLLOWER
I20250815 13:53:26.075994 2399 raft_consensus.cc:357] T 7ca6f9a1c69b47fb808152918b9801e6 P dc14e23fc1a442369d076203388e8238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3ed9c90b00b4a989da29172e5a58438" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 40785 } } peers { permanent_uuid: "f395daa183804551a30a8b85c95d579d" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 45197 } } peers { permanent_uuid: "dc14e23fc1a442369d076203388e8238" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 38805 } }
I20250815 13:53:26.076350 2397 consensus_queue.cc:260] T 7ca6f9a1c69b47fb808152918b9801e6 P b3ed9c90b00b4a989da29172e5a58438 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3ed9c90b00b4a989da29172e5a58438" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 40785 } } peers { permanent_uuid: "f395daa183804551a30a8b85c95d579d" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 45197 } } peers { permanent_uuid: "dc14e23fc1a442369d076203388e8238" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 38805 } }
I20250815 13:53:26.076999 2399 raft_consensus.cc:383] T 7ca6f9a1c69b47fb808152918b9801e6 P dc14e23fc1a442369d076203388e8238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 13:53:26.077467 2399 raft_consensus.cc:738] T 7ca6f9a1c69b47fb808152918b9801e6 P dc14e23fc1a442369d076203388e8238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc14e23fc1a442369d076203388e8238, State: Initialized, Role: FOLLOWER
I20250815 13:53:26.077064 2398 raft_consensus.cc:357] T 7ca6f9a1c69b47fb808152918b9801e6 P f395daa183804551a30a8b85c95d579d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3ed9c90b00b4a989da29172e5a58438" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 40785 } } peers { permanent_uuid: "f395daa183804551a30a8b85c95d579d" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 45197 } } peers { permanent_uuid: "dc14e23fc1a442369d076203388e8238" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 38805 } }
I20250815 13:53:26.078220 2398 raft_consensus.cc:383] T 7ca6f9a1c69b47fb808152918b9801e6 P f395daa183804551a30a8b85c95d579d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 13:53:26.078547 2398 raft_consensus.cc:738] T 7ca6f9a1c69b47fb808152918b9801e6 P f395daa183804551a30a8b85c95d579d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f395daa183804551a30a8b85c95d579d, State: Initialized, Role: FOLLOWER
I20250815 13:53:26.078408 2399 consensus_queue.cc:260] T 7ca6f9a1c69b47fb808152918b9801e6 P dc14e23fc1a442369d076203388e8238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3ed9c90b00b4a989da29172e5a58438" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 40785 } } peers { permanent_uuid: "f395daa183804551a30a8b85c95d579d" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 45197 } } peers { permanent_uuid: "dc14e23fc1a442369d076203388e8238" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 38805 } }
I20250815 13:53:26.079478 2398 consensus_queue.cc:260] T 7ca6f9a1c69b47fb808152918b9801e6 P f395daa183804551a30a8b85c95d579d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3ed9c90b00b4a989da29172e5a58438" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 40785 } } peers { permanent_uuid: "f395daa183804551a30a8b85c95d579d" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 45197 } } peers { permanent_uuid: "dc14e23fc1a442369d076203388e8238" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 38805 } }
I20250815 13:53:26.090202 2398 ts_tablet_manager.cc:1428] T 7ca6f9a1c69b47fb808152918b9801e6 P f395daa183804551a30a8b85c95d579d: Time spent starting tablet: real 0.042s user 0.029s sys 0.009s
I20250815 13:53:26.092869 2397 ts_tablet_manager.cc:1428] T 7ca6f9a1c69b47fb808152918b9801e6 P b3ed9c90b00b4a989da29172e5a58438: Time spent starting tablet: real 0.048s user 0.033s sys 0.003s
I20250815 13:53:26.095645 2378 heartbeater.cc:499] Master 127.0.106.190:33945 was elected leader, sending a full tablet report...
I20250815 13:53:26.097093 2399 ts_tablet_manager.cc:1428] T 7ca6f9a1c69b47fb808152918b9801e6 P dc14e23fc1a442369d076203388e8238: Time spent starting tablet: real 0.046s user 0.032s sys 0.012s
W20250815 13:53:26.276310 2112 tablet.cc:2378] T 7ca6f9a1c69b47fb808152918b9801e6 P f395daa183804551a30a8b85c95d579d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250815 13:53:26.319834 2246 tablet.cc:2378] T 7ca6f9a1c69b47fb808152918b9801e6 P b3ed9c90b00b4a989da29172e5a58438: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250815 13:53:26.325839 2379 tablet.cc:2378] T 7ca6f9a1c69b47fb808152918b9801e6 P dc14e23fc1a442369d076203388e8238: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 13:53:26.426666 2405 raft_consensus.cc:491] T 7ca6f9a1c69b47fb808152918b9801e6 P b3ed9c90b00b4a989da29172e5a58438 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 13:53:26.427301 2405 raft_consensus.cc:513] T 7ca6f9a1c69b47fb808152918b9801e6 P b3ed9c90b00b4a989da29172e5a58438 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3ed9c90b00b4a989da29172e5a58438" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 40785 } } peers { permanent_uuid: "f395daa183804551a30a8b85c95d579d" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 45197 } } peers { permanent_uuid: "dc14e23fc1a442369d076203388e8238" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 38805 } }
I20250815 13:53:26.429961 2405 leader_election.cc:290] T 7ca6f9a1c69b47fb808152918b9801e6 P b3ed9c90b00b4a989da29172e5a58438 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f395daa183804551a30a8b85c95d579d (127.0.106.129:45197), dc14e23fc1a442369d076203388e8238 (127.0.106.131:38805)
I20250815 13:53:26.431783 2403 raft_consensus.cc:491] T 7ca6f9a1c69b47fb808152918b9801e6 P dc14e23fc1a442369d076203388e8238 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 13:53:26.432607 2403 raft_consensus.cc:513] T 7ca6f9a1c69b47fb808152918b9801e6 P dc14e23fc1a442369d076203388e8238 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3ed9c90b00b4a989da29172e5a58438" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 40785 } } peers { permanent_uuid: "f395daa183804551a30a8b85c95d579d" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 45197 } } peers { permanent_uuid: "dc14e23fc1a442369d076203388e8238" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 38805 } }
I20250815 13:53:26.442097 2403 leader_election.cc:290] T 7ca6f9a1c69b47fb808152918b9801e6 P dc14e23fc1a442369d076203388e8238 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3ed9c90b00b4a989da29172e5a58438 (127.0.106.130:40785), f395daa183804551a30a8b85c95d579d (127.0.106.129:45197)
I20250815 13:53:26.449967 2333 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca6f9a1c69b47fb808152918b9801e6" candidate_uuid: "b3ed9c90b00b4a989da29172e5a58438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc14e23fc1a442369d076203388e8238" is_pre_election: true
I20250815 13:53:26.450824 2333 raft_consensus.cc:2466] T 7ca6f9a1c69b47fb808152918b9801e6 P dc14e23fc1a442369d076203388e8238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3ed9c90b00b4a989da29172e5a58438 in term 0.
I20250815 13:53:26.452575 2134 leader_election.cc:304] T 7ca6f9a1c69b47fb808152918b9801e6 P b3ed9c90b00b4a989da29172e5a58438 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3ed9c90b00b4a989da29172e5a58438, dc14e23fc1a442369d076203388e8238; no voters:
I20250815 13:53:26.453706 2405 raft_consensus.cc:2802] T 7ca6f9a1c69b47fb808152918b9801e6 P b3ed9c90b00b4a989da29172e5a58438 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250815 13:53:26.454119 2405 raft_consensus.cc:491] T 7ca6f9a1c69b47fb808152918b9801e6 P b3ed9c90b00b4a989da29172e5a58438 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 13:53:26.454519 2405 raft_consensus.cc:3058] T 7ca6f9a1c69b47fb808152918b9801e6 P b3ed9c90b00b4a989da29172e5a58438 [term 0 FOLLOWER]: Advancing to term 1
I20250815 13:53:26.457686 2066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca6f9a1c69b47fb808152918b9801e6" candidate_uuid: "b3ed9c90b00b4a989da29172e5a58438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f395daa183804551a30a8b85c95d579d" is_pre_election: true
I20250815 13:53:26.458701 2066 raft_consensus.cc:2466] T 7ca6f9a1c69b47fb808152918b9801e6 P f395daa183804551a30a8b85c95d579d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3ed9c90b00b4a989da29172e5a58438 in term 0.
I20250815 13:53:26.461226 2066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca6f9a1c69b47fb808152918b9801e6" candidate_uuid: "dc14e23fc1a442369d076203388e8238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f395daa183804551a30a8b85c95d579d" is_pre_election: true
I20250815 13:53:26.461894 2066 raft_consensus.cc:2466] T 7ca6f9a1c69b47fb808152918b9801e6 P f395daa183804551a30a8b85c95d579d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc14e23fc1a442369d076203388e8238 in term 0.
I20250815 13:53:26.461692 2405 raft_consensus.cc:513] T 7ca6f9a1c69b47fb808152918b9801e6 P b3ed9c90b00b4a989da29172e5a58438 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3ed9c90b00b4a989da29172e5a58438" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 40785 } } peers { permanent_uuid: "f395daa183804551a30a8b85c95d579d" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 45197 } } peers { permanent_uuid: "dc14e23fc1a442369d076203388e8238" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 38805 } }
I20250815 13:53:26.461701 2200 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca6f9a1c69b47fb808152918b9801e6" candidate_uuid: "dc14e23fc1a442369d076203388e8238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3ed9c90b00b4a989da29172e5a58438" is_pre_election: true
I20250815 13:53:26.462940 2200 raft_consensus.cc:2391] T 7ca6f9a1c69b47fb808152918b9801e6 P b3ed9c90b00b4a989da29172e5a58438 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dc14e23fc1a442369d076203388e8238 in current term 1: Already voted for candidate b3ed9c90b00b4a989da29172e5a58438 in this term.
I20250815 13:53:26.463305 2268 leader_election.cc:304] T 7ca6f9a1c69b47fb808152918b9801e6 P dc14e23fc1a442369d076203388e8238 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc14e23fc1a442369d076203388e8238, f395daa183804551a30a8b85c95d579d; no voters:
I20250815 13:53:26.464125 2405 leader_election.cc:290] T 7ca6f9a1c69b47fb808152918b9801e6 P b3ed9c90b00b4a989da29172e5a58438 [CANDIDATE]: Term 1 election: Requested vote from peers f395daa183804551a30a8b85c95d579d (127.0.106.129:45197), dc14e23fc1a442369d076203388e8238 (127.0.106.131:38805)
I20250815 13:53:26.464329 2403 raft_consensus.cc:2802] T 7ca6f9a1c69b47fb808152918b9801e6 P dc14e23fc1a442369d076203388e8238 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250815 13:53:26.464735 2403 raft_consensus.cc:491] T 7ca6f9a1c69b47fb808152918b9801e6 P dc14e23fc1a442369d076203388e8238 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 13:53:26.464828 2066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca6f9a1c69b47fb808152918b9801e6" candidate_uuid: "b3ed9c90b00b4a989da29172e5a58438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f395daa183804551a30a8b85c95d579d"
I20250815 13:53:26.464907 2333 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca6f9a1c69b47fb808152918b9801e6" candidate_uuid: "b3ed9c90b00b4a989da29172e5a58438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc14e23fc1a442369d076203388e8238"
I20250815 13:53:26.465095 2403 raft_consensus.cc:3058] T 7ca6f9a1c69b47fb808152918b9801e6 P dc14e23fc1a442369d076203388e8238 [term 0 FOLLOWER]: Advancing to term 1
I20250815 13:53:26.465406 2066 raft_consensus.cc:3058] T 7ca6f9a1c69b47fb808152918b9801e6 P f395daa183804551a30a8b85c95d579d [term 0 FOLLOWER]: Advancing to term 1
I20250815 13:53:26.472513 2066 raft_consensus.cc:2466] T 7ca6f9a1c69b47fb808152918b9801e6 P f395daa183804551a30a8b85c95d579d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3ed9c90b00b4a989da29172e5a58438 in term 1.
I20250815 13:53:26.472519 2403 raft_consensus.cc:513] T 7ca6f9a1c69b47fb808152918b9801e6 P dc14e23fc1a442369d076203388e8238 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3ed9c90b00b4a989da29172e5a58438" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 40785 } } peers { permanent_uuid: "f395daa183804551a30a8b85c95d579d" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 45197 } } peers { permanent_uuid: "dc14e23fc1a442369d076203388e8238" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 38805 } }
I20250815 13:53:26.473912 2333 raft_consensus.cc:2391] T 7ca6f9a1c69b47fb808152918b9801e6 P dc14e23fc1a442369d076203388e8238 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b3ed9c90b00b4a989da29172e5a58438 in current term 1: Already voted for candidate dc14e23fc1a442369d076203388e8238 in this term.
I20250815 13:53:26.473948 2135 leader_election.cc:304] T 7ca6f9a1c69b47fb808152918b9801e6 P b3ed9c90b00b4a989da29172e5a58438 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3ed9c90b00b4a989da29172e5a58438, f395daa183804551a30a8b85c95d579d; no voters:
I20250815 13:53:26.474740 2403 leader_election.cc:290] T 7ca6f9a1c69b47fb808152918b9801e6 P dc14e23fc1a442369d076203388e8238 [CANDIDATE]: Term 1 election: Requested vote from peers b3ed9c90b00b4a989da29172e5a58438 (127.0.106.130:40785), f395daa183804551a30a8b85c95d579d (127.0.106.129:45197)
I20250815 13:53:26.475014 2405 raft_consensus.cc:2802] T 7ca6f9a1c69b47fb808152918b9801e6 P b3ed9c90b00b4a989da29172e5a58438 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 13:53:26.475778 2200 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca6f9a1c69b47fb808152918b9801e6" candidate_uuid: "dc14e23fc1a442369d076203388e8238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3ed9c90b00b4a989da29172e5a58438"
I20250815 13:53:26.477232 2066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ca6f9a1c69b47fb808152918b9801e6" candidate_uuid: "dc14e23fc1a442369d076203388e8238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f395daa183804551a30a8b85c95d579d"
I20250815 13:53:26.477380 2405 raft_consensus.cc:695] T 7ca6f9a1c69b47fb808152918b9801e6 P b3ed9c90b00b4a989da29172e5a58438 [term 1 LEADER]: Becoming Leader. State: Replica: b3ed9c90b00b4a989da29172e5a58438, State: Running, Role: LEADER
I20250815 13:53:26.478104 2066 raft_consensus.cc:2391] T 7ca6f9a1c69b47fb808152918b9801e6 P f395daa183804551a30a8b85c95d579d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate dc14e23fc1a442369d076203388e8238 in current term 1: Already voted for candidate b3ed9c90b00b4a989da29172e5a58438 in this term.
I20250815 13:53:26.478539 2405 consensus_queue.cc:237] T 7ca6f9a1c69b47fb808152918b9801e6 P b3ed9c90b00b4a989da29172e5a58438 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3ed9c90b00b4a989da29172e5a58438" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 40785 } } peers { permanent_uuid: "f395daa183804551a30a8b85c95d579d" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 45197 } } peers { permanent_uuid: "dc14e23fc1a442369d076203388e8238" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 38805 } }
I20250815 13:53:26.489225 2266 leader_election.cc:304] T 7ca6f9a1c69b47fb808152918b9801e6 P dc14e23fc1a442369d076203388e8238 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc14e23fc1a442369d076203388e8238; no voters: b3ed9c90b00b4a989da29172e5a58438, f395daa183804551a30a8b85c95d579d
I20250815 13:53:26.490010 2403 raft_consensus.cc:2747] T 7ca6f9a1c69b47fb808152918b9801e6 P dc14e23fc1a442369d076203388e8238 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250815 13:53:26.490906 1923 catalog_manager.cc:5582] T 7ca6f9a1c69b47fb808152918b9801e6 P b3ed9c90b00b4a989da29172e5a58438 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3ed9c90b00b4a989da29172e5a58438 (127.0.106.130). New cstate: current_term: 1 leader_uuid: "b3ed9c90b00b4a989da29172e5a58438" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3ed9c90b00b4a989da29172e5a58438" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 40785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f395daa183804551a30a8b85c95d579d" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 45197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc14e23fc1a442369d076203388e8238" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 38805 } health_report { overall_health: UNKNOWN } } }
I20250815 13:53:26.573168 426 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 13:53:26.577155 426 ts_itest-base.cc:246] Waiting for 1 tablets on tserver f395daa183804551a30a8b85c95d579d to finish bootstrapping
I20250815 13:53:26.591809 426 ts_itest-base.cc:246] Waiting for 1 tablets on tserver b3ed9c90b00b4a989da29172e5a58438 to finish bootstrapping
I20250815 13:53:26.604224 426 ts_itest-base.cc:246] Waiting for 1 tablets on tserver dc14e23fc1a442369d076203388e8238 to finish bootstrapping
I20250815 13:53:26.620829 1923 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56788:
name: "TestAnotherTable"
schema {
columns {
name: "foo"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "bar"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
comment: "comment for bar"
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "foo"
}
}
}
W20250815 13:53:26.622339 1923 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAnotherTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250815 13:53:26.642761 2046 tablet_service.cc:1468] Processing CreateTablet for tablet 8f54ee5c21bf48b5bb54a9a01c4561d4 (DEFAULT_TABLE table=TestAnotherTable [id=08e4d01148404fd7a0334ca90d5e4140]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250815 13:53:26.643970 2046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f54ee5c21bf48b5bb54a9a01c4561d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:53:26.644233 2180 tablet_service.cc:1468] Processing CreateTablet for tablet 8f54ee5c21bf48b5bb54a9a01c4561d4 (DEFAULT_TABLE table=TestAnotherTable [id=08e4d01148404fd7a0334ca90d5e4140]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250815 13:53:26.645444 2180 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f54ee5c21bf48b5bb54a9a01c4561d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:53:26.649248 2313 tablet_service.cc:1468] Processing CreateTablet for tablet 8f54ee5c21bf48b5bb54a9a01c4561d4 (DEFAULT_TABLE table=TestAnotherTable [id=08e4d01148404fd7a0334ca90d5e4140]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250815 13:53:26.650463 2313 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f54ee5c21bf48b5bb54a9a01c4561d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:53:26.666525 2417 tablet_bootstrap.cc:492] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P f395daa183804551a30a8b85c95d579d: Bootstrap starting.
I20250815 13:53:26.667151 2418 tablet_bootstrap.cc:492] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P dc14e23fc1a442369d076203388e8238: Bootstrap starting.
I20250815 13:53:26.671594 2419 tablet_bootstrap.cc:492] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P b3ed9c90b00b4a989da29172e5a58438: Bootstrap starting.
I20250815 13:53:26.672966 2417 tablet_bootstrap.cc:654] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P f395daa183804551a30a8b85c95d579d: Neither blocks nor log segments found. Creating new log.
I20250815 13:53:26.673141 2418 tablet_bootstrap.cc:654] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P dc14e23fc1a442369d076203388e8238: Neither blocks nor log segments found. Creating new log.
I20250815 13:53:26.678627 2419 tablet_bootstrap.cc:654] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P b3ed9c90b00b4a989da29172e5a58438: Neither blocks nor log segments found. Creating new log.
I20250815 13:53:26.679546 2417 tablet_bootstrap.cc:492] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P f395daa183804551a30a8b85c95d579d: No bootstrap required, opened a new log
I20250815 13:53:26.679607 2418 tablet_bootstrap.cc:492] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P dc14e23fc1a442369d076203388e8238: No bootstrap required, opened a new log
I20250815 13:53:26.680084 2418 ts_tablet_manager.cc:1397] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P dc14e23fc1a442369d076203388e8238: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.001s
I20250815 13:53:26.680325 2417 ts_tablet_manager.cc:1397] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P f395daa183804551a30a8b85c95d579d: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20250815 13:53:26.683089 2418 raft_consensus.cc:357] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P dc14e23fc1a442369d076203388e8238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f395daa183804551a30a8b85c95d579d" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 45197 } } peers { permanent_uuid: "b3ed9c90b00b4a989da29172e5a58438" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 40785 } } peers { permanent_uuid: "dc14e23fc1a442369d076203388e8238" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 38805 } }
I20250815 13:53:26.683266 2417 raft_consensus.cc:357] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P f395daa183804551a30a8b85c95d579d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f395daa183804551a30a8b85c95d579d" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 45197 } } peers { permanent_uuid: "b3ed9c90b00b4a989da29172e5a58438" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 40785 } } peers { permanent_uuid: "dc14e23fc1a442369d076203388e8238" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 38805 } }
I20250815 13:53:26.684067 2418 raft_consensus.cc:383] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P dc14e23fc1a442369d076203388e8238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 13:53:26.684031 2417 raft_consensus.cc:383] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P f395daa183804551a30a8b85c95d579d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 13:53:26.684433 2418 raft_consensus.cc:738] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P dc14e23fc1a442369d076203388e8238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc14e23fc1a442369d076203388e8238, State: Initialized, Role: FOLLOWER
I20250815 13:53:26.684384 2417 raft_consensus.cc:738] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P f395daa183804551a30a8b85c95d579d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f395daa183804551a30a8b85c95d579d, State: Initialized, Role: FOLLOWER
I20250815 13:53:26.685356 2418 consensus_queue.cc:260] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P dc14e23fc1a442369d076203388e8238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f395daa183804551a30a8b85c95d579d" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 45197 } } peers { permanent_uuid: "b3ed9c90b00b4a989da29172e5a58438" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 40785 } } peers { permanent_uuid: "dc14e23fc1a442369d076203388e8238" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 38805 } }
I20250815 13:53:26.685223 2417 consensus_queue.cc:260] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P f395daa183804551a30a8b85c95d579d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f395daa183804551a30a8b85c95d579d" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 45197 } } peers { permanent_uuid: "b3ed9c90b00b4a989da29172e5a58438" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 40785 } } peers { permanent_uuid: "dc14e23fc1a442369d076203388e8238" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 38805 } }
I20250815 13:53:26.687659 2418 ts_tablet_manager.cc:1428] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P dc14e23fc1a442369d076203388e8238: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20250815 13:53:26.687825 2419 tablet_bootstrap.cc:492] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P b3ed9c90b00b4a989da29172e5a58438: No bootstrap required, opened a new log
I20250815 13:53:26.689630 2423 raft_consensus.cc:491] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P f395daa183804551a30a8b85c95d579d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 13:53:26.690165 2417 ts_tablet_manager.cc:1428] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P f395daa183804551a30a8b85c95d579d: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20250815 13:53:26.690270 2423 raft_consensus.cc:513] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P f395daa183804551a30a8b85c95d579d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f395daa183804551a30a8b85c95d579d" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 45197 } } peers { permanent_uuid: "b3ed9c90b00b4a989da29172e5a58438" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 40785 } } peers { permanent_uuid: "dc14e23fc1a442369d076203388e8238" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 38805 } }
I20250815 13:53:26.694116 2423 leader_election.cc:290] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P f395daa183804551a30a8b85c95d579d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3ed9c90b00b4a989da29172e5a58438 (127.0.106.130:40785), dc14e23fc1a442369d076203388e8238 (127.0.106.131:38805)
I20250815 13:53:26.695472 2419 ts_tablet_manager.cc:1397] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P b3ed9c90b00b4a989da29172e5a58438: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.010s
I20250815 13:53:26.701965 2419 raft_consensus.cc:357] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P b3ed9c90b00b4a989da29172e5a58438 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f395daa183804551a30a8b85c95d579d" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 45197 } } peers { permanent_uuid: "b3ed9c90b00b4a989da29172e5a58438" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 40785 } } peers { permanent_uuid: "dc14e23fc1a442369d076203388e8238" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 38805 } }
I20250815 13:53:26.705041 2419 raft_consensus.cc:383] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P b3ed9c90b00b4a989da29172e5a58438 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 13:53:26.705780 2419 raft_consensus.cc:738] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P b3ed9c90b00b4a989da29172e5a58438 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3ed9c90b00b4a989da29172e5a58438, State: Initialized, Role: FOLLOWER
I20250815 13:53:26.707005 2419 consensus_queue.cc:260] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P b3ed9c90b00b4a989da29172e5a58438 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f395daa183804551a30a8b85c95d579d" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 45197 } } peers { permanent_uuid: "b3ed9c90b00b4a989da29172e5a58438" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 40785 } } peers { permanent_uuid: "dc14e23fc1a442369d076203388e8238" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 38805 } }
I20250815 13:53:26.709018 2405 raft_consensus.cc:491] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P b3ed9c90b00b4a989da29172e5a58438 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 13:53:26.709370 2419 ts_tablet_manager.cc:1428] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P b3ed9c90b00b4a989da29172e5a58438: Time spent starting tablet: real 0.013s user 0.004s sys 0.002s
I20250815 13:53:26.709479 2405 raft_consensus.cc:513] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P b3ed9c90b00b4a989da29172e5a58438 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f395daa183804551a30a8b85c95d579d" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 45197 } } peers { permanent_uuid: "b3ed9c90b00b4a989da29172e5a58438" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 40785 } } peers { permanent_uuid: "dc14e23fc1a442369d076203388e8238" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 38805 } }
I20250815 13:53:26.711350 2405 leader_election.cc:290] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P b3ed9c90b00b4a989da29172e5a58438 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f395daa183804551a30a8b85c95d579d (127.0.106.129:45197), dc14e23fc1a442369d076203388e8238 (127.0.106.131:38805)
I20250815 13:53:26.712929 2333 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f54ee5c21bf48b5bb54a9a01c4561d4" candidate_uuid: "b3ed9c90b00b4a989da29172e5a58438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc14e23fc1a442369d076203388e8238" is_pre_election: true
I20250815 13:53:26.713536 2333 raft_consensus.cc:2466] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P dc14e23fc1a442369d076203388e8238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3ed9c90b00b4a989da29172e5a58438 in term 0.
I20250815 13:53:26.714499 2134 leader_election.cc:304] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P b3ed9c90b00b4a989da29172e5a58438 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3ed9c90b00b4a989da29172e5a58438, dc14e23fc1a442369d076203388e8238; no voters:
I20250815 13:53:26.714666 2066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f54ee5c21bf48b5bb54a9a01c4561d4" candidate_uuid: "b3ed9c90b00b4a989da29172e5a58438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f395daa183804551a30a8b85c95d579d" is_pre_election: true
I20250815 13:53:26.715602 2405 raft_consensus.cc:2802] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P b3ed9c90b00b4a989da29172e5a58438 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250815 13:53:26.716118 2405 raft_consensus.cc:491] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P b3ed9c90b00b4a989da29172e5a58438 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 13:53:26.716477 2405 raft_consensus.cc:3058] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P b3ed9c90b00b4a989da29172e5a58438 [term 0 FOLLOWER]: Advancing to term 1
I20250815 13:53:26.715276 2066 raft_consensus.cc:2466] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P f395daa183804551a30a8b85c95d579d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3ed9c90b00b4a989da29172e5a58438 in term 0.
I20250815 13:53:26.722997 2405 raft_consensus.cc:513] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P b3ed9c90b00b4a989da29172e5a58438 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f395daa183804551a30a8b85c95d579d" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 45197 } } peers { permanent_uuid: "b3ed9c90b00b4a989da29172e5a58438" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 40785 } } peers { permanent_uuid: "dc14e23fc1a442369d076203388e8238" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 38805 } }
I20250815 13:53:26.726639 2200 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f54ee5c21bf48b5bb54a9a01c4561d4" candidate_uuid: "f395daa183804551a30a8b85c95d579d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3ed9c90b00b4a989da29172e5a58438" is_pre_election: true
I20250815 13:53:26.726645 2405 leader_election.cc:290] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P b3ed9c90b00b4a989da29172e5a58438 [CANDIDATE]: Term 1 election: Requested vote from peers f395daa183804551a30a8b85c95d579d (127.0.106.129:45197), dc14e23fc1a442369d076203388e8238 (127.0.106.131:38805)
I20250815 13:53:26.727123 2066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f54ee5c21bf48b5bb54a9a01c4561d4" candidate_uuid: "b3ed9c90b00b4a989da29172e5a58438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f395daa183804551a30a8b85c95d579d"
I20250815 13:53:26.727545 2200 raft_consensus.cc:2391] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P b3ed9c90b00b4a989da29172e5a58438 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f395daa183804551a30a8b85c95d579d in current term 1: Already voted for candidate b3ed9c90b00b4a989da29172e5a58438 in this term.
I20250815 13:53:26.727648 2066 raft_consensus.cc:3058] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P f395daa183804551a30a8b85c95d579d [term 0 FOLLOWER]: Advancing to term 1
I20250815 13:53:26.727890 2333 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f54ee5c21bf48b5bb54a9a01c4561d4" candidate_uuid: "b3ed9c90b00b4a989da29172e5a58438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc14e23fc1a442369d076203388e8238"
I20250815 13:53:26.728374 2333 raft_consensus.cc:3058] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P dc14e23fc1a442369d076203388e8238 [term 0 FOLLOWER]: Advancing to term 1
I20250815 13:53:26.731855 2332 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f54ee5c21bf48b5bb54a9a01c4561d4" candidate_uuid: "f395daa183804551a30a8b85c95d579d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc14e23fc1a442369d076203388e8238" is_pre_election: true
I20250815 13:53:26.733686 2000 leader_election.cc:304] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P f395daa183804551a30a8b85c95d579d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f395daa183804551a30a8b85c95d579d; no voters: b3ed9c90b00b4a989da29172e5a58438, dc14e23fc1a442369d076203388e8238
I20250815 13:53:26.734939 2333 raft_consensus.cc:2466] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P dc14e23fc1a442369d076203388e8238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3ed9c90b00b4a989da29172e5a58438 in term 1.
I20250815 13:53:26.735839 2134 leader_election.cc:304] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P b3ed9c90b00b4a989da29172e5a58438 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3ed9c90b00b4a989da29172e5a58438, dc14e23fc1a442369d076203388e8238; no voters:
I20250815 13:53:26.736157 2066 raft_consensus.cc:2466] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P f395daa183804551a30a8b85c95d579d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3ed9c90b00b4a989da29172e5a58438 in term 1.
I20250815 13:53:26.736589 2405 raft_consensus.cc:2802] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P b3ed9c90b00b4a989da29172e5a58438 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 13:53:26.736939 2424 raft_consensus.cc:2747] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P f395daa183804551a30a8b85c95d579d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250815 13:53:26.738030 2405 raft_consensus.cc:695] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P b3ed9c90b00b4a989da29172e5a58438 [term 1 LEADER]: Becoming Leader. State: Replica: b3ed9c90b00b4a989da29172e5a58438, State: Running, Role: LEADER
I20250815 13:53:26.739085 2405 consensus_queue.cc:237] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P b3ed9c90b00b4a989da29172e5a58438 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f395daa183804551a30a8b85c95d579d" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 45197 } } peers { permanent_uuid: "b3ed9c90b00b4a989da29172e5a58438" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 40785 } } peers { permanent_uuid: "dc14e23fc1a442369d076203388e8238" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 38805 } }
I20250815 13:53:26.747684 1921 catalog_manager.cc:5582] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P b3ed9c90b00b4a989da29172e5a58438 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3ed9c90b00b4a989da29172e5a58438 (127.0.106.130). New cstate: current_term: 1 leader_uuid: "b3ed9c90b00b4a989da29172e5a58438" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f395daa183804551a30a8b85c95d579d" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 45197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3ed9c90b00b4a989da29172e5a58438" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 40785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc14e23fc1a442369d076203388e8238" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 38805 } health_report { overall_health: UNKNOWN } } }
I20250815 13:53:26.931478 2405 consensus_queue.cc:1035] T 7ca6f9a1c69b47fb808152918b9801e6 P b3ed9c90b00b4a989da29172e5a58438 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f395daa183804551a30a8b85c95d579d" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 45197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250815 13:53:26.951920 2411 consensus_queue.cc:1035] T 7ca6f9a1c69b47fb808152918b9801e6 P b3ed9c90b00b4a989da29172e5a58438 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc14e23fc1a442369d076203388e8238" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 38805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250815 13:53:27.085222 2428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:53:27.085819 2428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:53:27.118062 2428 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250815 13:53:27.122156 2411 consensus_queue.cc:1035] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P b3ed9c90b00b4a989da29172e5a58438 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc14e23fc1a442369d076203388e8238" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 38805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250815 13:53:27.134248 2405 consensus_queue.cc:1035] T 8f54ee5c21bf48b5bb54a9a01c4561d4 P b3ed9c90b00b4a989da29172e5a58438 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f395daa183804551a30a8b85c95d579d" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 45197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250815 13:53:28.351013 2428 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.188s user 0.505s sys 0.679s
W20250815 13:53:28.351356 2428 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.189s user 0.509s sys 0.679s
W20250815 13:53:29.739708 2450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:53:29.740463 2450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:53:29.780380 2450 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250815 13:53:30.980723 2450 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.160s user 0.515s sys 0.643s
W20250815 13:53:30.981158 2450 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.160s user 0.515s sys 0.643s
W20250815 13:53:32.418450 2464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:53:32.419243 2464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:53:32.452970 2464 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250815 13:53:33.664021 2464 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.169s user 0.498s sys 0.670s
W20250815 13:53:33.664572 2464 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.170s user 0.499s sys 0.671s
W20250815 13:53:35.095141 2481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:53:35.095861 2481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:53:35.129519 2481 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250815 13:53:36.337538 2481 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.169s user 0.458s sys 0.708s
W20250815 13:53:36.337941 2481 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.169s user 0.458s sys 0.708s
I20250815 13:53:37.426156 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 1982
I20250815 13:53:37.451315 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 2115
I20250815 13:53:37.474858 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 2249
I20250815 13:53:37.496248 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 1890
2025-08-15T13:53:37Z chronyd exiting
[ OK ] AdminCliTest.TestDescribeTableColumnFlags (19795 ms)
[ RUN ] AdminCliTest.TestAuthzResetCacheNotAuthorized
I20250815 13:53:37.554116 426 test_util.cc:276] Using random seed: -1216975201
I20250815 13:53:37.559221 426 ts_itest-base.cc:115] Starting cluster with:
I20250815 13:53:37.559423 426 ts_itest-base.cc:116] --------------
I20250815 13:53:37.559546 426 ts_itest-base.cc:117] 3 tablet servers
I20250815 13:53:37.559655 426 ts_itest-base.cc:118] 3 replicas per TS
I20250815 13:53:37.559770 426 ts_itest-base.cc:119] --------------
2025-08-15T13:53:37Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-15T13:53:37Z Disabled control of system clock
I20250815 13:53:37.602681 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.190:33107
--webserver_interface=127.0.106.190
--webserver_port=0
--builtin_ntp_servers=127.0.106.148:37851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.106.190:33107
--superuser_acl=no-such-user with env {}
W20250815 13:53:37.906736 2502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:53:37.907512 2502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:53:37.908077 2502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:53:37.942010 2502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 13:53:37.942428 2502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:53:37.942725 2502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 13:53:37.942987 2502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 13:53:37.982139 2502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:37851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.106.190:33107
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.190:33107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.106.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:53:37.983801 2502 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:53:37.985649 2502 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:53:37.998294 2508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:53:39.401758 2507 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ec0 after lost signal to thread 2502
W20250815 13:53:39.566931 2507 kernel_stack_watchdog.cc:198] Thread 2502 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250815 13:53:37.998577 2509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:53:39.567749 2502 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.570s user 0.620s sys 0.949s
W20250815 13:53:39.568310 2502 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.570s user 0.620s sys 0.949s
W20250815 13:53:39.570506 2511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:53:39.573993 2510 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1574 milliseconds
I20250815 13:53:39.574040 2502 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 13:53:39.575619 2502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:53:39.578415 2502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:53:39.579885 2502 hybrid_clock.cc:648] HybridClock initialized: now 1755266019579819 us; error 79 us; skew 500 ppm
I20250815 13:53:39.580907 2502 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:53:39.588505 2502 webserver.cc:480] Webserver started at http://127.0.106.190:38589/ using document root <none> and password file <none>
I20250815 13:53:39.589684 2502 fs_manager.cc:362] Metadata directory not provided
I20250815 13:53:39.589928 2502 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:53:39.590498 2502 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 13:53:39.595878 2502 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "8e941df414bc4397b613db5503bee8d4"
format_stamp: "Formatted at 2025-08-15 13:53:39 on dist-test-slave-kt02"
I20250815 13:53:39.597249 2502 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "8e941df414bc4397b613db5503bee8d4"
format_stamp: "Formatted at 2025-08-15 13:53:39 on dist-test-slave-kt02"
I20250815 13:53:39.605928 2502 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.005s sys 0.005s
I20250815 13:53:39.612524 2518 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:53:39.613860 2502 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.005s
I20250815 13:53:39.614284 2502 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
uuid: "8e941df414bc4397b613db5503bee8d4"
format_stamp: "Formatted at 2025-08-15 13:53:39 on dist-test-slave-kt02"
I20250815 13:53:39.614662 2502 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:53:39.666436 2502 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:53:39.668277 2502 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:53:39.668784 2502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:53:39.748648 2502 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.190:33107
I20250815 13:53:39.748715 2569 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.190:33107 every 8 connection(s)
I20250815 13:53:39.751879 2502 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250815 13:53:39.757547 2570 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:53:39.760833 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 2502
I20250815 13:53:39.761271 426 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250815 13:53:39.780738 2570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e941df414bc4397b613db5503bee8d4: Bootstrap starting.
I20250815 13:53:39.787406 2570 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e941df414bc4397b613db5503bee8d4: Neither blocks nor log segments found. Creating new log.
I20250815 13:53:39.789134 2570 log.cc:826] T 00000000000000000000000000000000 P 8e941df414bc4397b613db5503bee8d4: Log is configured to *not* fsync() on all Append() calls
I20250815 13:53:39.793905 2570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e941df414bc4397b613db5503bee8d4: No bootstrap required, opened a new log
I20250815 13:53:39.812052 2570 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8e941df414bc4397b613db5503bee8d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e941df414bc4397b613db5503bee8d4" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 33107 } }
I20250815 13:53:39.812740 2570 raft_consensus.cc:383] T 00000000000000000000000000000000 P 8e941df414bc4397b613db5503bee8d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 13:53:39.812995 2570 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8e941df414bc4397b613db5503bee8d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e941df414bc4397b613db5503bee8d4, State: Initialized, Role: FOLLOWER
I20250815 13:53:39.813760 2570 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e941df414bc4397b613db5503bee8d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e941df414bc4397b613db5503bee8d4" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 33107 } }
I20250815 13:53:39.814291 2570 raft_consensus.cc:397] T 00000000000000000000000000000000 P 8e941df414bc4397b613db5503bee8d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 13:53:39.814574 2570 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8e941df414bc4397b613db5503bee8d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 13:53:39.814996 2570 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 8e941df414bc4397b613db5503bee8d4 [term 0 FOLLOWER]: Advancing to term 1
I20250815 13:53:39.819429 2570 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8e941df414bc4397b613db5503bee8d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e941df414bc4397b613db5503bee8d4" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 33107 } }
I20250815 13:53:39.820174 2570 leader_election.cc:304] T 00000000000000000000000000000000 P 8e941df414bc4397b613db5503bee8d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e941df414bc4397b613db5503bee8d4; no voters:
I20250815 13:53:39.822010 2570 leader_election.cc:290] T 00000000000000000000000000000000 P 8e941df414bc4397b613db5503bee8d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 13:53:39.823232 2575 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 8e941df414bc4397b613db5503bee8d4 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 13:53:39.825839 2575 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8e941df414bc4397b613db5503bee8d4 [term 1 LEADER]: Becoming Leader. State: Replica: 8e941df414bc4397b613db5503bee8d4, State: Running, Role: LEADER
I20250815 13:53:39.826222 2570 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e941df414bc4397b613db5503bee8d4 [sys.catalog]: configured and running, proceeding with master startup.
I20250815 13:53:39.826813 2575 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e941df414bc4397b613db5503bee8d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e941df414bc4397b613db5503bee8d4" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 33107 } }
I20250815 13:53:39.838788 2576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e941df414bc4397b613db5503bee8d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e941df414bc4397b613db5503bee8d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e941df414bc4397b613db5503bee8d4" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 33107 } } }
I20250815 13:53:39.838713 2577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e941df414bc4397b613db5503bee8d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e941df414bc4397b613db5503bee8d4. Latest consensus state: current_term: 1 leader_uuid: "8e941df414bc4397b613db5503bee8d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e941df414bc4397b613db5503bee8d4" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 33107 } } }
I20250815 13:53:39.839519 2577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e941df414bc4397b613db5503bee8d4 [sys.catalog]: This master's current role is: LEADER
I20250815 13:53:39.839519 2576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e941df414bc4397b613db5503bee8d4 [sys.catalog]: This master's current role is: LEADER
I20250815 13:53:39.849676 2583 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 13:53:39.862358 2583 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 13:53:39.881705 2583 catalog_manager.cc:1349] Generated new cluster ID: 6639debf2b21419ea0fc789d28016cfc
I20250815 13:53:39.882042 2583 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 13:53:39.902904 2583 catalog_manager.cc:1372] Generated new certificate authority record
I20250815 13:53:39.904762 2583 catalog_manager.cc:1506] Loading token signing keys...
I20250815 13:53:39.930083 2583 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 8e941df414bc4397b613db5503bee8d4: Generated new TSK 0
I20250815 13:53:39.931083 2583 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250815 13:53:39.959134 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.129:0
--local_ip_for_outbound_sockets=127.0.106.129
--webserver_interface=127.0.106.129
--webserver_port=0
--tserver_master_addrs=127.0.106.190:33107
--builtin_ntp_servers=127.0.106.148:37851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250815 13:53:40.264469 2594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:53:40.264997 2594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:53:40.265511 2594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:53:40.298930 2594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:53:40.299818 2594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.129
I20250815 13:53:40.334887 2594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:37851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.106.129
--webserver_port=0
--tserver_master_addrs=127.0.106.190:33107
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.129
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:53:40.336277 2594 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:53:40.337826 2594 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:53:40.350507 2600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 13:53:40.358855 2594 server_base.cc:1047] running on GCE node
W20250815 13:53:40.358007 2601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:53:41.462391 2603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 13:53:41.463819 2594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:53:41.466423 2594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:53:41.467867 2594 hybrid_clock.cc:648] HybridClock initialized: now 1755266021467803 us; error 79 us; skew 500 ppm
I20250815 13:53:41.468752 2594 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:53:41.475808 2594 webserver.cc:480] Webserver started at http://127.0.106.129:36207/ using document root <none> and password file <none>
I20250815 13:53:41.476742 2594 fs_manager.cc:362] Metadata directory not provided
I20250815 13:53:41.476949 2594 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:53:41.477428 2594 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 13:53:41.481842 2594 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "49e971313af5480f9434879948ddff05"
format_stamp: "Formatted at 2025-08-15 13:53:41 on dist-test-slave-kt02"
I20250815 13:53:41.482966 2594 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "49e971313af5480f9434879948ddff05"
format_stamp: "Formatted at 2025-08-15 13:53:41 on dist-test-slave-kt02"
I20250815 13:53:41.490365 2594 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20250815 13:53:41.496763 2610 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:53:41.498286 2594 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.003s
I20250815 13:53:41.498790 2594 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "49e971313af5480f9434879948ddff05"
format_stamp: "Formatted at 2025-08-15 13:53:41 on dist-test-slave-kt02"
I20250815 13:53:41.499307 2594 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:53:41.562139 2594 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:53:41.563627 2594 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:53:41.564074 2594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:53:41.566709 2594 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 13:53:41.571074 2594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 13:53:41.571342 2594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250815 13:53:41.571594 2594 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 13:53:41.571776 2594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 13:53:41.729503 2594 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.129:43101
I20250815 13:53:41.729650 2722 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.129:43101 every 8 connection(s)
I20250815 13:53:41.732105 2594 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250815 13:53:41.735903 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 2594
I20250815 13:53:41.736420 426 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250815 13:53:41.743034 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.130:0
--local_ip_for_outbound_sockets=127.0.106.130
--webserver_interface=127.0.106.130
--webserver_port=0
--tserver_master_addrs=127.0.106.190:33107
--builtin_ntp_servers=127.0.106.148:37851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 13:53:41.763971 2723 heartbeater.cc:344] Connected to a master server at 127.0.106.190:33107
I20250815 13:53:41.764516 2723 heartbeater.cc:461] Registering TS with master...
I20250815 13:53:41.765888 2723 heartbeater.cc:507] Master 127.0.106.190:33107 requested a full tablet report, sending...
I20250815 13:53:41.769552 2535 ts_manager.cc:194] Registered new tserver with Master: 49e971313af5480f9434879948ddff05 (127.0.106.129:43101)
I20250815 13:53:41.772658 2535 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.129:33115
W20250815 13:53:42.073319 2727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:53:42.073884 2727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:53:42.074365 2727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:53:42.108338 2727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:53:42.109253 2727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.130
I20250815 13:53:42.147337 2727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:37851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.106.130
--webserver_port=0
--tserver_master_addrs=127.0.106.190:33107
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.130
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:53:42.148918 2727 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:53:42.151224 2727 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:53:42.166121 2733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 13:53:42.776950 2723 heartbeater.cc:499] Master 127.0.106.190:33107 was elected leader, sending a full tablet report...
W20250815 13:53:42.166677 2734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:53:43.296720 2736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:53:43.301734 2735 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1133 milliseconds
W20250815 13:53:43.301698 2727 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.135s user 0.418s sys 0.705s
W20250815 13:53:43.302186 2727 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.136s user 0.419s sys 0.705s
I20250815 13:53:43.302496 2727 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 13:53:43.303711 2727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:53:43.306623 2727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:53:43.308173 2727 hybrid_clock.cc:648] HybridClock initialized: now 1755266023308060 us; error 89 us; skew 500 ppm
I20250815 13:53:43.310088 2727 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:53:43.319816 2727 webserver.cc:480] Webserver started at http://127.0.106.130:36263/ using document root <none> and password file <none>
I20250815 13:53:43.321352 2727 fs_manager.cc:362] Metadata directory not provided
I20250815 13:53:43.321718 2727 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:53:43.322443 2727 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 13:53:43.328671 2727 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "315b45714dc74a91b53678ba4ba149b0"
format_stamp: "Formatted at 2025-08-15 13:53:43 on dist-test-slave-kt02"
I20250815 13:53:43.329895 2727 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "315b45714dc74a91b53678ba4ba149b0"
format_stamp: "Formatted at 2025-08-15 13:53:43 on dist-test-slave-kt02"
I20250815 13:53:43.339565 2727 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.010s sys 0.001s
I20250815 13:53:43.346529 2744 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:53:43.348098 2727 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.007s sys 0.000s
I20250815 13:53:43.348546 2727 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "315b45714dc74a91b53678ba4ba149b0"
format_stamp: "Formatted at 2025-08-15 13:53:43 on dist-test-slave-kt02"
I20250815 13:53:43.348923 2727 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:53:43.418175 2727 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:53:43.419936 2727 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:53:43.420398 2727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:53:43.423334 2727 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 13:53:43.428004 2727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 13:53:43.428224 2727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 13:53:43.428483 2727 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 13:53:43.428635 2727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 13:53:43.582959 2727 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.130:46303
I20250815 13:53:43.583161 2856 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.130:46303 every 8 connection(s)
I20250815 13:53:43.585901 2727 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250815 13:53:43.596407 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 2727
I20250815 13:53:43.597018 426 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250815 13:53:43.604529 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.131:0
--local_ip_for_outbound_sockets=127.0.106.131
--webserver_interface=127.0.106.131
--webserver_port=0
--tserver_master_addrs=127.0.106.190:33107
--builtin_ntp_servers=127.0.106.148:37851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 13:53:43.608394 2857 heartbeater.cc:344] Connected to a master server at 127.0.106.190:33107
I20250815 13:53:43.608877 2857 heartbeater.cc:461] Registering TS with master...
I20250815 13:53:43.609989 2857 heartbeater.cc:507] Master 127.0.106.190:33107 requested a full tablet report, sending...
I20250815 13:53:43.612377 2535 ts_manager.cc:194] Registered new tserver with Master: 315b45714dc74a91b53678ba4ba149b0 (127.0.106.130:46303)
I20250815 13:53:43.613631 2535 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.130:58595
W20250815 13:53:43.952922 2861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:53:43.953505 2861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:53:43.954048 2861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:53:43.988031 2861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:53:43.989010 2861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.131
I20250815 13:53:44.027971 2861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:37851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.106.131
--webserver_port=0
--tserver_master_addrs=127.0.106.190:33107
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.131
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:53:44.029573 2861 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:53:44.031395 2861 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:53:44.044996 2867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 13:53:44.617210 2857 heartbeater.cc:499] Master 127.0.106.190:33107 was elected leader, sending a full tablet report...
W20250815 13:53:45.447383 2866 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 2861
W20250815 13:53:45.518510 2861 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.473s user 0.656s sys 0.816s
W20250815 13:53:44.045979 2868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:53:45.519340 2861 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.474s user 0.656s sys 0.816s
W20250815 13:53:45.521853 2870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:53:45.526714 2869 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1480 milliseconds
I20250815 13:53:45.526744 2861 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 13:53:45.528947 2861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:53:45.531654 2861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:53:45.533211 2861 hybrid_clock.cc:648] HybridClock initialized: now 1755266025533122 us; error 83 us; skew 500 ppm
I20250815 13:53:45.534201 2861 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:53:45.541641 2861 webserver.cc:480] Webserver started at http://127.0.106.131:38443/ using document root <none> and password file <none>
I20250815 13:53:45.542855 2861 fs_manager.cc:362] Metadata directory not provided
I20250815 13:53:45.543128 2861 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:53:45.543676 2861 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 13:53:45.548702 2861 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "dc83690431414ca39ec025627efb6896"
format_stamp: "Formatted at 2025-08-15 13:53:45 on dist-test-slave-kt02"
I20250815 13:53:45.550030 2861 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "dc83690431414ca39ec025627efb6896"
format_stamp: "Formatted at 2025-08-15 13:53:45 on dist-test-slave-kt02"
I20250815 13:53:45.558449 2861 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.005s sys 0.005s
I20250815 13:53:45.565796 2877 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:53:45.567396 2861 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20250815 13:53:45.567834 2861 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "dc83690431414ca39ec025627efb6896"
format_stamp: "Formatted at 2025-08-15 13:53:45 on dist-test-slave-kt02"
I20250815 13:53:45.568225 2861 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:53:45.624311 2861 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:53:45.626101 2861 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:53:45.626601 2861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:53:45.629606 2861 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 13:53:45.634793 2861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 13:53:45.635025 2861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 13:53:45.635344 2861 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 13:53:45.635500 2861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 13:53:45.800467 2861 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.131:34451
I20250815 13:53:45.800583 2989 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.131:34451 every 8 connection(s)
I20250815 13:53:45.803447 2861 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250815 13:53:45.809808 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 2861
I20250815 13:53:45.810302 426 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250815 13:53:45.826738 2990 heartbeater.cc:344] Connected to a master server at 127.0.106.190:33107
I20250815 13:53:45.827206 2990 heartbeater.cc:461] Registering TS with master...
I20250815 13:53:45.828351 2990 heartbeater.cc:507] Master 127.0.106.190:33107 requested a full tablet report, sending...
I20250815 13:53:45.830964 2535 ts_manager.cc:194] Registered new tserver with Master: dc83690431414ca39ec025627efb6896 (127.0.106.131:34451)
I20250815 13:53:45.832561 2535 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.131:48445
I20250815 13:53:45.832796 426 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 13:53:45.872671 2535 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47078:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250815 13:53:45.892778 2535 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250815 13:53:45.947839 2792 tablet_service.cc:1468] Processing CreateTablet for tablet 3bec89c4a6434bdb98aa9659d6bfe40e (DEFAULT_TABLE table=TestTable [id=a75930e8658f42b284e29a5da1013e26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 13:53:45.949388 2792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bec89c4a6434bdb98aa9659d6bfe40e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:53:45.946260 2658 tablet_service.cc:1468] Processing CreateTablet for tablet 3bec89c4a6434bdb98aa9659d6bfe40e (DEFAULT_TABLE table=TestTable [id=a75930e8658f42b284e29a5da1013e26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 13:53:45.949234 2925 tablet_service.cc:1468] Processing CreateTablet for tablet 3bec89c4a6434bdb98aa9659d6bfe40e (DEFAULT_TABLE table=TestTable [id=a75930e8658f42b284e29a5da1013e26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 13:53:45.950863 2658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bec89c4a6434bdb98aa9659d6bfe40e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:53:45.951143 2925 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bec89c4a6434bdb98aa9659d6bfe40e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:53:45.970255 3009 tablet_bootstrap.cc:492] T 3bec89c4a6434bdb98aa9659d6bfe40e P 315b45714dc74a91b53678ba4ba149b0: Bootstrap starting.
I20250815 13:53:45.977584 3009 tablet_bootstrap.cc:654] T 3bec89c4a6434bdb98aa9659d6bfe40e P 315b45714dc74a91b53678ba4ba149b0: Neither blocks nor log segments found. Creating new log.
I20250815 13:53:45.980103 3009 log.cc:826] T 3bec89c4a6434bdb98aa9659d6bfe40e P 315b45714dc74a91b53678ba4ba149b0: Log is configured to *not* fsync() on all Append() calls
I20250815 13:53:45.980829 3011 tablet_bootstrap.cc:492] T 3bec89c4a6434bdb98aa9659d6bfe40e P dc83690431414ca39ec025627efb6896: Bootstrap starting.
I20250815 13:53:45.984268 3010 tablet_bootstrap.cc:492] T 3bec89c4a6434bdb98aa9659d6bfe40e P 49e971313af5480f9434879948ddff05: Bootstrap starting.
I20250815 13:53:45.987901 3011 tablet_bootstrap.cc:654] T 3bec89c4a6434bdb98aa9659d6bfe40e P dc83690431414ca39ec025627efb6896: Neither blocks nor log segments found. Creating new log.
I20250815 13:53:45.989642 3011 log.cc:826] T 3bec89c4a6434bdb98aa9659d6bfe40e P dc83690431414ca39ec025627efb6896: Log is configured to *not* fsync() on all Append() calls
I20250815 13:53:45.989923 3009 tablet_bootstrap.cc:492] T 3bec89c4a6434bdb98aa9659d6bfe40e P 315b45714dc74a91b53678ba4ba149b0: No bootstrap required, opened a new log
I20250815 13:53:45.990566 3009 ts_tablet_manager.cc:1397] T 3bec89c4a6434bdb98aa9659d6bfe40e P 315b45714dc74a91b53678ba4ba149b0: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.007s
I20250815 13:53:45.992239 3010 tablet_bootstrap.cc:654] T 3bec89c4a6434bdb98aa9659d6bfe40e P 49e971313af5480f9434879948ddff05: Neither blocks nor log segments found. Creating new log.
I20250815 13:53:45.994434 3011 tablet_bootstrap.cc:492] T 3bec89c4a6434bdb98aa9659d6bfe40e P dc83690431414ca39ec025627efb6896: No bootstrap required, opened a new log
I20250815 13:53:45.994724 3010 log.cc:826] T 3bec89c4a6434bdb98aa9659d6bfe40e P 49e971313af5480f9434879948ddff05: Log is configured to *not* fsync() on all Append() calls
I20250815 13:53:45.994875 3011 ts_tablet_manager.cc:1397] T 3bec89c4a6434bdb98aa9659d6bfe40e P dc83690431414ca39ec025627efb6896: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.009s
I20250815 13:53:46.004357 3010 tablet_bootstrap.cc:492] T 3bec89c4a6434bdb98aa9659d6bfe40e P 49e971313af5480f9434879948ddff05: No bootstrap required, opened a new log
I20250815 13:53:46.004966 3010 ts_tablet_manager.cc:1397] T 3bec89c4a6434bdb98aa9659d6bfe40e P 49e971313af5480f9434879948ddff05: Time spent bootstrapping tablet: real 0.022s user 0.019s sys 0.000s
I20250815 13:53:46.013851 3011 raft_consensus.cc:357] T 3bec89c4a6434bdb98aa9659d6bfe40e P dc83690431414ca39ec025627efb6896 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e971313af5480f9434879948ddff05" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 43101 } } peers { permanent_uuid: "315b45714dc74a91b53678ba4ba149b0" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46303 } } peers { permanent_uuid: "dc83690431414ca39ec025627efb6896" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 34451 } }
I20250815 13:53:46.014546 3011 raft_consensus.cc:383] T 3bec89c4a6434bdb98aa9659d6bfe40e P dc83690431414ca39ec025627efb6896 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 13:53:46.014794 3011 raft_consensus.cc:738] T 3bec89c4a6434bdb98aa9659d6bfe40e P dc83690431414ca39ec025627efb6896 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc83690431414ca39ec025627efb6896, State: Initialized, Role: FOLLOWER
I20250815 13:53:46.015671 3011 consensus_queue.cc:260] T 3bec89c4a6434bdb98aa9659d6bfe40e P dc83690431414ca39ec025627efb6896 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e971313af5480f9434879948ddff05" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 43101 } } peers { permanent_uuid: "315b45714dc74a91b53678ba4ba149b0" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46303 } } peers { permanent_uuid: "dc83690431414ca39ec025627efb6896" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 34451 } }
I20250815 13:53:46.017926 3009 raft_consensus.cc:357] T 3bec89c4a6434bdb98aa9659d6bfe40e P 315b45714dc74a91b53678ba4ba149b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e971313af5480f9434879948ddff05" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 43101 } } peers { permanent_uuid: "315b45714dc74a91b53678ba4ba149b0" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46303 } } peers { permanent_uuid: "dc83690431414ca39ec025627efb6896" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 34451 } }
I20250815 13:53:46.018939 3009 raft_consensus.cc:383] T 3bec89c4a6434bdb98aa9659d6bfe40e P 315b45714dc74a91b53678ba4ba149b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 13:53:46.019295 3009 raft_consensus.cc:738] T 3bec89c4a6434bdb98aa9659d6bfe40e P 315b45714dc74a91b53678ba4ba149b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315b45714dc74a91b53678ba4ba149b0, State: Initialized, Role: FOLLOWER
I20250815 13:53:46.019415 2990 heartbeater.cc:499] Master 127.0.106.190:33107 was elected leader, sending a full tablet report...
I20250815 13:53:46.020254 3009 consensus_queue.cc:260] T 3bec89c4a6434bdb98aa9659d6bfe40e P 315b45714dc74a91b53678ba4ba149b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e971313af5480f9434879948ddff05" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 43101 } } peers { permanent_uuid: "315b45714dc74a91b53678ba4ba149b0" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46303 } } peers { permanent_uuid: "dc83690431414ca39ec025627efb6896" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 34451 } }
I20250815 13:53:46.020809 3011 ts_tablet_manager.cc:1428] T 3bec89c4a6434bdb98aa9659d6bfe40e P dc83690431414ca39ec025627efb6896: Time spent starting tablet: real 0.026s user 0.020s sys 0.006s
I20250815 13:53:46.024847 3009 ts_tablet_manager.cc:1428] T 3bec89c4a6434bdb98aa9659d6bfe40e P 315b45714dc74a91b53678ba4ba149b0: Time spent starting tablet: real 0.034s user 0.029s sys 0.007s
I20250815 13:53:46.031999 3010 raft_consensus.cc:357] T 3bec89c4a6434bdb98aa9659d6bfe40e P 49e971313af5480f9434879948ddff05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e971313af5480f9434879948ddff05" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 43101 } } peers { permanent_uuid: "315b45714dc74a91b53678ba4ba149b0" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46303 } } peers { permanent_uuid: "dc83690431414ca39ec025627efb6896" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 34451 } }
I20250815 13:53:46.033044 3010 raft_consensus.cc:383] T 3bec89c4a6434bdb98aa9659d6bfe40e P 49e971313af5480f9434879948ddff05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 13:53:46.033314 3010 raft_consensus.cc:738] T 3bec89c4a6434bdb98aa9659d6bfe40e P 49e971313af5480f9434879948ddff05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e971313af5480f9434879948ddff05, State: Initialized, Role: FOLLOWER
I20250815 13:53:46.034145 3010 consensus_queue.cc:260] T 3bec89c4a6434bdb98aa9659d6bfe40e P 49e971313af5480f9434879948ddff05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e971313af5480f9434879948ddff05" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 43101 } } peers { permanent_uuid: "315b45714dc74a91b53678ba4ba149b0" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46303 } } peers { permanent_uuid: "dc83690431414ca39ec025627efb6896" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 34451 } }
I20250815 13:53:46.038007 3010 ts_tablet_manager.cc:1428] T 3bec89c4a6434bdb98aa9659d6bfe40e P 49e971313af5480f9434879948ddff05: Time spent starting tablet: real 0.033s user 0.022s sys 0.011s
W20250815 13:53:46.059011 2991 tablet.cc:2378] T 3bec89c4a6434bdb98aa9659d6bfe40e P dc83690431414ca39ec025627efb6896: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250815 13:53:46.093719 2858 tablet.cc:2378] T 3bec89c4a6434bdb98aa9659d6bfe40e P 315b45714dc74a91b53678ba4ba149b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250815 13:53:46.250005 2724 tablet.cc:2378] T 3bec89c4a6434bdb98aa9659d6bfe40e P 49e971313af5480f9434879948ddff05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 13:53:46.324640 3015 raft_consensus.cc:491] T 3bec89c4a6434bdb98aa9659d6bfe40e P dc83690431414ca39ec025627efb6896 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 13:53:46.325131 3015 raft_consensus.cc:513] T 3bec89c4a6434bdb98aa9659d6bfe40e P dc83690431414ca39ec025627efb6896 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e971313af5480f9434879948ddff05" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 43101 } } peers { permanent_uuid: "315b45714dc74a91b53678ba4ba149b0" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46303 } } peers { permanent_uuid: "dc83690431414ca39ec025627efb6896" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 34451 } }
I20250815 13:53:46.327827 3015 leader_election.cc:290] T 3bec89c4a6434bdb98aa9659d6bfe40e P dc83690431414ca39ec025627efb6896 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49e971313af5480f9434879948ddff05 (127.0.106.129:43101), 315b45714dc74a91b53678ba4ba149b0 (127.0.106.130:46303)
I20250815 13:53:46.340066 2678 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3bec89c4a6434bdb98aa9659d6bfe40e" candidate_uuid: "dc83690431414ca39ec025627efb6896" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e971313af5480f9434879948ddff05" is_pre_election: true
I20250815 13:53:46.340992 2678 raft_consensus.cc:2466] T 3bec89c4a6434bdb98aa9659d6bfe40e P 49e971313af5480f9434879948ddff05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc83690431414ca39ec025627efb6896 in term 0.
I20250815 13:53:46.342530 2812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3bec89c4a6434bdb98aa9659d6bfe40e" candidate_uuid: "dc83690431414ca39ec025627efb6896" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315b45714dc74a91b53678ba4ba149b0" is_pre_election: true
I20250815 13:53:46.343367 2880 leader_election.cc:304] T 3bec89c4a6434bdb98aa9659d6bfe40e P dc83690431414ca39ec025627efb6896 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e971313af5480f9434879948ddff05, dc83690431414ca39ec025627efb6896; no voters:
I20250815 13:53:46.343559 2812 raft_consensus.cc:2466] T 3bec89c4a6434bdb98aa9659d6bfe40e P 315b45714dc74a91b53678ba4ba149b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc83690431414ca39ec025627efb6896 in term 0.
I20250815 13:53:46.344441 3015 raft_consensus.cc:2802] T 3bec89c4a6434bdb98aa9659d6bfe40e P dc83690431414ca39ec025627efb6896 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250815 13:53:46.344924 3015 raft_consensus.cc:491] T 3bec89c4a6434bdb98aa9659d6bfe40e P dc83690431414ca39ec025627efb6896 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 13:53:46.345300 3015 raft_consensus.cc:3058] T 3bec89c4a6434bdb98aa9659d6bfe40e P dc83690431414ca39ec025627efb6896 [term 0 FOLLOWER]: Advancing to term 1
I20250815 13:53:46.357908 3015 raft_consensus.cc:513] T 3bec89c4a6434bdb98aa9659d6bfe40e P dc83690431414ca39ec025627efb6896 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e971313af5480f9434879948ddff05" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 43101 } } peers { permanent_uuid: "315b45714dc74a91b53678ba4ba149b0" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46303 } } peers { permanent_uuid: "dc83690431414ca39ec025627efb6896" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 34451 } }
I20250815 13:53:46.359555 3015 leader_election.cc:290] T 3bec89c4a6434bdb98aa9659d6bfe40e P dc83690431414ca39ec025627efb6896 [CANDIDATE]: Term 1 election: Requested vote from peers 49e971313af5480f9434879948ddff05 (127.0.106.129:43101), 315b45714dc74a91b53678ba4ba149b0 (127.0.106.130:46303)
I20250815 13:53:46.360580 2678 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3bec89c4a6434bdb98aa9659d6bfe40e" candidate_uuid: "dc83690431414ca39ec025627efb6896" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e971313af5480f9434879948ddff05"
I20250815 13:53:46.361119 2678 raft_consensus.cc:3058] T 3bec89c4a6434bdb98aa9659d6bfe40e P 49e971313af5480f9434879948ddff05 [term 0 FOLLOWER]: Advancing to term 1
I20250815 13:53:46.362414 2812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3bec89c4a6434bdb98aa9659d6bfe40e" candidate_uuid: "dc83690431414ca39ec025627efb6896" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315b45714dc74a91b53678ba4ba149b0"
I20250815 13:53:46.363104 2812 raft_consensus.cc:3058] T 3bec89c4a6434bdb98aa9659d6bfe40e P 315b45714dc74a91b53678ba4ba149b0 [term 0 FOLLOWER]: Advancing to term 1
I20250815 13:53:46.367527 2678 raft_consensus.cc:2466] T 3bec89c4a6434bdb98aa9659d6bfe40e P 49e971313af5480f9434879948ddff05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc83690431414ca39ec025627efb6896 in term 1.
I20250815 13:53:46.368721 2880 leader_election.cc:304] T 3bec89c4a6434bdb98aa9659d6bfe40e P dc83690431414ca39ec025627efb6896 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e971313af5480f9434879948ddff05, dc83690431414ca39ec025627efb6896; no voters:
I20250815 13:53:46.369261 2812 raft_consensus.cc:2466] T 3bec89c4a6434bdb98aa9659d6bfe40e P 315b45714dc74a91b53678ba4ba149b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc83690431414ca39ec025627efb6896 in term 1.
I20250815 13:53:46.370509 3015 raft_consensus.cc:2802] T 3bec89c4a6434bdb98aa9659d6bfe40e P dc83690431414ca39ec025627efb6896 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 13:53:46.372738 3015 raft_consensus.cc:695] T 3bec89c4a6434bdb98aa9659d6bfe40e P dc83690431414ca39ec025627efb6896 [term 1 LEADER]: Becoming Leader. State: Replica: dc83690431414ca39ec025627efb6896, State: Running, Role: LEADER
I20250815 13:53:46.373962 3015 consensus_queue.cc:237] T 3bec89c4a6434bdb98aa9659d6bfe40e P dc83690431414ca39ec025627efb6896 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e971313af5480f9434879948ddff05" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 43101 } } peers { permanent_uuid: "315b45714dc74a91b53678ba4ba149b0" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46303 } } peers { permanent_uuid: "dc83690431414ca39ec025627efb6896" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 34451 } }
I20250815 13:53:46.401878 2535 catalog_manager.cc:5582] T 3bec89c4a6434bdb98aa9659d6bfe40e P dc83690431414ca39ec025627efb6896 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc83690431414ca39ec025627efb6896 (127.0.106.131). New cstate: current_term: 1 leader_uuid: "dc83690431414ca39ec025627efb6896" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e971313af5480f9434879948ddff05" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 43101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315b45714dc74a91b53678ba4ba149b0" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc83690431414ca39ec025627efb6896" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 34451 } health_report { overall_health: HEALTHY } } }
I20250815 13:53:46.523483 426 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 13:53:46.527827 426 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 49e971313af5480f9434879948ddff05 to finish bootstrapping
I20250815 13:53:46.541985 426 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 315b45714dc74a91b53678ba4ba149b0 to finish bootstrapping
I20250815 13:53:46.554759 426 ts_itest-base.cc:246] Waiting for 1 tablets on tserver dc83690431414ca39ec025627efb6896 to finish bootstrapping
I20250815 13:53:46.886247 3015 consensus_queue.cc:1035] T 3bec89c4a6434bdb98aa9659d6bfe40e P dc83690431414ca39ec025627efb6896 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e971313af5480f9434879948ddff05" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 43101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250815 13:53:46.907528 3031 consensus_queue.cc:1035] T 3bec89c4a6434bdb98aa9659d6bfe40e P dc83690431414ca39ec025627efb6896 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315b45714dc74a91b53678ba4ba149b0" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250815 13:53:47.376112 2566 debug-util.cc:398] Leaking SignalData structure 0x7b08000a70e0 after lost signal to thread 2503
W20250815 13:53:47.376931 2566 debug-util.cc:398] Leaking SignalData structure 0x7b080006f1a0 after lost signal to thread 2569
W20250815 13:53:48.233928 2534 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.RefreshAuthzCache from {username='slave'} at 127.0.0.1:47096
I20250815 13:53:49.277976 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 2594
I20250815 13:53:49.303642 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 2727
I20250815 13:53:49.331385 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 2861
I20250815 13:53:49.358614 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 2502
2025-08-15T13:53:49Z chronyd exiting
[ OK ] AdminCliTest.TestAuthzResetCacheNotAuthorized (11859 ms)
[ RUN ] AdminCliTest.TestRebuildTables
I20250815 13:53:49.413793 426 test_util.cc:276] Using random seed: -1205115522
I20250815 13:53:49.418427 426 ts_itest-base.cc:115] Starting cluster with:
I20250815 13:53:49.418613 426 ts_itest-base.cc:116] --------------
I20250815 13:53:49.418772 426 ts_itest-base.cc:117] 3 tablet servers
I20250815 13:53:49.418913 426 ts_itest-base.cc:118] 3 replicas per TS
I20250815 13:53:49.419181 426 ts_itest-base.cc:119] --------------
2025-08-15T13:53:49Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-15T13:53:49Z Disabled control of system clock
I20250815 13:53:49.461464 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.190:35951
--webserver_interface=127.0.106.190
--webserver_port=0
--builtin_ntp_servers=127.0.106.148:39749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.106.190:35951 with env {}
W20250815 13:53:49.762617 3060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:53:49.763340 3060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:53:49.763799 3060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:53:49.794729 3060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 13:53:49.795071 3060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:53:49.795362 3060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 13:53:49.795598 3060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 13:53:49.830773 3060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:39749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.106.190:35951
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.190:35951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.106.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:53:49.832151 3060 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:53:49.833729 3060 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:53:49.844738 3066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:53:49.845512 3067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:53:50.961297 3069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:53:50.964645 3060 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.120s user 0.386s sys 0.733s
W20250815 13:53:50.964993 3068 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1113 milliseconds
W20250815 13:53:50.965068 3060 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.121s user 0.386s sys 0.733s
I20250815 13:53:50.965477 3060 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 13:53:50.966882 3060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:53:50.969899 3060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:53:50.971295 3060 hybrid_clock.cc:648] HybridClock initialized: now 1755266030971239 us; error 68 us; skew 500 ppm
I20250815 13:53:50.972242 3060 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:53:50.980777 3060 webserver.cc:480] Webserver started at http://127.0.106.190:36725/ using document root <none> and password file <none>
I20250815 13:53:50.981824 3060 fs_manager.cc:362] Metadata directory not provided
I20250815 13:53:50.982070 3060 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:53:50.982543 3060 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 13:53:50.987531 3060 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "993462eb9d054343874cf1ca224fbc13"
format_stamp: "Formatted at 2025-08-15 13:53:50 on dist-test-slave-kt02"
I20250815 13:53:50.988801 3060 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "993462eb9d054343874cf1ca224fbc13"
format_stamp: "Formatted at 2025-08-15 13:53:50 on dist-test-slave-kt02"
I20250815 13:53:50.997861 3060 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.000s sys 0.009s
I20250815 13:53:51.004810 3076 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:53:51.006155 3060 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.000s sys 0.005s
I20250815 13:53:51.006526 3060 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
uuid: "993462eb9d054343874cf1ca224fbc13"
format_stamp: "Formatted at 2025-08-15 13:53:50 on dist-test-slave-kt02"
I20250815 13:53:51.006925 3060 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:53:51.105334 3060 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:53:51.107103 3060 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:53:51.107604 3060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:53:51.187469 3060 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.190:35951
I20250815 13:53:51.187564 3127 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.190:35951 every 8 connection(s)
I20250815 13:53:51.190572 3060 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250815 13:53:51.192052 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 3060
I20250815 13:53:51.192605 426 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250815 13:53:51.197455 3128 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:53:51.224040 3128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13: Bootstrap starting.
I20250815 13:53:51.230600 3128 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13: Neither blocks nor log segments found. Creating new log.
I20250815 13:53:51.232609 3128 log.cc:826] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13: Log is configured to *not* fsync() on all Append() calls
I20250815 13:53:51.237962 3128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13: No bootstrap required, opened a new log
I20250815 13:53:51.257112 3128 raft_consensus.cc:357] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993462eb9d054343874cf1ca224fbc13" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 35951 } }
I20250815 13:53:51.258060 3128 raft_consensus.cc:383] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 13:53:51.258373 3128 raft_consensus.cc:738] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993462eb9d054343874cf1ca224fbc13, State: Initialized, Role: FOLLOWER
I20250815 13:53:51.259277 3128 consensus_queue.cc:260] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993462eb9d054343874cf1ca224fbc13" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 35951 } }
I20250815 13:53:51.260016 3128 raft_consensus.cc:397] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 13:53:51.260354 3128 raft_consensus.cc:491] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 13:53:51.260660 3128 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [term 0 FOLLOWER]: Advancing to term 1
I20250815 13:53:51.265237 3128 raft_consensus.cc:513] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993462eb9d054343874cf1ca224fbc13" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 35951 } }
I20250815 13:53:51.266088 3128 leader_election.cc:304] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 993462eb9d054343874cf1ca224fbc13; no voters:
I20250815 13:53:51.268059 3128 leader_election.cc:290] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 13:53:51.268937 3133 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 13:53:51.272186 3133 raft_consensus.cc:695] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [term 1 LEADER]: Becoming Leader. State: Replica: 993462eb9d054343874cf1ca224fbc13, State: Running, Role: LEADER
I20250815 13:53:51.273115 3133 consensus_queue.cc:237] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993462eb9d054343874cf1ca224fbc13" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 35951 } }
I20250815 13:53:51.274132 3128 sys_catalog.cc:564] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [sys.catalog]: configured and running, proceeding with master startup.
I20250815 13:53:51.281260 3135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 993462eb9d054343874cf1ca224fbc13. Latest consensus state: current_term: 1 leader_uuid: "993462eb9d054343874cf1ca224fbc13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993462eb9d054343874cf1ca224fbc13" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 35951 } } }
I20250815 13:53:51.281601 3134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "993462eb9d054343874cf1ca224fbc13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993462eb9d054343874cf1ca224fbc13" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 35951 } } }
I20250815 13:53:51.282279 3135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [sys.catalog]: This master's current role is: LEADER
I20250815 13:53:51.282415 3134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [sys.catalog]: This master's current role is: LEADER
I20250815 13:53:51.288344 3139 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 13:53:51.299853 3139 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 13:53:51.315925 3139 catalog_manager.cc:1349] Generated new cluster ID: 115e3d3900b64ba7a9bb5cbf73264c90
I20250815 13:53:51.316264 3139 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 13:53:51.335156 3139 catalog_manager.cc:1372] Generated new certificate authority record
I20250815 13:53:51.336665 3139 catalog_manager.cc:1506] Loading token signing keys...
I20250815 13:53:51.356163 3139 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13: Generated new TSK 0
I20250815 13:53:51.357084 3139 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250815 13:53:51.381995 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.129:0
--local_ip_for_outbound_sockets=127.0.106.129
--webserver_interface=127.0.106.129
--webserver_port=0
--tserver_master_addrs=127.0.106.190:35951
--builtin_ntp_servers=127.0.106.148:39749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250815 13:53:51.677754 3152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:53:51.678261 3152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:53:51.678756 3152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:53:51.710693 3152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:53:51.711608 3152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.129
I20250815 13:53:51.747655 3152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:39749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.106.129
--webserver_port=0
--tserver_master_addrs=127.0.106.190:35951
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.129
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:53:51.748937 3152 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:53:51.750459 3152 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:53:51.764185 3158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:53:53.165943 3157 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 3152
W20250815 13:53:53.308933 3152 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.545s user 0.000s sys 0.002s
W20250815 13:53:51.764830 3159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:53:53.309309 3152 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.546s user 0.000s sys 0.002s
W20250815 13:53:53.311657 3161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:53:53.316107 3160 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1550 milliseconds
I20250815 13:53:53.316152 3152 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 13:53:53.317569 3152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:53:53.320084 3152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:53:53.321524 3152 hybrid_clock.cc:648] HybridClock initialized: now 1755266033321468 us; error 75 us; skew 500 ppm
I20250815 13:53:53.322351 3152 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:53:53.329064 3152 webserver.cc:480] Webserver started at http://127.0.106.129:44961/ using document root <none> and password file <none>
I20250815 13:53:53.329991 3152 fs_manager.cc:362] Metadata directory not provided
I20250815 13:53:53.330207 3152 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:53:53.330631 3152 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 13:53:53.334897 3152 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "f86092b7fc7a46dfb843a939dd4fbc6a"
format_stamp: "Formatted at 2025-08-15 13:53:53 on dist-test-slave-kt02"
I20250815 13:53:53.336016 3152 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "f86092b7fc7a46dfb843a939dd4fbc6a"
format_stamp: "Formatted at 2025-08-15 13:53:53 on dist-test-slave-kt02"
I20250815 13:53:53.343304 3152 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250815 13:53:53.356566 3169 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:53:53.357872 3152 fs_manager.cc:730] Time spent opening block manager: real 0.010s user 0.005s sys 0.003s
I20250815 13:53:53.358244 3152 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f86092b7fc7a46dfb843a939dd4fbc6a"
format_stamp: "Formatted at 2025-08-15 13:53:53 on dist-test-slave-kt02"
I20250815 13:53:53.358577 3152 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:53:53.416388 3152 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:53:53.417793 3152 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:53:53.418243 3152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:53:53.420936 3152 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 13:53:53.425397 3152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 13:53:53.425616 3152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 13:53:53.425860 3152 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 13:53:53.426021 3152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 13:53:53.587270 3152 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.129:33117
I20250815 13:53:53.587515 3281 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.129:33117 every 8 connection(s)
I20250815 13:53:53.589782 3152 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250815 13:53:53.596598 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 3152
I20250815 13:53:53.597302 426 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250815 13:53:53.605958 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.130:0
--local_ip_for_outbound_sockets=127.0.106.130
--webserver_interface=127.0.106.130
--webserver_port=0
--tserver_master_addrs=127.0.106.190:35951
--builtin_ntp_servers=127.0.106.148:39749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 13:53:53.619719 3282 heartbeater.cc:344] Connected to a master server at 127.0.106.190:35951
I20250815 13:53:53.620290 3282 heartbeater.cc:461] Registering TS with master...
I20250815 13:53:53.621614 3282 heartbeater.cc:507] Master 127.0.106.190:35951 requested a full tablet report, sending...
I20250815 13:53:53.625056 3093 ts_manager.cc:194] Registered new tserver with Master: f86092b7fc7a46dfb843a939dd4fbc6a (127.0.106.129:33117)
I20250815 13:53:53.628177 3093 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.129:35993
W20250815 13:53:53.924235 3286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:53:53.924883 3286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:53:53.925629 3286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:53:53.958182 3286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:53:53.958995 3286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.130
I20250815 13:53:53.993810 3286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:39749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.106.130
--webserver_port=0
--tserver_master_addrs=127.0.106.190:35951
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.130
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:53:53.995075 3286 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:53:53.996538 3286 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:53:54.008709 3292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 13:53:54.633001 3282 heartbeater.cc:499] Master 127.0.106.190:35951 was elected leader, sending a full tablet report...
W20250815 13:53:54.009801 3293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:53:54.013904 3295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:53:55.108670 3294 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250815 13:53:55.108801 3286 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 13:53:55.113807 3286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:53:55.116896 3286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:53:55.118433 3286 hybrid_clock.cc:648] HybridClock initialized: now 1755266035118350 us; error 92 us; skew 500 ppm
I20250815 13:53:55.119434 3286 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:53:55.126978 3286 webserver.cc:480] Webserver started at http://127.0.106.130:34781/ using document root <none> and password file <none>
I20250815 13:53:55.128082 3286 fs_manager.cc:362] Metadata directory not provided
I20250815 13:53:55.128299 3286 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:53:55.128819 3286 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 13:53:55.133811 3286 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "90a13caa51354a19b2434ba2e76cf1f3"
format_stamp: "Formatted at 2025-08-15 13:53:55 on dist-test-slave-kt02"
I20250815 13:53:55.135088 3286 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "90a13caa51354a19b2434ba2e76cf1f3"
format_stamp: "Formatted at 2025-08-15 13:53:55 on dist-test-slave-kt02"
I20250815 13:53:55.143155 3286 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250815 13:53:55.149999 3302 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:53:55.151453 3286 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.007s sys 0.000s
I20250815 13:53:55.151881 3286 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "90a13caa51354a19b2434ba2e76cf1f3"
format_stamp: "Formatted at 2025-08-15 13:53:55 on dist-test-slave-kt02"
I20250815 13:53:55.152268 3286 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:53:55.200824 3286 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:53:55.202525 3286 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:53:55.203007 3286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:53:55.205964 3286 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 13:53:55.210757 3286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 13:53:55.210969 3286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 13:53:55.211246 3286 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 13:53:55.211405 3286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 13:53:55.367727 3286 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.130:46037
I20250815 13:53:55.367839 3414 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.130:46037 every 8 connection(s)
I20250815 13:53:55.370704 3286 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250815 13:53:55.380839 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 3286
I20250815 13:53:55.381261 426 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250815 13:53:55.388007 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.131:0
--local_ip_for_outbound_sockets=127.0.106.131
--webserver_interface=127.0.106.131
--webserver_port=0
--tserver_master_addrs=127.0.106.190:35951
--builtin_ntp_servers=127.0.106.148:39749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 13:53:55.395038 3415 heartbeater.cc:344] Connected to a master server at 127.0.106.190:35951
I20250815 13:53:55.395699 3415 heartbeater.cc:461] Registering TS with master...
I20250815 13:53:55.396916 3415 heartbeater.cc:507] Master 127.0.106.190:35951 requested a full tablet report, sending...
I20250815 13:53:55.399611 3093 ts_manager.cc:194] Registered new tserver with Master: 90a13caa51354a19b2434ba2e76cf1f3 (127.0.106.130:46037)
I20250815 13:53:55.401036 3093 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.130:50621
W20250815 13:53:55.710292 3419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:53:55.710769 3419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:53:55.711344 3419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:53:55.742612 3419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:53:55.743525 3419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.131
I20250815 13:53:55.778757 3419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:39749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.106.131
--webserver_port=0
--tserver_master_addrs=127.0.106.190:35951
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.131
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:53:55.780133 3419 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:53:55.781724 3419 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:53:55.794142 3425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 13:53:56.405257 3415 heartbeater.cc:499] Master 127.0.106.190:35951 was elected leader, sending a full tablet report...
W20250815 13:53:55.794643 3426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:53:57.108525 3428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:53:57.112876 3427 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1318 milliseconds
W20250815 13:53:57.113157 3419 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.319s user 0.530s sys 0.767s
W20250815 13:53:57.113617 3419 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.320s user 0.530s sys 0.767s
I20250815 13:53:57.113905 3419 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 13:53:57.115428 3419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:53:57.118366 3419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:53:57.119859 3419 hybrid_clock.cc:648] HybridClock initialized: now 1755266037119820 us; error 43 us; skew 500 ppm
I20250815 13:53:57.120956 3419 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:53:57.129881 3419 webserver.cc:480] Webserver started at http://127.0.106.131:36245/ using document root <none> and password file <none>
I20250815 13:53:57.131292 3419 fs_manager.cc:362] Metadata directory not provided
I20250815 13:53:57.131587 3419 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:53:57.132192 3419 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 13:53:57.138774 3419 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "34201ba276c14ae6bc93bb059132d83f"
format_stamp: "Formatted at 2025-08-15 13:53:57 on dist-test-slave-kt02"
I20250815 13:53:57.140344 3419 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "34201ba276c14ae6bc93bb059132d83f"
format_stamp: "Formatted at 2025-08-15 13:53:57 on dist-test-slave-kt02"
I20250815 13:53:57.150301 3419 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.006s sys 0.003s
I20250815 13:53:57.158020 3435 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:53:57.159482 3419 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.002s
I20250815 13:53:57.159929 3419 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "34201ba276c14ae6bc93bb059132d83f"
format_stamp: "Formatted at 2025-08-15 13:53:57 on dist-test-slave-kt02"
I20250815 13:53:57.160380 3419 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:53:57.247597 3419 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:53:57.249117 3419 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:53:57.249549 3419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:53:57.252131 3419 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 13:53:57.256670 3419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 13:53:57.256894 3419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 13:53:57.257140 3419 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 13:53:57.257313 3419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 13:53:57.400210 3419 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.131:36587
I20250815 13:53:57.400367 3547 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.131:36587 every 8 connection(s)
I20250815 13:53:57.402750 3419 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250815 13:53:57.412804 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 3419
I20250815 13:53:57.413442 426 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250815 13:53:57.426240 3548 heartbeater.cc:344] Connected to a master server at 127.0.106.190:35951
I20250815 13:53:57.426801 3548 heartbeater.cc:461] Registering TS with master...
I20250815 13:53:57.428027 3548 heartbeater.cc:507] Master 127.0.106.190:35951 requested a full tablet report, sending...
I20250815 13:53:57.430685 3093 ts_manager.cc:194] Registered new tserver with Master: 34201ba276c14ae6bc93bb059132d83f (127.0.106.131:36587)
I20250815 13:53:57.432367 3093 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.131:45183
I20250815 13:53:57.434855 426 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 13:53:57.473534 426 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 13:53:57.473922 426 test_util.cc:276] Using random seed: -1197055378
I20250815 13:53:57.517180 3093 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59504:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250815 13:53:57.564689 3350 tablet_service.cc:1468] Processing CreateTablet for tablet 9b39e98a33744b249184945f4451802d (DEFAULT_TABLE table=TestTable [id=eb9c39d423524369a4c25a1f1f22e7a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 13:53:57.566125 3350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b39e98a33744b249184945f4451802d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:53:57.587724 3568 tablet_bootstrap.cc:492] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3: Bootstrap starting.
I20250815 13:53:57.594295 3568 tablet_bootstrap.cc:654] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3: Neither blocks nor log segments found. Creating new log.
I20250815 13:53:57.596463 3568 log.cc:826] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3: Log is configured to *not* fsync() on all Append() calls
I20250815 13:53:57.601899 3568 tablet_bootstrap.cc:492] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3: No bootstrap required, opened a new log
I20250815 13:53:57.602375 3568 ts_tablet_manager.cc:1397] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.012s
I20250815 13:53:57.621070 3568 raft_consensus.cc:357] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } }
I20250815 13:53:57.621744 3568 raft_consensus.cc:383] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 13:53:57.621996 3568 raft_consensus.cc:738] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a13caa51354a19b2434ba2e76cf1f3, State: Initialized, Role: FOLLOWER
I20250815 13:53:57.622798 3568 consensus_queue.cc:260] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } }
I20250815 13:53:57.623454 3568 raft_consensus.cc:397] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 13:53:57.623797 3568 raft_consensus.cc:491] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 13:53:57.624145 3568 raft_consensus.cc:3058] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [term 0 FOLLOWER]: Advancing to term 1
I20250815 13:53:57.629539 3568 raft_consensus.cc:513] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } }
I20250815 13:53:57.630427 3568 leader_election.cc:304] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90a13caa51354a19b2434ba2e76cf1f3; no voters:
I20250815 13:53:57.632539 3568 leader_election.cc:290] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 13:53:57.632985 3570 raft_consensus.cc:2802] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 13:53:57.635686 3570 raft_consensus.cc:695] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [term 1 LEADER]: Becoming Leader. State: Replica: 90a13caa51354a19b2434ba2e76cf1f3, State: Running, Role: LEADER
I20250815 13:53:57.636626 3570 consensus_queue.cc:237] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } }
I20250815 13:53:57.637632 3568 ts_tablet_manager.cc:1428] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3: Time spent starting tablet: real 0.035s user 0.034s sys 0.001s
I20250815 13:53:57.651819 3092 catalog_manager.cc:5582] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90a13caa51354a19b2434ba2e76cf1f3 (127.0.106.130). New cstate: current_term: 1 leader_uuid: "90a13caa51354a19b2434ba2e76cf1f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } health_report { overall_health: HEALTHY } } }
I20250815 13:53:57.878863 426 test_util.cc:276] Using random seed: -1196650458
I20250815 13:53:57.902335 3092 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59512:
name: "TestTable1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250815 13:53:57.935026 3483 tablet_service.cc:1468] Processing CreateTablet for tablet c4cc2a5449c34c269c21d85e6cb450c0 (DEFAULT_TABLE table=TestTable1 [id=8468eb12f1614760854e54200c557e95]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 13:53:57.936793 3483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4cc2a5449c34c269c21d85e6cb450c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:53:57.957808 3589 tablet_bootstrap.cc:492] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f: Bootstrap starting.
I20250815 13:53:57.963593 3589 tablet_bootstrap.cc:654] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f: Neither blocks nor log segments found. Creating new log.
I20250815 13:53:57.965373 3589 log.cc:826] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f: Log is configured to *not* fsync() on all Append() calls
I20250815 13:53:57.970223 3589 tablet_bootstrap.cc:492] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f: No bootstrap required, opened a new log
I20250815 13:53:57.970649 3589 ts_tablet_manager.cc:1397] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20250815 13:53:57.987915 3589 raft_consensus.cc:357] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } }
I20250815 13:53:57.988483 3589 raft_consensus.cc:383] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 13:53:57.988720 3589 raft_consensus.cc:738] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34201ba276c14ae6bc93bb059132d83f, State: Initialized, Role: FOLLOWER
I20250815 13:53:57.989409 3589 consensus_queue.cc:260] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } }
I20250815 13:53:57.989907 3589 raft_consensus.cc:397] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 13:53:57.990172 3589 raft_consensus.cc:491] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 13:53:57.990490 3589 raft_consensus.cc:3058] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [term 0 FOLLOWER]: Advancing to term 1
I20250815 13:53:57.994836 3589 raft_consensus.cc:513] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } }
I20250815 13:53:57.995607 3589 leader_election.cc:304] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34201ba276c14ae6bc93bb059132d83f; no voters:
I20250815 13:53:57.997466 3589 leader_election.cc:290] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 13:53:57.998152 3591 raft_consensus.cc:2802] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [term 1 FOLLOWER]: Leader election won for term 1
I20250815 13:53:58.001152 3548 heartbeater.cc:499] Master 127.0.106.190:35951 was elected leader, sending a full tablet report...
I20250815 13:53:58.000705 3591 raft_consensus.cc:695] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [term 1 LEADER]: Becoming Leader. State: Replica: 34201ba276c14ae6bc93bb059132d83f, State: Running, Role: LEADER
I20250815 13:53:58.001773 3589 ts_tablet_manager.cc:1428] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f: Time spent starting tablet: real 0.031s user 0.030s sys 0.002s
I20250815 13:53:58.002485 3591 consensus_queue.cc:237] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } }
I20250815 13:53:58.012686 3092 catalog_manager.cc:5582] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f reported cstate change: term changed from 0 to 1, leader changed from <none> to 34201ba276c14ae6bc93bb059132d83f (127.0.106.131). New cstate: current_term: 1 leader_uuid: "34201ba276c14ae6bc93bb059132d83f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } health_report { overall_health: HEALTHY } } }
I20250815 13:53:58.208786 426 test_util.cc:276] Using random seed: -1196320526
I20250815 13:53:58.234789 3092 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59516:
name: "TestTable2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250815 13:53:58.269395 3217 tablet_service.cc:1468] Processing CreateTablet for tablet 672f8196d0cb413cb7c514171eb967fd (DEFAULT_TABLE table=TestTable2 [id=dfe6d186a7dd4b66afc85b372467b78d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 13:53:58.271205 3217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 672f8196d0cb413cb7c514171eb967fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:53:58.292725 3610 tablet_bootstrap.cc:492] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a: Bootstrap starting.
I20250815 13:53:58.298095 3610 tablet_bootstrap.cc:654] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a: Neither blocks nor log segments found. Creating new log.
I20250815 13:53:58.299791 3610 log.cc:826] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a: Log is configured to *not* fsync() on all Append() calls
I20250815 13:53:58.304774 3610 tablet_bootstrap.cc:492] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a: No bootstrap required, opened a new log
I20250815 13:53:58.305177 3610 ts_tablet_manager.cc:1397] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20250815 13:53:58.322638 3610 raft_consensus.cc:357] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } }
I20250815 13:53:58.323279 3610 raft_consensus.cc:383] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 13:53:58.323504 3610 raft_consensus.cc:738] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f86092b7fc7a46dfb843a939dd4fbc6a, State: Initialized, Role: FOLLOWER
I20250815 13:53:58.324158 3610 consensus_queue.cc:260] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } }
I20250815 13:53:58.324672 3610 raft_consensus.cc:397] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 13:53:58.324934 3610 raft_consensus.cc:491] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 13:53:58.325258 3610 raft_consensus.cc:3058] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a [term 0 FOLLOWER]: Advancing to term 1
I20250815 13:53:58.329645 3610 raft_consensus.cc:513] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } }
I20250815 13:53:58.330355 3610 leader_election.cc:304] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f86092b7fc7a46dfb843a939dd4fbc6a; no voters:
I20250815 13:53:58.332367 3610 leader_election.cc:290] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 13:53:58.332870 3612 raft_consensus.cc:2802] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a [term 1 FOLLOWER]: Leader election won for term 1
I20250815 13:53:58.335491 3612 raft_consensus.cc:695] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a [term 1 LEADER]: Becoming Leader. State: Replica: f86092b7fc7a46dfb843a939dd4fbc6a, State: Running, Role: LEADER
I20250815 13:53:58.335963 3610 ts_tablet_manager.cc:1428] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a: Time spent starting tablet: real 0.031s user 0.033s sys 0.000s
I20250815 13:53:58.337004 3612 consensus_queue.cc:237] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } }
I20250815 13:53:58.348750 3092 catalog_manager.cc:5582] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a reported cstate change: term changed from 0 to 1, leader changed from <none> to f86092b7fc7a46dfb843a939dd4fbc6a (127.0.106.129). New cstate: current_term: 1 leader_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } health_report { overall_health: HEALTHY } } }
I20250815 13:53:58.529036 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 3060
W20250815 13:53:58.670681 3415 heartbeater.cc:646] Failed to heartbeat to 127.0.106.190:35951 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.106.190:35951: connect: Connection refused (error 111)
W20250815 13:53:59.031723 3548 heartbeater.cc:646] Failed to heartbeat to 127.0.106.190:35951 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.106.190:35951: connect: Connection refused (error 111)
W20250815 13:53:59.383756 3282 heartbeater.cc:646] Failed to heartbeat to 127.0.106.190:35951 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.106.190:35951: connect: Connection refused (error 111)
I20250815 13:54:03.325683 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 3152
I20250815 13:54:03.348044 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 3286
I20250815 13:54:03.375792 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 3419
I20250815 13:54:03.404448 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.190:35951
--webserver_interface=127.0.106.190
--webserver_port=36725
--builtin_ntp_servers=127.0.106.148:39749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.106.190:35951 with env {}
W20250815 13:54:03.713281 3689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:54:03.713855 3689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:54:03.714294 3689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:54:03.746296 3689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 13:54:03.746605 3689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:54:03.746814 3689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 13:54:03.747010 3689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 13:54:03.784368 3689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:39749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.106.190:35951
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.190:35951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.106.190
--webserver_port=36725
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:54:03.785681 3689 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:54:03.787320 3689 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:54:03.798837 3695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:54:05.202395 3694 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 3689
W20250815 13:54:05.373009 3689 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.574s user 0.622s sys 0.951s
W20250815 13:54:05.373376 3689 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.574s user 0.622s sys 0.951s
W20250815 13:54:03.800375 3696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:54:05.375757 3698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:54:05.379303 3697 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1573 milliseconds
I20250815 13:54:05.379381 3689 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 13:54:05.380667 3689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:54:05.383193 3689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:54:05.384604 3689 hybrid_clock.cc:648] HybridClock initialized: now 1755266045384577 us; error 48 us; skew 500 ppm
I20250815 13:54:05.385458 3689 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:54:05.391688 3689 webserver.cc:480] Webserver started at http://127.0.106.190:36725/ using document root <none> and password file <none>
I20250815 13:54:05.392604 3689 fs_manager.cc:362] Metadata directory not provided
I20250815 13:54:05.392817 3689 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:54:05.401420 3689 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.003s sys 0.002s
I20250815 13:54:05.406908 3705 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:54:05.408428 3689 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250815 13:54:05.408782 3689 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
uuid: "993462eb9d054343874cf1ca224fbc13"
format_stamp: "Formatted at 2025-08-15 13:53:50 on dist-test-slave-kt02"
I20250815 13:54:05.410894 3689 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:54:05.469285 3689 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:54:05.471081 3689 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:54:05.471549 3689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:54:05.553637 3689 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.190:35951
I20250815 13:54:05.553658 3756 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.190:35951 every 8 connection(s)
I20250815 13:54:05.556988 3689 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250815 13:54:05.563937 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 3689
I20250815 13:54:05.565492 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.129:33117
--local_ip_for_outbound_sockets=127.0.106.129
--tserver_master_addrs=127.0.106.190:35951
--webserver_port=44961
--webserver_interface=127.0.106.129
--builtin_ntp_servers=127.0.106.148:39749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 13:54:05.568068 3757 sys_catalog.cc:263] Verifying existing consensus state
I20250815 13:54:05.572984 3757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13: Bootstrap starting.
I20250815 13:54:05.585711 3757 log.cc:826] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13: Log is configured to *not* fsync() on all Append() calls
I20250815 13:54:05.647082 3757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=13 ignored=0} mutations{seen=10 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 13:54:05.648046 3757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13: Bootstrap complete.
I20250815 13:54:05.670609 3757 raft_consensus.cc:357] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993462eb9d054343874cf1ca224fbc13" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 35951 } }
I20250815 13:54:05.672951 3757 raft_consensus.cc:738] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993462eb9d054343874cf1ca224fbc13, State: Initialized, Role: FOLLOWER
I20250815 13:54:05.673907 3757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993462eb9d054343874cf1ca224fbc13" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 35951 } }
I20250815 13:54:05.674448 3757 raft_consensus.cc:397] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 13:54:05.674739 3757 raft_consensus.cc:491] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 13:54:05.675122 3757 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [term 2 FOLLOWER]: Advancing to term 3
I20250815 13:54:05.681715 3757 raft_consensus.cc:513] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993462eb9d054343874cf1ca224fbc13" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 35951 } }
I20250815 13:54:05.682471 3757 leader_election.cc:304] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 993462eb9d054343874cf1ca224fbc13; no voters:
I20250815 13:54:05.684710 3757 leader_election.cc:290] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250815 13:54:05.685143 3761 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [term 3 FOLLOWER]: Leader election won for term 3
I20250815 13:54:05.687660 3761 raft_consensus.cc:695] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [term 3 LEADER]: Becoming Leader. State: Replica: 993462eb9d054343874cf1ca224fbc13, State: Running, Role: LEADER
I20250815 13:54:05.688704 3761 consensus_queue.cc:237] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993462eb9d054343874cf1ca224fbc13" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 35951 } }
I20250815 13:54:05.691008 3757 sys_catalog.cc:564] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [sys.catalog]: configured and running, proceeding with master startup.
I20250815 13:54:05.700177 3763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "993462eb9d054343874cf1ca224fbc13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993462eb9d054343874cf1ca224fbc13" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 35951 } } }
I20250815 13:54:05.701031 3763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [sys.catalog]: This master's current role is: LEADER
I20250815 13:54:05.701972 3762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 993462eb9d054343874cf1ca224fbc13. Latest consensus state: current_term: 3 leader_uuid: "993462eb9d054343874cf1ca224fbc13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993462eb9d054343874cf1ca224fbc13" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 35951 } } }
I20250815 13:54:05.702809 3762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [sys.catalog]: This master's current role is: LEADER
I20250815 13:54:05.715495 3768 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 13:54:05.729679 3768 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=3632567ed8a74011837e795a7dbc0e18]
I20250815 13:54:05.731674 3768 catalog_manager.cc:671] Loaded metadata for table TestTable [id=bd316c14bd5c4e619db0d925b1b7dda3]
I20250815 13:54:05.733505 3768 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=dfe6d186a7dd4b66afc85b372467b78d]
I20250815 13:54:05.742372 3768 tablet_loader.cc:96] loaded metadata for tablet 672f8196d0cb413cb7c514171eb967fd (table TestTable2 [id=dfe6d186a7dd4b66afc85b372467b78d])
I20250815 13:54:05.744046 3768 tablet_loader.cc:96] loaded metadata for tablet 9b39e98a33744b249184945f4451802d (table TestTable [id=bd316c14bd5c4e619db0d925b1b7dda3])
I20250815 13:54:05.745380 3768 tablet_loader.cc:96] loaded metadata for tablet c4cc2a5449c34c269c21d85e6cb450c0 (table TestTable1 [id=3632567ed8a74011837e795a7dbc0e18])
I20250815 13:54:05.747030 3768 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 13:54:05.753089 3768 catalog_manager.cc:1261] Loaded cluster ID: 115e3d3900b64ba7a9bb5cbf73264c90
I20250815 13:54:05.753502 3768 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 13:54:05.762826 3768 catalog_manager.cc:1506] Loading token signing keys...
I20250815 13:54:05.766150 3779 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250815 13:54:05.768285 3768 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13: Loaded TSK: 0
I20250815 13:54:05.769855 3768 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250815 13:54:05.950016 3759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:54:05.950614 3759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:54:05.951174 3759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:54:05.985666 3759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:54:05.986632 3759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.129
I20250815 13:54:06.024892 3759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:39749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.129:33117
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.106.129
--webserver_port=44961
--tserver_master_addrs=127.0.106.190:35951
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.129
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:54:06.026491 3759 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:54:06.028420 3759 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:54:06.042717 3785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:54:06.048292 3788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:54:06.046206 3786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:54:07.541021 3787 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1492 milliseconds
I20250815 13:54:07.541159 3759 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 13:54:07.542502 3759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:54:07.545137 3759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:54:07.546573 3759 hybrid_clock.cc:648] HybridClock initialized: now 1755266047546513 us; error 73 us; skew 500 ppm
I20250815 13:54:07.547488 3759 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:54:07.555313 3759 webserver.cc:480] Webserver started at http://127.0.106.129:44961/ using document root <none> and password file <none>
I20250815 13:54:07.556394 3759 fs_manager.cc:362] Metadata directory not provided
I20250815 13:54:07.556668 3759 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:54:07.566560 3759 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.002s sys 0.004s
I20250815 13:54:07.572206 3795 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:54:07.573678 3759 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.000s sys 0.004s
I20250815 13:54:07.574093 3759 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f86092b7fc7a46dfb843a939dd4fbc6a"
format_stamp: "Formatted at 2025-08-15 13:53:53 on dist-test-slave-kt02"
I20250815 13:54:07.576309 3759 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:54:07.625300 3759 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:54:07.626968 3759 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:54:07.627473 3759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:54:07.630319 3759 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 13:54:07.636878 3802 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250815 13:54:07.644721 3759 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250815 13:54:07.645085 3759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.002s
I20250815 13:54:07.645426 3759 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250815 13:54:07.651135 3759 ts_tablet_manager.cc:610] Registered 1 tablets
I20250815 13:54:07.651386 3759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s user 0.005s sys 0.001s
I20250815 13:54:07.651880 3802 tablet_bootstrap.cc:492] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a: Bootstrap starting.
I20250815 13:54:07.710198 3802 log.cc:826] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a: Log is configured to *not* fsync() on all Append() calls
I20250815 13:54:07.792168 3802 tablet_bootstrap.cc:492] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 13:54:07.792954 3802 tablet_bootstrap.cc:492] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a: Bootstrap complete.
I20250815 13:54:07.794179 3802 ts_tablet_manager.cc:1397] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a: Time spent bootstrapping tablet: real 0.143s user 0.084s sys 0.052s
I20250815 13:54:07.809522 3802 raft_consensus.cc:357] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } }
I20250815 13:54:07.811602 3802 raft_consensus.cc:738] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f86092b7fc7a46dfb843a939dd4fbc6a, State: Initialized, Role: FOLLOWER
I20250815 13:54:07.812443 3802 consensus_queue.cc:260] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } }
I20250815 13:54:07.812984 3802 raft_consensus.cc:397] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 13:54:07.813236 3802 raft_consensus.cc:491] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 13:54:07.813548 3802 raft_consensus.cc:3058] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a [term 1 FOLLOWER]: Advancing to term 2
I20250815 13:54:07.819839 3802 raft_consensus.cc:513] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } }
I20250815 13:54:07.820747 3802 leader_election.cc:304] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f86092b7fc7a46dfb843a939dd4fbc6a; no voters:
I20250815 13:54:07.823791 3802 leader_election.cc:290] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a [CANDIDATE]: Term 2 election: Requested vote from peers
I20250815 13:54:07.824186 3895 raft_consensus.cc:2802] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 FOLLOWER]: Leader election won for term 2
I20250815 13:54:07.828135 3802 ts_tablet_manager.cc:1428] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a: Time spent starting tablet: real 0.034s user 0.031s sys 0.004s
I20250815 13:54:07.830552 3895 raft_consensus.cc:695] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 LEADER]: Becoming Leader. State: Replica: f86092b7fc7a46dfb843a939dd4fbc6a, State: Running, Role: LEADER
I20250815 13:54:07.831724 3895 consensus_queue.cc:237] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } }
I20250815 13:54:07.851734 3759 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.129:33117
I20250815 13:54:07.852325 3914 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.129:33117 every 8 connection(s)
I20250815 13:54:07.854563 3759 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250815 13:54:07.860708 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 3759
I20250815 13:54:07.862419 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.130:46037
--local_ip_for_outbound_sockets=127.0.106.130
--tserver_master_addrs=127.0.106.190:35951
--webserver_port=34781
--webserver_interface=127.0.106.130
--builtin_ntp_servers=127.0.106.148:39749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 13:54:07.882273 3915 heartbeater.cc:344] Connected to a master server at 127.0.106.190:35951
I20250815 13:54:07.882766 3915 heartbeater.cc:461] Registering TS with master...
I20250815 13:54:07.883833 3915 heartbeater.cc:507] Master 127.0.106.190:35951 requested a full tablet report, sending...
I20250815 13:54:07.888052 3722 ts_manager.cc:194] Registered new tserver with Master: f86092b7fc7a46dfb843a939dd4fbc6a (127.0.106.129:33117)
I20250815 13:54:07.890703 3722 catalog_manager.cc:5582] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } health_report { overall_health: HEALTHY } } }
I20250815 13:54:07.925312 3722 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.129:49283
I20250815 13:54:07.929636 3915 heartbeater.cc:499] Master 127.0.106.190:35951 was elected leader, sending a full tablet report...
W20250815 13:54:08.197026 3919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:54:08.197655 3919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:54:08.198227 3919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:54:08.232401 3919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:54:08.233389 3919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.130
I20250815 13:54:08.273839 3919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:39749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.130:46037
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.106.130
--webserver_port=34781
--tserver_master_addrs=127.0.106.190:35951
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.130
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:54:08.275543 3919 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:54:08.277401 3919 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:54:08.291098 3929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:54:08.296607 3932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:54:08.292503 3930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:54:09.389638 3931 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250815 13:54:09.389871 3919 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 13:54:09.394868 3919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:54:09.397552 3919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:54:09.398972 3919 hybrid_clock.cc:648] HybridClock initialized: now 1755266049398892 us; error 82 us; skew 500 ppm
I20250815 13:54:09.399904 3919 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:54:09.406905 3919 webserver.cc:480] Webserver started at http://127.0.106.130:34781/ using document root <none> and password file <none>
I20250815 13:54:09.408262 3919 fs_manager.cc:362] Metadata directory not provided
I20250815 13:54:09.408567 3919 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:54:09.418968 3919 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.005s sys 0.000s
I20250815 13:54:09.424528 3939 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:54:09.425762 3919 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250815 13:54:09.426124 3919 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "90a13caa51354a19b2434ba2e76cf1f3"
format_stamp: "Formatted at 2025-08-15 13:53:55 on dist-test-slave-kt02"
I20250815 13:54:09.428581 3919 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:54:09.492523 3919 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:54:09.494191 3919 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:54:09.494688 3919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:54:09.497377 3919 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 13:54:09.503899 3946 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250815 13:54:09.513237 3919 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250815 13:54:09.513550 3919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s user 0.002s sys 0.000s
I20250815 13:54:09.513840 3919 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250815 13:54:09.519295 3919 ts_tablet_manager.cc:610] Registered 1 tablets
I20250815 13:54:09.519529 3919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s user 0.006s sys 0.000s
I20250815 13:54:09.520156 3946 tablet_bootstrap.cc:492] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3: Bootstrap starting.
I20250815 13:54:09.590086 3946 log.cc:826] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3: Log is configured to *not* fsync() on all Append() calls
I20250815 13:54:09.702253 3919 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.130:46037
I20250815 13:54:09.702396 4053 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.130:46037 every 8 connection(s)
I20250815 13:54:09.705911 3919 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250815 13:54:09.716744 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 3919
I20250815 13:54:09.718775 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.131:36587
--local_ip_for_outbound_sockets=127.0.106.131
--tserver_master_addrs=127.0.106.190:35951
--webserver_port=36245
--webserver_interface=127.0.106.131
--builtin_ntp_servers=127.0.106.148:39749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 13:54:09.721267 3946 tablet_bootstrap.cc:492] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 13:54:09.722093 3946 tablet_bootstrap.cc:492] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3: Bootstrap complete.
I20250815 13:54:09.723578 3946 ts_tablet_manager.cc:1397] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3: Time spent bootstrapping tablet: real 0.204s user 0.148s sys 0.051s
I20250815 13:54:09.737768 4054 heartbeater.cc:344] Connected to a master server at 127.0.106.190:35951
I20250815 13:54:09.738402 4054 heartbeater.cc:461] Registering TS with master...
I20250815 13:54:09.740061 4054 heartbeater.cc:507] Master 127.0.106.190:35951 requested a full tablet report, sending...
I20250815 13:54:09.738936 3946 raft_consensus.cc:357] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } }
I20250815 13:54:09.741320 3946 raft_consensus.cc:738] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a13caa51354a19b2434ba2e76cf1f3, State: Initialized, Role: FOLLOWER
I20250815 13:54:09.742053 3946 consensus_queue.cc:260] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } }
I20250815 13:54:09.743160 3946 raft_consensus.cc:397] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 13:54:09.743578 3946 raft_consensus.cc:491] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 13:54:09.744056 3946 raft_consensus.cc:3058] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [term 1 FOLLOWER]: Advancing to term 2
I20250815 13:54:09.751137 3722 ts_manager.cc:194] Registered new tserver with Master: 90a13caa51354a19b2434ba2e76cf1f3 (127.0.106.130:46037)
I20250815 13:54:09.751410 3946 raft_consensus.cc:513] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } }
I20250815 13:54:09.752147 3946 leader_election.cc:304] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90a13caa51354a19b2434ba2e76cf1f3; no voters:
I20250815 13:54:09.754621 3722 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.130:41599
I20250815 13:54:09.755534 3946 leader_election.cc:290] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250815 13:54:09.759904 4054 heartbeater.cc:499] Master 127.0.106.190:35951 was elected leader, sending a full tablet report...
I20250815 13:54:09.760253 3946 ts_tablet_manager.cc:1428] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3: Time spent starting tablet: real 0.036s user 0.034s sys 0.001s
I20250815 13:54:09.759538 4059 raft_consensus.cc:2802] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [term 2 FOLLOWER]: Leader election won for term 2
I20250815 13:54:09.760872 4059 raft_consensus.cc:695] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [term 2 LEADER]: Becoming Leader. State: Replica: 90a13caa51354a19b2434ba2e76cf1f3, State: Running, Role: LEADER
I20250815 13:54:09.761502 4059 consensus_queue.cc:237] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } }
I20250815 13:54:09.768601 3722 catalog_manager.cc:5582] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 reported cstate change: term changed from 0 to 2, leader changed from <none> to 90a13caa51354a19b2434ba2e76cf1f3 (127.0.106.130), VOTER 90a13caa51354a19b2434ba2e76cf1f3 (127.0.106.130) added. New cstate: current_term: 2 leader_uuid: "90a13caa51354a19b2434ba2e76cf1f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } health_report { overall_health: HEALTHY } } }
I20250815 13:54:09.804823 4009 consensus_queue.cc:237] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: NON_VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: true } }
I20250815 13:54:09.808837 4059 raft_consensus.cc:2953] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index -1 to 10, NON_VOTER f86092b7fc7a46dfb843a939dd4fbc6a (127.0.106.129) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: NON_VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: true } } }
I20250815 13:54:09.817977 3708 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9b39e98a33744b249184945f4451802d with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250815 13:54:09.821158 3722 catalog_manager.cc:5582] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 reported cstate change: config changed from index -1 to 10, NON_VOTER f86092b7fc7a46dfb843a939dd4fbc6a (127.0.106.129) added. New cstate: current_term: 2 leader_uuid: "90a13caa51354a19b2434ba2e76cf1f3" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: NON_VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250815 13:54:09.824461 3942 consensus_peers.cc:489] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 -> Peer f86092b7fc7a46dfb843a939dd4fbc6a (127.0.106.129:33117): Couldn't send request to peer f86092b7fc7a46dfb843a939dd4fbc6a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b39e98a33744b249184945f4451802d. This is attempt 1: this message will repeat every 5th retry.
W20250815 13:54:09.830466 3722 catalog_manager.cc:5260] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9b39e98a33744b249184945f4451802d with cas_config_opid_index 10: no extra replica candidate found for tablet 9b39e98a33744b249184945f4451802d (table TestTable [id=bd316c14bd5c4e619db0d925b1b7dda3]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (2) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
W20250815 13:54:10.077564 4058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:54:10.078178 4058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:54:10.078675 4058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:54:10.116815 4058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:54:10.117867 4058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.131
I20250815 13:54:10.158365 4058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:39749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.131:36587
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.106.131
--webserver_port=36245
--tserver_master_addrs=127.0.106.190:35951
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.131
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:54:10.159988 4058 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:54:10.161834 4058 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:54:10.177011 4075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 13:54:10.286936 4081 ts_tablet_manager.cc:927] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a: Initiating tablet copy from peer 90a13caa51354a19b2434ba2e76cf1f3 (127.0.106.130:46037)
I20250815 13:54:10.290081 4081 tablet_copy_client.cc:323] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.106.130:46037
I20250815 13:54:10.306051 4029 tablet_copy_service.cc:140] P 90a13caa51354a19b2434ba2e76cf1f3: Received BeginTabletCopySession request for tablet 9b39e98a33744b249184945f4451802d from peer f86092b7fc7a46dfb843a939dd4fbc6a ({username='slave'} at 127.0.106.129:50179)
I20250815 13:54:10.306711 4029 tablet_copy_service.cc:161] P 90a13caa51354a19b2434ba2e76cf1f3: Beginning new tablet copy session on tablet 9b39e98a33744b249184945f4451802d from peer f86092b7fc7a46dfb843a939dd4fbc6a at {username='slave'} at 127.0.106.129:50179: session id = f86092b7fc7a46dfb843a939dd4fbc6a-9b39e98a33744b249184945f4451802d
I20250815 13:54:10.314929 4029 tablet_copy_source_session.cc:215] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3: Tablet Copy: opened 0 blocks and 1 log segments
I20250815 13:54:10.321077 4081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b39e98a33744b249184945f4451802d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:54:10.341958 4081 tablet_copy_client.cc:806] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a: tablet copy: Starting download of 0 data blocks...
I20250815 13:54:10.342736 4081 tablet_copy_client.cc:670] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a: tablet copy: Starting download of 1 WAL segments...
I20250815 13:54:10.349413 4081 tablet_copy_client.cc:538] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250815 13:54:10.360488 4081 tablet_bootstrap.cc:492] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a: Bootstrap starting.
I20250815 13:54:10.528494 4081 tablet_bootstrap.cc:492] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 13:54:10.529410 4081 tablet_bootstrap.cc:492] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a: Bootstrap complete.
I20250815 13:54:10.530045 4081 ts_tablet_manager.cc:1397] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a: Time spent bootstrapping tablet: real 0.170s user 0.137s sys 0.026s
I20250815 13:54:10.532689 4081 raft_consensus.cc:357] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: NON_VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: true } }
I20250815 13:54:10.533368 4081 raft_consensus.cc:738] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: f86092b7fc7a46dfb843a939dd4fbc6a, State: Initialized, Role: LEARNER
I20250815 13:54:10.533962 4081 consensus_queue.cc:260] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: NON_VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: true } }
I20250815 13:54:10.537986 4081 ts_tablet_manager.cc:1428] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a: Time spent starting tablet: real 0.008s user 0.006s sys 0.004s
I20250815 13:54:10.540937 4029 tablet_copy_service.cc:342] P 90a13caa51354a19b2434ba2e76cf1f3: Request end of tablet copy session f86092b7fc7a46dfb843a939dd4fbc6a-9b39e98a33744b249184945f4451802d received from {username='slave'} at 127.0.106.129:50179
I20250815 13:54:10.541484 4029 tablet_copy_service.cc:434] P 90a13caa51354a19b2434ba2e76cf1f3: ending tablet copy session f86092b7fc7a46dfb843a939dd4fbc6a-9b39e98a33744b249184945f4451802d on tablet 9b39e98a33744b249184945f4451802d with peer f86092b7fc7a46dfb843a939dd4fbc6a
I20250815 13:54:10.907819 3865 raft_consensus.cc:1215] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 LEARNER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
W20250815 13:54:10.177315 4076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:54:11.362555 4078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:54:11.365335 4077 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1182 milliseconds
I20250815 13:54:11.365499 4058 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 13:54:11.366874 4058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:54:11.369557 4058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:54:11.371093 4058 hybrid_clock.cc:648] HybridClock initialized: now 1755266051370984 us; error 69 us; skew 500 ppm
I20250815 13:54:11.372072 4058 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:54:11.379323 4058 webserver.cc:480] Webserver started at http://127.0.106.131:36245/ using document root <none> and password file <none>
I20250815 13:54:11.380409 4058 fs_manager.cc:362] Metadata directory not provided
I20250815 13:54:11.380666 4058 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:54:11.389534 4058 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.001s sys 0.004s
I20250815 13:54:11.395442 4093 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:54:11.396895 4058 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20250815 13:54:11.397322 4058 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "34201ba276c14ae6bc93bb059132d83f"
format_stamp: "Formatted at 2025-08-15 13:53:57 on dist-test-slave-kt02"
I20250815 13:54:11.399636 4058 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:54:11.443511 4096 raft_consensus.cc:1062] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3: attempting to promote NON_VOTER f86092b7fc7a46dfb843a939dd4fbc6a to VOTER
I20250815 13:54:11.444947 4096 consensus_queue.cc:237] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } }
I20250815 13:54:11.450135 3865 raft_consensus.cc:1273] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 LEARNER]: Refusing update from remote peer 90a13caa51354a19b2434ba2e76cf1f3: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250815 13:54:11.451613 4096 consensus_queue.cc:1035] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250815 13:54:11.459247 4099 raft_consensus.cc:2953] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, f86092b7fc7a46dfb843a939dd4fbc6a (127.0.106.129) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } } }
I20250815 13:54:11.460656 3865 raft_consensus.cc:2953] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, f86092b7fc7a46dfb843a939dd4fbc6a (127.0.106.129) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } } }
I20250815 13:54:11.466405 4058 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:54:11.468521 4058 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:54:11.469146 4058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:54:11.472783 4058 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 13:54:11.473601 3722 catalog_manager.cc:5582] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 reported cstate change: config changed from index 10 to 11, f86092b7fc7a46dfb843a939dd4fbc6a (127.0.106.129) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "90a13caa51354a19b2434ba2e76cf1f3" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250815 13:54:11.479619 4107 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250815 13:54:11.490290 4058 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250815 13:54:11.490579 4058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.002s sys 0.000s
I20250815 13:54:11.490869 4058 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250815 13:54:11.496435 4058 ts_tablet_manager.cc:610] Registered 1 tablets
I20250815 13:54:11.496717 4058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s user 0.005s sys 0.000s
I20250815 13:54:11.497136 4107 tablet_bootstrap.cc:492] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f: Bootstrap starting.
I20250815 13:54:11.581714 4107 log.cc:826] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f: Log is configured to *not* fsync() on all Append() calls
I20250815 13:54:11.669838 4107 tablet_bootstrap.cc:492] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 13:54:11.670655 4107 tablet_bootstrap.cc:492] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f: Bootstrap complete.
I20250815 13:54:11.672345 4107 ts_tablet_manager.cc:1397] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f: Time spent bootstrapping tablet: real 0.176s user 0.137s sys 0.036s
I20250815 13:54:11.688130 4107 raft_consensus.cc:357] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } }
I20250815 13:54:11.690737 4107 raft_consensus.cc:738] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34201ba276c14ae6bc93bb059132d83f, State: Initialized, Role: FOLLOWER
I20250815 13:54:11.691603 4107 consensus_queue.cc:260] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } }
I20250815 13:54:11.692283 4107 raft_consensus.cc:397] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 13:54:11.692590 4107 raft_consensus.cc:491] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 13:54:11.693029 4107 raft_consensus.cc:3058] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [term 1 FOLLOWER]: Advancing to term 2
I20250815 13:54:11.695276 4058 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.131:36587
I20250815 13:54:11.695461 4217 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.131:36587 every 8 connection(s)
I20250815 13:54:11.698490 4058 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250815 13:54:11.698832 4107 raft_consensus.cc:513] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } }
I20250815 13:54:11.699752 4107 leader_election.cc:304] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34201ba276c14ae6bc93bb059132d83f; no voters:
I20250815 13:54:11.707835 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 4058
I20250815 13:54:11.727598 4107 leader_election.cc:290] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [CANDIDATE]: Term 2 election: Requested vote from peers
I20250815 13:54:11.728219 4219 raft_consensus.cc:2802] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [term 2 FOLLOWER]: Leader election won for term 2
I20250815 13:54:11.736948 4219 raft_consensus.cc:695] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [term 2 LEADER]: Becoming Leader. State: Replica: 34201ba276c14ae6bc93bb059132d83f, State: Running, Role: LEADER
I20250815 13:54:11.737838 4107 ts_tablet_manager.cc:1428] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f: Time spent starting tablet: real 0.065s user 0.039s sys 0.024s
I20250815 13:54:11.737965 4219 consensus_queue.cc:237] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } }
I20250815 13:54:11.752233 4218 heartbeater.cc:344] Connected to a master server at 127.0.106.190:35951
I20250815 13:54:11.752718 4218 heartbeater.cc:461] Registering TS with master...
I20250815 13:54:11.753769 4218 heartbeater.cc:507] Master 127.0.106.190:35951 requested a full tablet report, sending...
I20250815 13:54:11.757633 3722 ts_manager.cc:194] Registered new tserver with Master: 34201ba276c14ae6bc93bb059132d83f (127.0.106.131:36587)
I20250815 13:54:11.758807 3722 catalog_manager.cc:5582] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f reported cstate change: term changed from 0 to 2, leader changed from <none> to 34201ba276c14ae6bc93bb059132d83f (127.0.106.131), VOTER 34201ba276c14ae6bc93bb059132d83f (127.0.106.131) added. New cstate: current_term: 2 leader_uuid: "34201ba276c14ae6bc93bb059132d83f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } health_report { overall_health: HEALTHY } } }
I20250815 13:54:11.762496 426 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 13:54:11.767681 426 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 13:54:11.769076 3722 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.131:34673
I20250815 13:54:11.772562 4218 heartbeater.cc:499] Master 127.0.106.190:35951 was elected leader, sending a full tablet report...
W20250815 13:54:11.782896 426 ts_itest-base.cc:209] found only 2 out of 3 replicas of tablet 9b39e98a33744b249184945f4451802d: tablet_id: "9b39e98a33744b249184945f4451802d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER } interned_replicas { ts_info_idx: 1 role: FOLLOWER }
I20250815 13:54:11.790719 4173 consensus_queue.cc:237] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } } peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: NON_VOTER last_known_addr { host: "127.0.106.130" port: 46037 } attrs { promote: true } }
I20250815 13:54:11.794919 4225 raft_consensus.cc:2953] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index -1 to 9, NON_VOTER 90a13caa51354a19b2434ba2e76cf1f3 (127.0.106.130) added. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } } peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: NON_VOTER last_known_addr { host: "127.0.106.130" port: 46037 } attrs { promote: true } } }
I20250815 13:54:11.804437 3706 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c4cc2a5449c34c269c21d85e6cb450c0 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250815 13:54:11.807356 4097 consensus_peers.cc:489] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f -> Peer 90a13caa51354a19b2434ba2e76cf1f3 (127.0.106.130:46037): Couldn't send request to peer 90a13caa51354a19b2434ba2e76cf1f3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c4cc2a5449c34c269c21d85e6cb450c0. This is attempt 1: this message will repeat every 5th retry.
I20250815 13:54:11.807926 3722 catalog_manager.cc:5582] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f reported cstate change: config changed from index -1 to 9, NON_VOTER 90a13caa51354a19b2434ba2e76cf1f3 (127.0.106.130) added. New cstate: current_term: 2 leader_uuid: "34201ba276c14ae6bc93bb059132d83f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: NON_VOTER last_known_addr { host: "127.0.106.130" port: 46037 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250815 13:54:11.818300 4173 consensus_queue.cc:237] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } } peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: NON_VOTER last_known_addr { host: "127.0.106.130" port: 46037 } attrs { promote: true } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: NON_VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: true } }
I20250815 13:54:11.822037 4219 raft_consensus.cc:2953] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, NON_VOTER f86092b7fc7a46dfb843a939dd4fbc6a (127.0.106.129) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } } peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: NON_VOTER last_known_addr { host: "127.0.106.130" port: 46037 } attrs { promote: true } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: NON_VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: true } } }
W20250815 13:54:11.824611 4097 consensus_peers.cc:489] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f -> Peer 90a13caa51354a19b2434ba2e76cf1f3 (127.0.106.130:46037): Couldn't send request to peer 90a13caa51354a19b2434ba2e76cf1f3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c4cc2a5449c34c269c21d85e6cb450c0. This is attempt 1: this message will repeat every 5th retry.
I20250815 13:54:11.830319 3706 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c4cc2a5449c34c269c21d85e6cb450c0 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250815 13:54:11.833626 4097 consensus_peers.cc:489] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f -> Peer f86092b7fc7a46dfb843a939dd4fbc6a (127.0.106.129:33117): Couldn't send request to peer f86092b7fc7a46dfb843a939dd4fbc6a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c4cc2a5449c34c269c21d85e6cb450c0. This is attempt 1: this message will repeat every 5th retry.
I20250815 13:54:11.834412 4009 consensus_queue.cc:237] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } } peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: NON_VOTER last_known_addr { host: "127.0.106.131" port: 36587 } attrs { promote: true } }
I20250815 13:54:11.834383 3722 catalog_manager.cc:5582] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f reported cstate change: config changed from index 9 to 10, NON_VOTER f86092b7fc7a46dfb843a939dd4fbc6a (127.0.106.129) added. New cstate: current_term: 2 leader_uuid: "34201ba276c14ae6bc93bb059132d83f" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: NON_VOTER last_known_addr { host: "127.0.106.130" port: 46037 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: NON_VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250815 13:54:11.839726 3865 raft_consensus.cc:1273] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 FOLLOWER]: Refusing update from remote peer 90a13caa51354a19b2434ba2e76cf1f3: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250815 13:54:11.841455 4096 consensus_queue.cc:1035] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250815 13:54:11.848001 4099 raft_consensus.cc:2953] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 34201ba276c14ae6bc93bb059132d83f (127.0.106.131) added. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } } peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: NON_VOTER last_known_addr { host: "127.0.106.131" port: 36587 } attrs { promote: true } } }
W20250815 13:54:11.851714 3940 consensus_peers.cc:489] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 -> Peer 34201ba276c14ae6bc93bb059132d83f (127.0.106.131:36587): Couldn't send request to peer 34201ba276c14ae6bc93bb059132d83f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b39e98a33744b249184945f4451802d. This is attempt 1: this message will repeat every 5th retry.
I20250815 13:54:11.850804 3865 raft_consensus.cc:2953] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 34201ba276c14ae6bc93bb059132d83f (127.0.106.131) added. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } } peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: NON_VOTER last_known_addr { host: "127.0.106.131" port: 36587 } attrs { promote: true } } }
I20250815 13:54:11.858685 3708 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9b39e98a33744b249184945f4451802d with cas_config_opid_index 11: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 5)
I20250815 13:54:11.861786 3722 catalog_manager.cc:5582] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 reported cstate change: config changed from index 11 to 12, NON_VOTER 34201ba276c14ae6bc93bb059132d83f (127.0.106.131) added. New cstate: current_term: 2 leader_uuid: "90a13caa51354a19b2434ba2e76cf1f3" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: NON_VOTER last_known_addr { host: "127.0.106.131" port: 36587 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250815 13:54:12.132995 3707 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9b39e98a33744b249184945f4451802d with cas_config_opid_index 10: aborting the task: latest config opid_index 12; task opid_index 10
W20250815 13:54:12.167549 3753 debug-util.cc:398] Leaking SignalData structure 0x7b08000873e0 after lost signal to thread 3690
I20250815 13:54:12.256208 4234 ts_tablet_manager.cc:927] T c4cc2a5449c34c269c21d85e6cb450c0 P 90a13caa51354a19b2434ba2e76cf1f3: Initiating tablet copy from peer 34201ba276c14ae6bc93bb059132d83f (127.0.106.131:36587)
I20250815 13:54:12.258500 4234 tablet_copy_client.cc:323] T c4cc2a5449c34c269c21d85e6cb450c0 P 90a13caa51354a19b2434ba2e76cf1f3: tablet copy: Beginning tablet copy session from remote peer at address 127.0.106.131:36587
I20250815 13:54:12.260705 4193 tablet_copy_service.cc:140] P 34201ba276c14ae6bc93bb059132d83f: Received BeginTabletCopySession request for tablet c4cc2a5449c34c269c21d85e6cb450c0 from peer 90a13caa51354a19b2434ba2e76cf1f3 ({username='slave'} at 127.0.106.130:34265)
I20250815 13:54:12.261413 4193 tablet_copy_service.cc:161] P 34201ba276c14ae6bc93bb059132d83f: Beginning new tablet copy session on tablet c4cc2a5449c34c269c21d85e6cb450c0 from peer 90a13caa51354a19b2434ba2e76cf1f3 at {username='slave'} at 127.0.106.130:34265: session id = 90a13caa51354a19b2434ba2e76cf1f3-c4cc2a5449c34c269c21d85e6cb450c0
I20250815 13:54:12.269114 4193 tablet_copy_source_session.cc:215] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f: Tablet Copy: opened 0 blocks and 1 log segments
I20250815 13:54:12.273684 4234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4cc2a5449c34c269c21d85e6cb450c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:54:12.283885 4234 tablet_copy_client.cc:806] T c4cc2a5449c34c269c21d85e6cb450c0 P 90a13caa51354a19b2434ba2e76cf1f3: tablet copy: Starting download of 0 data blocks...
I20250815 13:54:12.284438 4234 tablet_copy_client.cc:670] T c4cc2a5449c34c269c21d85e6cb450c0 P 90a13caa51354a19b2434ba2e76cf1f3: tablet copy: Starting download of 1 WAL segments...
I20250815 13:54:12.289299 4234 tablet_copy_client.cc:538] T c4cc2a5449c34c269c21d85e6cb450c0 P 90a13caa51354a19b2434ba2e76cf1f3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250815 13:54:12.294977 4234 tablet_bootstrap.cc:492] T c4cc2a5449c34c269c21d85e6cb450c0 P 90a13caa51354a19b2434ba2e76cf1f3: Bootstrap starting.
I20250815 13:54:12.333249 4237 ts_tablet_manager.cc:927] T 9b39e98a33744b249184945f4451802d P 34201ba276c14ae6bc93bb059132d83f: Initiating tablet copy from peer 90a13caa51354a19b2434ba2e76cf1f3 (127.0.106.130:46037)
I20250815 13:54:12.335887 4237 tablet_copy_client.cc:323] T 9b39e98a33744b249184945f4451802d P 34201ba276c14ae6bc93bb059132d83f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.106.130:46037
I20250815 13:54:12.337885 4029 tablet_copy_service.cc:140] P 90a13caa51354a19b2434ba2e76cf1f3: Received BeginTabletCopySession request for tablet 9b39e98a33744b249184945f4451802d from peer 34201ba276c14ae6bc93bb059132d83f ({username='slave'} at 127.0.106.131:34021)
I20250815 13:54:12.338552 4029 tablet_copy_service.cc:161] P 90a13caa51354a19b2434ba2e76cf1f3: Beginning new tablet copy session on tablet 9b39e98a33744b249184945f4451802d from peer 34201ba276c14ae6bc93bb059132d83f at {username='slave'} at 127.0.106.131:34021: session id = 34201ba276c14ae6bc93bb059132d83f-9b39e98a33744b249184945f4451802d
I20250815 13:54:12.345911 4029 tablet_copy_source_session.cc:215] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3: Tablet Copy: opened 0 blocks and 1 log segments
I20250815 13:54:12.349804 4237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b39e98a33744b249184945f4451802d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:54:12.369267 4237 tablet_copy_client.cc:806] T 9b39e98a33744b249184945f4451802d P 34201ba276c14ae6bc93bb059132d83f: tablet copy: Starting download of 0 data blocks...
I20250815 13:54:12.369904 4237 tablet_copy_client.cc:670] T 9b39e98a33744b249184945f4451802d P 34201ba276c14ae6bc93bb059132d83f: tablet copy: Starting download of 1 WAL segments...
I20250815 13:54:12.372924 4239 ts_tablet_manager.cc:927] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a: Initiating tablet copy from peer 34201ba276c14ae6bc93bb059132d83f (127.0.106.131:36587)
I20250815 13:54:12.374850 4237 tablet_copy_client.cc:538] T 9b39e98a33744b249184945f4451802d P 34201ba276c14ae6bc93bb059132d83f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250815 13:54:12.384485 4237 tablet_bootstrap.cc:492] T 9b39e98a33744b249184945f4451802d P 34201ba276c14ae6bc93bb059132d83f: Bootstrap starting.
I20250815 13:54:12.401258 4234 tablet_bootstrap.cc:492] T c4cc2a5449c34c269c21d85e6cb450c0 P 90a13caa51354a19b2434ba2e76cf1f3: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 13:54:12.402266 4234 tablet_bootstrap.cc:492] T c4cc2a5449c34c269c21d85e6cb450c0 P 90a13caa51354a19b2434ba2e76cf1f3: Bootstrap complete.
I20250815 13:54:12.402951 4234 ts_tablet_manager.cc:1397] T c4cc2a5449c34c269c21d85e6cb450c0 P 90a13caa51354a19b2434ba2e76cf1f3: Time spent bootstrapping tablet: real 0.108s user 0.098s sys 0.008s
I20250815 13:54:12.403530 4239 tablet_copy_client.cc:323] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.106.131:36587
I20250815 13:54:12.405324 4234 raft_consensus.cc:357] T c4cc2a5449c34c269c21d85e6cb450c0 P 90a13caa51354a19b2434ba2e76cf1f3 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } } peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: NON_VOTER last_known_addr { host: "127.0.106.130" port: 46037 } attrs { promote: true } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: NON_VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: true } }
I20250815 13:54:12.406214 4234 raft_consensus.cc:738] T c4cc2a5449c34c269c21d85e6cb450c0 P 90a13caa51354a19b2434ba2e76cf1f3 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 90a13caa51354a19b2434ba2e76cf1f3, State: Initialized, Role: LEARNER
I20250815 13:54:12.406832 4234 consensus_queue.cc:260] T c4cc2a5449c34c269c21d85e6cb450c0 P 90a13caa51354a19b2434ba2e76cf1f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } } peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: NON_VOTER last_known_addr { host: "127.0.106.130" port: 46037 } attrs { promote: true } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: NON_VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: true } }
I20250815 13:54:12.410411 4234 ts_tablet_manager.cc:1428] T c4cc2a5449c34c269c21d85e6cb450c0 P 90a13caa51354a19b2434ba2e76cf1f3: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250815 13:54:12.414815 4193 tablet_copy_service.cc:342] P 34201ba276c14ae6bc93bb059132d83f: Request end of tablet copy session 90a13caa51354a19b2434ba2e76cf1f3-c4cc2a5449c34c269c21d85e6cb450c0 received from {username='slave'} at 127.0.106.130:34265
I20250815 13:54:12.415287 4193 tablet_copy_service.cc:434] P 34201ba276c14ae6bc93bb059132d83f: ending tablet copy session 90a13caa51354a19b2434ba2e76cf1f3-c4cc2a5449c34c269c21d85e6cb450c0 on tablet c4cc2a5449c34c269c21d85e6cb450c0 with peer 90a13caa51354a19b2434ba2e76cf1f3
I20250815 13:54:12.426000 4193 tablet_copy_service.cc:140] P 34201ba276c14ae6bc93bb059132d83f: Received BeginTabletCopySession request for tablet c4cc2a5449c34c269c21d85e6cb450c0 from peer f86092b7fc7a46dfb843a939dd4fbc6a ({username='slave'} at 127.0.106.129:38535)
I20250815 13:54:12.426573 4193 tablet_copy_service.cc:161] P 34201ba276c14ae6bc93bb059132d83f: Beginning new tablet copy session on tablet c4cc2a5449c34c269c21d85e6cb450c0 from peer f86092b7fc7a46dfb843a939dd4fbc6a at {username='slave'} at 127.0.106.129:38535: session id = f86092b7fc7a46dfb843a939dd4fbc6a-c4cc2a5449c34c269c21d85e6cb450c0
I20250815 13:54:12.433509 4193 tablet_copy_source_session.cc:215] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f: Tablet Copy: opened 0 blocks and 1 log segments
I20250815 13:54:12.436857 4239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4cc2a5449c34c269c21d85e6cb450c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:54:12.453240 4239 tablet_copy_client.cc:806] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a: tablet copy: Starting download of 0 data blocks...
I20250815 13:54:12.453867 4239 tablet_copy_client.cc:670] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a: tablet copy: Starting download of 1 WAL segments...
I20250815 13:54:12.458789 4239 tablet_copy_client.cc:538] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250815 13:54:12.467998 4239 tablet_bootstrap.cc:492] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a: Bootstrap starting.
I20250815 13:54:12.527617 4237 tablet_bootstrap.cc:492] T 9b39e98a33744b249184945f4451802d P 34201ba276c14ae6bc93bb059132d83f: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 13:54:12.528554 4237 tablet_bootstrap.cc:492] T 9b39e98a33744b249184945f4451802d P 34201ba276c14ae6bc93bb059132d83f: Bootstrap complete.
I20250815 13:54:12.529177 4237 ts_tablet_manager.cc:1397] T 9b39e98a33744b249184945f4451802d P 34201ba276c14ae6bc93bb059132d83f: Time spent bootstrapping tablet: real 0.145s user 0.133s sys 0.008s
I20250815 13:54:12.531761 4237 raft_consensus.cc:357] T 9b39e98a33744b249184945f4451802d P 34201ba276c14ae6bc93bb059132d83f [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } } peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: NON_VOTER last_known_addr { host: "127.0.106.131" port: 36587 } attrs { promote: true } }
I20250815 13:54:12.532485 4237 raft_consensus.cc:738] T 9b39e98a33744b249184945f4451802d P 34201ba276c14ae6bc93bb059132d83f [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 34201ba276c14ae6bc93bb059132d83f, State: Initialized, Role: LEARNER
I20250815 13:54:12.533052 4237 consensus_queue.cc:260] T 9b39e98a33744b249184945f4451802d P 34201ba276c14ae6bc93bb059132d83f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } } peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: NON_VOTER last_known_addr { host: "127.0.106.131" port: 36587 } attrs { promote: true } }
I20250815 13:54:12.535168 4237 ts_tablet_manager.cc:1428] T 9b39e98a33744b249184945f4451802d P 34201ba276c14ae6bc93bb059132d83f: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20250815 13:54:12.537045 4029 tablet_copy_service.cc:342] P 90a13caa51354a19b2434ba2e76cf1f3: Request end of tablet copy session 34201ba276c14ae6bc93bb059132d83f-9b39e98a33744b249184945f4451802d received from {username='slave'} at 127.0.106.131:34021
I20250815 13:54:12.537482 4029 tablet_copy_service.cc:434] P 90a13caa51354a19b2434ba2e76cf1f3: ending tablet copy session 34201ba276c14ae6bc93bb059132d83f-9b39e98a33744b249184945f4451802d on tablet 9b39e98a33744b249184945f4451802d with peer 34201ba276c14ae6bc93bb059132d83f
I20250815 13:54:12.557770 4239 tablet_bootstrap.cc:492] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 13:54:12.558390 4239 tablet_bootstrap.cc:492] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a: Bootstrap complete.
I20250815 13:54:12.558813 4239 ts_tablet_manager.cc:1397] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a: Time spent bootstrapping tablet: real 0.091s user 0.083s sys 0.004s
I20250815 13:54:12.560380 4239 raft_consensus.cc:357] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } } peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: NON_VOTER last_known_addr { host: "127.0.106.130" port: 46037 } attrs { promote: true } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: NON_VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: true } }
I20250815 13:54:12.560882 4239 raft_consensus.cc:738] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: f86092b7fc7a46dfb843a939dd4fbc6a, State: Initialized, Role: LEARNER
I20250815 13:54:12.561305 4239 consensus_queue.cc:260] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } } peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: NON_VOTER last_known_addr { host: "127.0.106.130" port: 46037 } attrs { promote: true } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: NON_VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: true } }
I20250815 13:54:12.564121 4239 ts_tablet_manager.cc:1428] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a: Time spent starting tablet: real 0.005s user 0.006s sys 0.002s
I20250815 13:54:12.565939 4193 tablet_copy_service.cc:342] P 34201ba276c14ae6bc93bb059132d83f: Request end of tablet copy session f86092b7fc7a46dfb843a939dd4fbc6a-c4cc2a5449c34c269c21d85e6cb450c0 received from {username='slave'} at 127.0.106.129:38535
I20250815 13:54:12.566300 4193 tablet_copy_service.cc:434] P 34201ba276c14ae6bc93bb059132d83f: ending tablet copy session f86092b7fc7a46dfb843a939dd4fbc6a-c4cc2a5449c34c269c21d85e6cb450c0 on tablet c4cc2a5449c34c269c21d85e6cb450c0 with peer f86092b7fc7a46dfb843a939dd4fbc6a
I20250815 13:54:12.788591 426 ts_itest-base.cc:246] Waiting for 1 tablets on tserver f86092b7fc7a46dfb843a939dd4fbc6a to finish bootstrapping
I20250815 13:54:12.802978 3865 raft_consensus.cc:1215] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 LEARNER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20250815 13:54:12.808104 426 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 90a13caa51354a19b2434ba2e76cf1f3 to finish bootstrapping
I20250815 13:54:12.825901 426 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 34201ba276c14ae6bc93bb059132d83f to finish bootstrapping
I20250815 13:54:12.849514 4009 raft_consensus.cc:1215] T c4cc2a5449c34c269c21d85e6cb450c0 P 90a13caa51354a19b2434ba2e76cf1f3 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20250815 13:54:12.874042 4173 raft_consensus.cc:1215] T 9b39e98a33744b249184945f4451802d P 34201ba276c14ae6bc93bb059132d83f [term 2 LEARNER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
I20250815 13:54:13.159615 3989 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250815 13:54:13.162907 4152 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250815 13:54:13.172616 3845 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250815 13:54:13.313016 4250 raft_consensus.cc:1062] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f: attempting to promote NON_VOTER f86092b7fc7a46dfb843a939dd4fbc6a to VOTER
I20250815 13:54:13.315078 4250 consensus_queue.cc:237] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } } peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: NON_VOTER last_known_addr { host: "127.0.106.130" port: 46037 } attrs { promote: true } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } }
I20250815 13:54:13.337390 3865 raft_consensus.cc:1273] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 LEARNER]: Refusing update from remote peer 34201ba276c14ae6bc93bb059132d83f: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250815 13:54:13.338136 4009 raft_consensus.cc:1273] T c4cc2a5449c34c269c21d85e6cb450c0 P 90a13caa51354a19b2434ba2e76cf1f3 [term 2 LEARNER]: Refusing update from remote peer 34201ba276c14ae6bc93bb059132d83f: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250815 13:54:13.339808 4250 consensus_queue.cc:1035] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: NON_VOTER last_known_addr { host: "127.0.106.130" port: 46037 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250815 13:54:13.341260 4250 consensus_queue.cc:1035] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250815 13:54:13.363767 4250 raft_consensus.cc:2953] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, f86092b7fc7a46dfb843a939dd4fbc6a (127.0.106.129) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } } peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: NON_VOTER last_known_addr { host: "127.0.106.130" port: 46037 } attrs { promote: true } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } } }
I20250815 13:54:13.366364 3864 raft_consensus.cc:2953] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, f86092b7fc7a46dfb843a939dd4fbc6a (127.0.106.129) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } } peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: NON_VOTER last_known_addr { host: "127.0.106.130" port: 46037 } attrs { promote: true } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } } }
I20250815 13:54:13.377899 3720 catalog_manager.cc:5582] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f reported cstate change: config changed from index 10 to 11, f86092b7fc7a46dfb843a939dd4fbc6a (127.0.106.129) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "34201ba276c14ae6bc93bb059132d83f" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: NON_VOTER last_known_addr { host: "127.0.106.130" port: 46037 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250815 13:54:13.391290 4225 raft_consensus.cc:1062] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f: attempting to promote NON_VOTER 90a13caa51354a19b2434ba2e76cf1f3 to VOTER
I20250815 13:54:13.388782 4009 raft_consensus.cc:2953] T c4cc2a5449c34c269c21d85e6cb450c0 P 90a13caa51354a19b2434ba2e76cf1f3 [term 2 LEARNER]: Committing config change with OpId 2.11: config changed from index 10 to 11, f86092b7fc7a46dfb843a939dd4fbc6a (127.0.106.129) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } } peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: NON_VOTER last_known_addr { host: "127.0.106.130" port: 46037 } attrs { promote: true } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } } }
I20250815 13:54:13.393069 4225 consensus_queue.cc:237] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } } peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } attrs { promote: false } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } }
I20250815 13:54:13.403340 3864 raft_consensus.cc:1273] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 FOLLOWER]: Refusing update from remote peer 34201ba276c14ae6bc93bb059132d83f: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250815 13:54:13.404896 4225 consensus_queue.cc:1035] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250815 13:54:13.412485 4249 raft_consensus.cc:2953] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 90a13caa51354a19b2434ba2e76cf1f3 (127.0.106.130) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } } peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } attrs { promote: false } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } } }
I20250815 13:54:13.424736 4008 raft_consensus.cc:1273] T c4cc2a5449c34c269c21d85e6cb450c0 P 90a13caa51354a19b2434ba2e76cf1f3 [term 2 LEARNER]: Refusing update from remote peer 34201ba276c14ae6bc93bb059132d83f: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250815 13:54:13.425500 3864 raft_consensus.cc:2953] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 90a13caa51354a19b2434ba2e76cf1f3 (127.0.106.130) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } } peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } attrs { promote: false } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } } }
I20250815 13:54:13.428737 4249 consensus_queue.cc:1035] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250815 13:54:13.438706 4008 raft_consensus.cc:2953] T c4cc2a5449c34c269c21d85e6cb450c0 P 90a13caa51354a19b2434ba2e76cf1f3 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 90a13caa51354a19b2434ba2e76cf1f3 (127.0.106.130) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } } peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } attrs { promote: false } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } } }
I20250815 13:54:13.481953 3721 catalog_manager.cc:5582] T c4cc2a5449c34c269c21d85e6cb450c0 P 90a13caa51354a19b2434ba2e76cf1f3 reported cstate change: config changed from index 11 to 12, 90a13caa51354a19b2434ba2e76cf1f3 (127.0.106.130) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "34201ba276c14ae6bc93bb059132d83f" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } } peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } attrs { promote: false } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } } }
I20250815 13:54:13.498054 4286 raft_consensus.cc:1062] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3: attempting to promote NON_VOTER 34201ba276c14ae6bc93bb059132d83f to VOTER
I20250815 13:54:13.500483 4286 consensus_queue.cc:237] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } } peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } attrs { promote: false } }
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
993462eb9d054343874cf1ca224fbc13 | 127.0.106.190:35951 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
I20250815 13:54:13.509727 4173 raft_consensus.cc:1273] T 9b39e98a33744b249184945f4451802d P 34201ba276c14ae6bc93bb059132d83f [term 2 LEARNER]: Refusing update from remote peer 90a13caa51354a19b2434ba2e76cf1f3: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.0.106.148:39749 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
34201ba276c14ae6bc93bb059132d83f | 127.0.106.131:36587 | HEALTHY | <none> | 1 | 0
90a13caa51354a19b2434ba2e76cf1f3 | 127.0.106.130:46037 | HEALTHY | <none> | 1 | 0
f86092b7fc7a46dfb843a939dd4fbc6a | 127.0.106.129:33117 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
I20250815 13:54:13.511502 3864 raft_consensus.cc:1273] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 FOLLOWER]: Refusing update from remote peer 90a13caa51354a19b2434ba2e76cf1f3: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
I20250815 13:54:13.511997 4286 consensus_queue.cc:1035] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.0.106.129 | experimental | 127.0.106.129:33117
local_ip_for_outbound_sockets | 127.0.106.130 | experimental | 127.0.106.130:46037
local_ip_for_outbound_sockets | 127.0.106.131 | experimental | 127.0.106.131:36587
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
I20250815 13:54:13.513172 4259 consensus_queue.cc:1035] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.106.129:33117
server_dump_info_path | /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.106.130:46037
server_dump_info_path | /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.106.131:36587
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.0.106.148:39749 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable1 | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable2 | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 3
Tablets | 3
Replicas | 7
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250815 13:54:13.523422 4259 raft_consensus.cc:2953] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 34201ba276c14ae6bc93bb059132d83f (127.0.106.131) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } } peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } attrs { promote: false } } }
I20250815 13:54:13.528754 3864 raft_consensus.cc:2953] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 34201ba276c14ae6bc93bb059132d83f (127.0.106.131) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } } peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } attrs { promote: false } } }
I20250815 13:54:13.534922 426 log_verifier.cc:126] Checking tablet 672f8196d0cb413cb7c514171eb967fd
I20250815 13:54:13.542394 4173 raft_consensus.cc:2953] T 9b39e98a33744b249184945f4451802d P 34201ba276c14ae6bc93bb059132d83f [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 34201ba276c14ae6bc93bb059132d83f (127.0.106.131) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } } peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } attrs { promote: false } } }
I20250815 13:54:13.545606 3720 catalog_manager.cc:5582] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 reported cstate change: config changed from index 12 to 13, 34201ba276c14ae6bc93bb059132d83f (127.0.106.131) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "90a13caa51354a19b2434ba2e76cf1f3" committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250815 13:54:13.584100 426 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 672f8196d0cb413cb7c514171eb967fd
I20250815 13:54:13.584396 426 log_verifier.cc:126] Checking tablet 9b39e98a33744b249184945f4451802d
I20250815 13:54:13.702545 426 log_verifier.cc:177] Verified matching terms for 13 ops in tablet 9b39e98a33744b249184945f4451802d
I20250815 13:54:13.702944 426 log_verifier.cc:126] Checking tablet c4cc2a5449c34c269c21d85e6cb450c0
I20250815 13:54:13.810719 426 log_verifier.cc:177] Verified matching terms for 12 ops in tablet c4cc2a5449c34c269c21d85e6cb450c0
I20250815 13:54:13.811192 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 3689
I20250815 13:54:13.834654 426 minidump.cc:252] Setting minidump size limit to 20M
I20250815 13:54:13.836035 426 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:54:13.837073 426 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:54:13.848121 4292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:54:13.899873 4218 heartbeater.cc:646] Failed to heartbeat to 127.0.106.190:35951 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.106.190:35951: connect: Connection refused (error 111)
W20250815 13:54:13.849359 4295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:54:13.848114 4293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 13:54:13.982702 426 server_base.cc:1047] running on GCE node
I20250815 13:54:13.984058 426 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250815 13:54:13.984283 426 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250815 13:54:13.984427 426 hybrid_clock.cc:648] HybridClock initialized: now 1755266053984413 us; error 0 us; skew 500 ppm
I20250815 13:54:13.985091 426 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:54:13.995548 426 webserver.cc:480] Webserver started at http://0.0.0.0:37221/ using document root <none> and password file <none>
I20250815 13:54:13.996999 426 fs_manager.cc:362] Metadata directory not provided
I20250815 13:54:13.997372 426 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:54:14.006558 426 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.001s
I20250815 13:54:14.012096 4301 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:54:14.013386 426 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250815 13:54:14.013777 426 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
uuid: "993462eb9d054343874cf1ca224fbc13"
format_stamp: "Formatted at 2025-08-15 13:53:50 on dist-test-slave-kt02"
I20250815 13:54:14.015727 426 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:54:14.039975 426 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:54:14.042012 426 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:54:14.042600 426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:54:14.053045 426 sys_catalog.cc:263] Verifying existing consensus state
W20250815 13:54:14.056488 426 sys_catalog.cc:243] For a single master config, on-disk Raft master: 127.0.106.190:35951 exists but no master address supplied!
I20250815 13:54:14.058498 426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13: Bootstrap starting.
I20250815 13:54:14.100796 426 log.cc:826] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13: Log is configured to *not* fsync() on all Append() calls
I20250815 13:54:14.163079 426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=13 ignored=0} mutations{seen=21 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 13:54:14.163797 426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13: Bootstrap complete.
I20250815 13:54:14.176831 426 raft_consensus.cc:357] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993462eb9d054343874cf1ca224fbc13" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 35951 } }
I20250815 13:54:14.177438 426 raft_consensus.cc:738] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993462eb9d054343874cf1ca224fbc13, State: Initialized, Role: FOLLOWER
I20250815 13:54:14.178333 426 consensus_queue.cc:260] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993462eb9d054343874cf1ca224fbc13" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 35951 } }
I20250815 13:54:14.178836 426 raft_consensus.cc:397] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [term 3 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 13:54:14.179154 426 raft_consensus.cc:491] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [term 3 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 13:54:14.179502 426 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [term 3 FOLLOWER]: Advancing to term 4
I20250815 13:54:14.185009 426 raft_consensus.cc:513] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993462eb9d054343874cf1ca224fbc13" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 35951 } }
I20250815 13:54:14.185745 426 leader_election.cc:304] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 993462eb9d054343874cf1ca224fbc13; no voters:
I20250815 13:54:14.186931 426 leader_election.cc:290] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [CANDIDATE]: Term 4 election: Requested vote from peers
I20250815 13:54:14.187398 4309 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [term 4 FOLLOWER]: Leader election won for term 4
I20250815 13:54:14.188926 4309 raft_consensus.cc:695] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [term 4 LEADER]: Becoming Leader. State: Replica: 993462eb9d054343874cf1ca224fbc13, State: Running, Role: LEADER
I20250815 13:54:14.189836 4309 consensus_queue.cc:237] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993462eb9d054343874cf1ca224fbc13" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 35951 } }
I20250815 13:54:14.196964 4310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 leader_uuid: "993462eb9d054343874cf1ca224fbc13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993462eb9d054343874cf1ca224fbc13" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 35951 } } }
I20250815 13:54:14.197523 4310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [sys.catalog]: This master's current role is: LEADER
I20250815 13:54:14.198527 4311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 993462eb9d054343874cf1ca224fbc13. Latest consensus state: current_term: 4 leader_uuid: "993462eb9d054343874cf1ca224fbc13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993462eb9d054343874cf1ca224fbc13" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 35951 } } }
I20250815 13:54:14.199199 4311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [sys.catalog]: This master's current role is: LEADER
I20250815 13:54:14.228969 426 tablet_replica.cc:331] stopping tablet replica
I20250815 13:54:14.229619 426 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [term 4 LEADER]: Raft consensus shutting down.
I20250815 13:54:14.230088 426 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250815 13:54:14.233028 426 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250815 13:54:14.233572 426 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250815 13:54:14.265694 426 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
W20250815 13:54:14.583354 4054 heartbeater.cc:646] Failed to heartbeat to 127.0.106.190:35951 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.106.190:35951: connect: Connection refused (error 111)
W20250815 13:54:14.584419 3915 heartbeater.cc:646] Failed to heartbeat to 127.0.106.190:35951 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.106.190:35951: connect: Connection refused (error 111)
I20250815 13:54:19.404485 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 3759
I20250815 13:54:19.430251 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 3919
I20250815 13:54:19.458760 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 4058
I20250815 13:54:19.488628 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.190:35951
--webserver_interface=127.0.106.190
--webserver_port=36725
--builtin_ntp_servers=127.0.106.148:39749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.106.190:35951 with env {}
W20250815 13:54:19.826241 4386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:54:19.826977 4386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:54:19.827517 4386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:54:19.861115 4386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 13:54:19.861485 4386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:54:19.861814 4386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 13:54:19.862069 4386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 13:54:19.900652 4386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:39749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.106.190:35951
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.190:35951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.106.190
--webserver_port=36725
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:54:19.902346 4386 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:54:19.904325 4386 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:54:19.916697 4392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:54:19.917909 4393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:54:21.119220 4394 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1196 milliseconds
W20250815 13:54:21.121300 4395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:54:21.123329 4386 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.206s user 0.004s sys 0.007s
W20250815 13:54:21.123622 4386 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.206s user 0.004s sys 0.007s
I20250815 13:54:21.123876 4386 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 13:54:21.125012 4386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:54:21.128157 4386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:54:21.129705 4386 hybrid_clock.cc:648] HybridClock initialized: now 1755266061129673 us; error 43 us; skew 500 ppm
I20250815 13:54:21.130594 4386 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:54:21.139011 4386 webserver.cc:480] Webserver started at http://127.0.106.190:36725/ using document root <none> and password file <none>
I20250815 13:54:21.140034 4386 fs_manager.cc:362] Metadata directory not provided
I20250815 13:54:21.140247 4386 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:54:21.148864 4386 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.000s sys 0.004s
I20250815 13:54:21.154501 4402 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:54:21.156085 4386 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20250815 13:54:21.156545 4386 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
uuid: "993462eb9d054343874cf1ca224fbc13"
format_stamp: "Formatted at 2025-08-15 13:53:50 on dist-test-slave-kt02"
I20250815 13:54:21.158867 4386 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:54:21.228477 4386 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:54:21.229914 4386 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:54:21.230348 4386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:54:21.303035 4386 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.190:35951
I20250815 13:54:21.303174 4453 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.190:35951 every 8 connection(s)
I20250815 13:54:21.306138 4386 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250815 13:54:21.306936 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 4386
I20250815 13:54:21.309038 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.129:33117
--local_ip_for_outbound_sockets=127.0.106.129
--tserver_master_addrs=127.0.106.190:35951
--webserver_port=44961
--webserver_interface=127.0.106.129
--builtin_ntp_servers=127.0.106.148:39749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 13:54:21.323920 4454 sys_catalog.cc:263] Verifying existing consensus state
I20250815 13:54:21.329222 4454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13: Bootstrap starting.
I20250815 13:54:21.339569 4454 log.cc:826] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13: Log is configured to *not* fsync() on all Append() calls
I20250815 13:54:21.417590 4454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=15 ignored=0} mutations{seen=23 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 13:54:21.418363 4454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13: Bootstrap complete.
I20250815 13:54:21.436960 4454 raft_consensus.cc:357] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993462eb9d054343874cf1ca224fbc13" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 35951 } }
I20250815 13:54:21.439070 4454 raft_consensus.cc:738] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993462eb9d054343874cf1ca224fbc13, State: Initialized, Role: FOLLOWER
I20250815 13:54:21.439836 4454 consensus_queue.cc:260] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993462eb9d054343874cf1ca224fbc13" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 35951 } }
I20250815 13:54:21.440325 4454 raft_consensus.cc:397] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [term 5 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 13:54:21.440594 4454 raft_consensus.cc:491] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [term 5 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 13:54:21.440876 4454 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [term 5 FOLLOWER]: Advancing to term 6
I20250815 13:54:21.446050 4454 raft_consensus.cc:513] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993462eb9d054343874cf1ca224fbc13" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 35951 } }
I20250815 13:54:21.446743 4454 leader_election.cc:304] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 993462eb9d054343874cf1ca224fbc13; no voters:
I20250815 13:54:21.448724 4454 leader_election.cc:290] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [CANDIDATE]: Term 6 election: Requested vote from peers
I20250815 13:54:21.449220 4458 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [term 6 FOLLOWER]: Leader election won for term 6
I20250815 13:54:21.452540 4458 raft_consensus.cc:695] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [term 6 LEADER]: Becoming Leader. State: Replica: 993462eb9d054343874cf1ca224fbc13, State: Running, Role: LEADER
I20250815 13:54:21.453708 4454 sys_catalog.cc:564] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [sys.catalog]: configured and running, proceeding with master startup.
I20250815 13:54:21.453495 4458 consensus_queue.cc:237] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 6, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993462eb9d054343874cf1ca224fbc13" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 35951 } }
I20250815 13:54:21.462118 4460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 993462eb9d054343874cf1ca224fbc13. Latest consensus state: current_term: 6 leader_uuid: "993462eb9d054343874cf1ca224fbc13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993462eb9d054343874cf1ca224fbc13" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 35951 } } }
I20250815 13:54:21.462956 4460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [sys.catalog]: This master's current role is: LEADER
I20250815 13:54:21.476197 4459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 leader_uuid: "993462eb9d054343874cf1ca224fbc13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993462eb9d054343874cf1ca224fbc13" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 35951 } } }
I20250815 13:54:21.476985 4459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13 [sys.catalog]: This master's current role is: LEADER
I20250815 13:54:21.478633 4464 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 13:54:21.492408 4464 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=3632567ed8a74011837e795a7dbc0e18]
I20250815 13:54:21.494102 4464 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=dfe6d186a7dd4b66afc85b372467b78d]
I20250815 13:54:21.495708 4464 catalog_manager.cc:671] Loaded metadata for table TestTable [id=f593be896db44a1a99cfc5fd1ec98fd1]
I20250815 13:54:21.504227 4464 tablet_loader.cc:96] loaded metadata for tablet 672f8196d0cb413cb7c514171eb967fd (table TestTable2 [id=dfe6d186a7dd4b66afc85b372467b78d])
I20250815 13:54:21.505791 4464 tablet_loader.cc:96] loaded metadata for tablet 9b39e98a33744b249184945f4451802d (table TestTable [id=f593be896db44a1a99cfc5fd1ec98fd1])
I20250815 13:54:21.507299 4464 tablet_loader.cc:96] loaded metadata for tablet c4cc2a5449c34c269c21d85e6cb450c0 (table TestTable1 [id=3632567ed8a74011837e795a7dbc0e18])
I20250815 13:54:21.509692 4464 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 13:54:21.516571 4464 catalog_manager.cc:1261] Loaded cluster ID: 115e3d3900b64ba7a9bb5cbf73264c90
I20250815 13:54:21.516942 4464 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 13:54:21.527040 4464 catalog_manager.cc:1506] Loading token signing keys...
I20250815 13:54:21.531440 4464 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 993462eb9d054343874cf1ca224fbc13: Loaded TSK: 0
I20250815 13:54:21.532687 4464 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250815 13:54:21.693014 4456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:54:21.693627 4456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:54:21.694155 4456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:54:21.728811 4456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:54:21.729836 4456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.129
I20250815 13:54:21.768931 4456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:39749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.129:33117
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.106.129
--webserver_port=44961
--tserver_master_addrs=127.0.106.190:35951
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.129
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:54:21.770584 4456 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:54:21.772645 4456 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:54:21.787580 4481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:54:21.789641 4482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:54:23.152407 4484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:54:23.155747 4456 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.368s user 0.458s sys 0.877s
W20250815 13:54:23.156250 4456 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.369s user 0.458s sys 0.878s
W20250815 13:54:23.158509 4483 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1363 milliseconds
I20250815 13:54:23.158536 4456 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 13:54:23.160300 4456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:54:23.163300 4456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:54:23.164825 4456 hybrid_clock.cc:648] HybridClock initialized: now 1755266063164764 us; error 84 us; skew 500 ppm
I20250815 13:54:23.166018 4456 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:54:23.175493 4456 webserver.cc:480] Webserver started at http://127.0.106.129:44961/ using document root <none> and password file <none>
I20250815 13:54:23.176914 4456 fs_manager.cc:362] Metadata directory not provided
I20250815 13:54:23.177260 4456 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:54:23.189993 4456 fs_manager.cc:714] Time spent opening directory manager: real 0.008s user 0.005s sys 0.003s
I20250815 13:54:23.198237 4492 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:54:23.199960 4456 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.005s sys 0.001s
I20250815 13:54:23.200608 4456 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f86092b7fc7a46dfb843a939dd4fbc6a"
format_stamp: "Formatted at 2025-08-15 13:53:53 on dist-test-slave-kt02"
I20250815 13:54:23.203930 4456 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:54:23.292169 4456 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:54:23.293682 4456 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:54:23.294114 4456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:54:23.296829 4456 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 13:54:23.303733 4499 ts_tablet_manager.cc:542] Loading tablet metadata (0/3 complete)
I20250815 13:54:23.332192 4456 ts_tablet_manager.cc:579] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20250815 13:54:23.332527 4456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.031s user 0.000s sys 0.003s
I20250815 13:54:23.332868 4456 ts_tablet_manager.cc:594] Registering tablets (0/3 complete)
I20250815 13:54:23.338837 4499 tablet_bootstrap.cc:492] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a: Bootstrap starting.
I20250815 13:54:23.344933 4456 ts_tablet_manager.cc:610] Registered 3 tablets
I20250815 13:54:23.345175 4456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.012s user 0.011s sys 0.000s
I20250815 13:54:23.405596 4499 log.cc:826] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a: Log is configured to *not* fsync() on all Append() calls
I20250815 13:54:23.546909 4499 tablet_bootstrap.cc:492] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 13:54:23.548329 4499 tablet_bootstrap.cc:492] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a: Bootstrap complete.
I20250815 13:54:23.550446 4499 ts_tablet_manager.cc:1397] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a: Time spent bootstrapping tablet: real 0.212s user 0.166s sys 0.041s
I20250815 13:54:23.560923 4456 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.129:33117
I20250815 13:54:23.561089 4606 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.129:33117 every 8 connection(s)
I20250815 13:54:23.564870 4456 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250815 13:54:23.568624 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 4456
I20250815 13:54:23.570889 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.130:46037
--local_ip_for_outbound_sockets=127.0.106.130
--tserver_master_addrs=127.0.106.190:35951
--webserver_port=34781
--webserver_interface=127.0.106.130
--builtin_ntp_servers=127.0.106.148:39749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 13:54:23.570372 4499 raft_consensus.cc:357] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } } peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } attrs { promote: false } }
I20250815 13:54:23.573791 4499 raft_consensus.cc:738] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f86092b7fc7a46dfb843a939dd4fbc6a, State: Initialized, Role: FOLLOWER
I20250815 13:54:23.574991 4499 consensus_queue.cc:260] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } } peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } attrs { promote: false } }
I20250815 13:54:23.589172 4499 ts_tablet_manager.cc:1428] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a: Time spent starting tablet: real 0.038s user 0.024s sys 0.008s
I20250815 13:54:23.590198 4499 tablet_bootstrap.cc:492] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a: Bootstrap starting.
I20250815 13:54:23.639034 4607 heartbeater.cc:344] Connected to a master server at 127.0.106.190:35951
I20250815 13:54:23.639770 4607 heartbeater.cc:461] Registering TS with master...
I20250815 13:54:23.641608 4607 heartbeater.cc:507] Master 127.0.106.190:35951 requested a full tablet report, sending...
I20250815 13:54:23.648290 4419 ts_manager.cc:194] Registered new tserver with Master: f86092b7fc7a46dfb843a939dd4fbc6a (127.0.106.129:33117)
I20250815 13:54:23.653270 4419 catalog_manager.cc:5582] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a reported cstate change: config changed from index -1 to 13, term changed from 0 to 2, VOTER 34201ba276c14ae6bc93bb059132d83f (127.0.106.131) added, VOTER 90a13caa51354a19b2434ba2e76cf1f3 (127.0.106.130) added, VOTER f86092b7fc7a46dfb843a939dd4fbc6a (127.0.106.129) added. New cstate: current_term: 2 committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } } peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } attrs { promote: false } } }
I20250815 13:54:23.709138 4419 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.129:56639
I20250815 13:54:23.713984 4607 heartbeater.cc:499] Master 127.0.106.190:35951 was elected leader, sending a full tablet report...
I20250815 13:54:23.757333 4499 tablet_bootstrap.cc:492] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 13:54:23.758318 4499 tablet_bootstrap.cc:492] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a: Bootstrap complete.
I20250815 13:54:23.760066 4499 ts_tablet_manager.cc:1397] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a: Time spent bootstrapping tablet: real 0.170s user 0.154s sys 0.007s
I20250815 13:54:23.762529 4499 raft_consensus.cc:357] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } } peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } attrs { promote: false } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } }
I20250815 13:54:23.763281 4499 raft_consensus.cc:738] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f86092b7fc7a46dfb843a939dd4fbc6a, State: Initialized, Role: FOLLOWER
I20250815 13:54:23.763944 4499 consensus_queue.cc:260] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } } peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } attrs { promote: false } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } }
I20250815 13:54:23.765921 4499 ts_tablet_manager.cc:1428] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250815 13:54:23.766677 4499 tablet_bootstrap.cc:492] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a: Bootstrap starting.
I20250815 13:54:23.899605 4499 tablet_bootstrap.cc:492] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 13:54:23.900568 4499 tablet_bootstrap.cc:492] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a: Bootstrap complete.
I20250815 13:54:23.902041 4499 ts_tablet_manager.cc:1397] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a: Time spent bootstrapping tablet: real 0.136s user 0.115s sys 0.016s
I20250815 13:54:23.904387 4499 raft_consensus.cc:357] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } }
I20250815 13:54:23.904912 4499 raft_consensus.cc:738] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f86092b7fc7a46dfb843a939dd4fbc6a, State: Initialized, Role: FOLLOWER
I20250815 13:54:23.905511 4499 consensus_queue.cc:260] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } }
I20250815 13:54:23.906065 4499 raft_consensus.cc:397] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 13:54:23.906409 4499 raft_consensus.cc:491] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 13:54:23.906876 4499 raft_consensus.cc:3058] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 FOLLOWER]: Advancing to term 3
I20250815 13:54:23.915666 4499 raft_consensus.cc:513] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } }
I20250815 13:54:23.916615 4499 leader_election.cc:304] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f86092b7fc7a46dfb843a939dd4fbc6a; no voters:
I20250815 13:54:23.917315 4499 leader_election.cc:290] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a [CANDIDATE]: Term 3 election: Requested vote from peers
I20250815 13:54:23.917629 4608 raft_consensus.cc:2802] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a [term 3 FOLLOWER]: Leader election won for term 3
I20250815 13:54:23.928181 4608 raft_consensus.cc:695] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a [term 3 LEADER]: Becoming Leader. State: Replica: f86092b7fc7a46dfb843a939dd4fbc6a, State: Running, Role: LEADER
I20250815 13:54:23.929278 4608 consensus_queue.cc:237] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } }
I20250815 13:54:23.944706 4419 catalog_manager.cc:5582] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } health_report { overall_health: HEALTHY } } }
I20250815 13:54:23.963295 4499 ts_tablet_manager.cc:1428] T 672f8196d0cb413cb7c514171eb967fd P f86092b7fc7a46dfb843a939dd4fbc6a: Time spent starting tablet: real 0.061s user 0.056s sys 0.008s
W20250815 13:54:23.980731 4610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:54:23.981237 4610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:54:23.981737 4610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:54:24.013701 4610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:54:24.014571 4610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.130
I20250815 13:54:24.051975 4610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:39749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.130:46037
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.106.130
--webserver_port=34781
--tserver_master_addrs=127.0.106.190:35951
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.130
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:54:24.053743 4610 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:54:24.055896 4610 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:54:24.071425 4628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 13:54:24.927350 4632 raft_consensus.cc:491] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 13:54:24.928030 4632 raft_consensus.cc:513] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } } peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } attrs { promote: false } }
I20250815 13:54:24.932665 4632 leader_election.cc:290] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 90a13caa51354a19b2434ba2e76cf1f3 (127.0.106.130:46037), 34201ba276c14ae6bc93bb059132d83f (127.0.106.131:36587)
W20250815 13:54:24.941290 4495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.106.130:46037: connect: Connection refused (error 111)
W20250815 13:54:24.947115 4495 leader_election.cc:336] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 90a13caa51354a19b2434ba2e76cf1f3 (127.0.106.130:46037): Network error: Client connection negotiation failed: client connection to 127.0.106.130:46037: connect: Connection refused (error 111)
W20250815 13:54:24.947866 4493 leader_election.cc:336] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 34201ba276c14ae6bc93bb059132d83f (127.0.106.131:36587): Network error: Client connection negotiation failed: client connection to 127.0.106.131:36587: connect: Connection refused (error 111)
I20250815 13:54:24.948335 4493 leader_election.cc:304] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f86092b7fc7a46dfb843a939dd4fbc6a; no voters: 34201ba276c14ae6bc93bb059132d83f, 90a13caa51354a19b2434ba2e76cf1f3
I20250815 13:54:24.949262 4632 raft_consensus.cc:2747] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250815 13:54:25.221316 4632 raft_consensus.cc:491] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 13:54:25.221935 4632 raft_consensus.cc:513] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } } peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } attrs { promote: false } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } }
I20250815 13:54:25.224073 4632 leader_election.cc:290] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 34201ba276c14ae6bc93bb059132d83f (127.0.106.131:36587), 90a13caa51354a19b2434ba2e76cf1f3 (127.0.106.130:46037)
W20250815 13:54:25.229317 4493 leader_election.cc:336] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 34201ba276c14ae6bc93bb059132d83f (127.0.106.131:36587): Network error: Client connection negotiation failed: client connection to 127.0.106.131:36587: connect: Connection refused (error 111)
W20250815 13:54:25.229967 4495 leader_election.cc:336] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 90a13caa51354a19b2434ba2e76cf1f3 (127.0.106.130:46037): Network error: Client connection negotiation failed: client connection to 127.0.106.130:46037: connect: Connection refused (error 111)
I20250815 13:54:25.230455 4495 leader_election.cc:304] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f86092b7fc7a46dfb843a939dd4fbc6a; no voters: 34201ba276c14ae6bc93bb059132d83f, 90a13caa51354a19b2434ba2e76cf1f3
I20250815 13:54:25.231292 4632 raft_consensus.cc:2747] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250815 13:54:24.072572 4629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:54:25.251437 4610 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.181s user 0.496s sys 0.682s
W20250815 13:54:25.251771 4610 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.182s user 0.496s sys 0.682s
I20250815 13:54:25.253265 4610 server_base.cc:1047] running on GCE node
W20250815 13:54:25.255532 4637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 13:54:25.256867 4610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:54:25.259241 4610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:54:25.260838 4610 hybrid_clock.cc:648] HybridClock initialized: now 1755266065260777 us; error 74 us; skew 500 ppm
I20250815 13:54:25.261731 4610 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:54:25.269773 4610 webserver.cc:480] Webserver started at http://127.0.106.130:34781/ using document root <none> and password file <none>
I20250815 13:54:25.270692 4610 fs_manager.cc:362] Metadata directory not provided
I20250815 13:54:25.270913 4610 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:54:25.279727 4610 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20250815 13:54:25.285759 4642 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:54:25.286903 4610 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.002s
I20250815 13:54:25.287252 4610 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "90a13caa51354a19b2434ba2e76cf1f3"
format_stamp: "Formatted at 2025-08-15 13:53:55 on dist-test-slave-kt02"
I20250815 13:54:25.289322 4610 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:54:25.370373 4610 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:54:25.371956 4610 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:54:25.372427 4610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:54:25.375085 4610 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 13:54:25.381644 4649 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250815 13:54:25.395735 4610 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250815 13:54:25.396183 4610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.002s sys 0.000s
I20250815 13:54:25.396498 4610 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250815 13:54:25.402738 4649 tablet_bootstrap.cc:492] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3: Bootstrap starting.
I20250815 13:54:25.405349 4610 ts_tablet_manager.cc:610] Registered 2 tablets
I20250815 13:54:25.405609 4610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.009s user 0.007s sys 0.000s
I20250815 13:54:25.474159 4649 log.cc:826] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3: Log is configured to *not* fsync() on all Append() calls
I20250815 13:54:25.593920 4610 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.130:46037
I20250815 13:54:25.594126 4756 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.130:46037 every 8 connection(s)
I20250815 13:54:25.597856 4610 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250815 13:54:25.604012 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 4610
I20250815 13:54:25.606016 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.131:36587
--local_ip_for_outbound_sockets=127.0.106.131
--tserver_master_addrs=127.0.106.190:35951
--webserver_port=36245
--webserver_interface=127.0.106.131
--builtin_ntp_servers=127.0.106.148:39749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 13:54:25.626127 4649 tablet_bootstrap.cc:492] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 13:54:25.627270 4649 tablet_bootstrap.cc:492] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3: Bootstrap complete.
I20250815 13:54:25.628805 4649 ts_tablet_manager.cc:1397] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3: Time spent bootstrapping tablet: real 0.227s user 0.163s sys 0.060s
I20250815 13:54:25.628746 4757 heartbeater.cc:344] Connected to a master server at 127.0.106.190:35951
I20250815 13:54:25.629503 4757 heartbeater.cc:461] Registering TS with master...
I20250815 13:54:25.631032 4757 heartbeater.cc:507] Master 127.0.106.190:35951 requested a full tablet report, sending...
I20250815 13:54:25.635983 4419 ts_manager.cc:194] Registered new tserver with Master: 90a13caa51354a19b2434ba2e76cf1f3 (127.0.106.130:46037)
I20250815 13:54:25.640399 4419 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.130:39785
I20250815 13:54:25.643733 4649 raft_consensus.cc:357] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } } peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } attrs { promote: false } }
I20250815 13:54:25.645977 4649 raft_consensus.cc:738] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a13caa51354a19b2434ba2e76cf1f3, State: Initialized, Role: FOLLOWER
I20250815 13:54:25.646742 4649 consensus_queue.cc:260] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } } peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } attrs { promote: false } }
I20250815 13:54:25.649895 4757 heartbeater.cc:499] Master 127.0.106.190:35951 was elected leader, sending a full tablet report...
I20250815 13:54:25.650614 4649 ts_tablet_manager.cc:1428] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3: Time spent starting tablet: real 0.021s user 0.020s sys 0.000s
I20250815 13:54:25.651500 4649 tablet_bootstrap.cc:492] T c4cc2a5449c34c269c21d85e6cb450c0 P 90a13caa51354a19b2434ba2e76cf1f3: Bootstrap starting.
I20250815 13:54:25.755335 4649 tablet_bootstrap.cc:492] T c4cc2a5449c34c269c21d85e6cb450c0 P 90a13caa51354a19b2434ba2e76cf1f3: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 13:54:25.756194 4649 tablet_bootstrap.cc:492] T c4cc2a5449c34c269c21d85e6cb450c0 P 90a13caa51354a19b2434ba2e76cf1f3: Bootstrap complete.
I20250815 13:54:25.757404 4649 ts_tablet_manager.cc:1397] T c4cc2a5449c34c269c21d85e6cb450c0 P 90a13caa51354a19b2434ba2e76cf1f3: Time spent bootstrapping tablet: real 0.106s user 0.084s sys 0.019s
I20250815 13:54:25.759227 4649 raft_consensus.cc:357] T c4cc2a5449c34c269c21d85e6cb450c0 P 90a13caa51354a19b2434ba2e76cf1f3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } } peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } attrs { promote: false } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } }
I20250815 13:54:25.759691 4649 raft_consensus.cc:738] T c4cc2a5449c34c269c21d85e6cb450c0 P 90a13caa51354a19b2434ba2e76cf1f3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a13caa51354a19b2434ba2e76cf1f3, State: Initialized, Role: FOLLOWER
I20250815 13:54:25.760248 4649 consensus_queue.cc:260] T c4cc2a5449c34c269c21d85e6cb450c0 P 90a13caa51354a19b2434ba2e76cf1f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } } peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } attrs { promote: false } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } }
I20250815 13:54:25.761894 4649 ts_tablet_manager.cc:1428] T c4cc2a5449c34c269c21d85e6cb450c0 P 90a13caa51354a19b2434ba2e76cf1f3: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20250815 13:54:25.919871 4761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:54:25.920364 4761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:54:25.920858 4761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:54:25.952466 4761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:54:25.953303 4761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.131
I20250815 13:54:25.990468 4761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:39749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.131:36587
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.106.131
--webserver_port=36245
--tserver_master_addrs=127.0.106.190:35951
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.131
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:54:25.991840 4761 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:54:25.993417 4761 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:54:26.006544 4769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 13:54:26.908721 4775 raft_consensus.cc:491] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 13:54:26.909339 4775 raft_consensus.cc:513] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } } peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } attrs { promote: false } }
I20250815 13:54:26.911753 4775 leader_election.cc:290] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 90a13caa51354a19b2434ba2e76cf1f3 (127.0.106.130:46037), 34201ba276c14ae6bc93bb059132d83f (127.0.106.131:36587)
W20250815 13:54:26.935366 4493 leader_election.cc:336] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 34201ba276c14ae6bc93bb059132d83f (127.0.106.131:36587): Network error: Client connection negotiation failed: client connection to 127.0.106.131:36587: connect: Connection refused (error 111)
I20250815 13:54:26.937405 4712 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9b39e98a33744b249184945f4451802d" candidate_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "90a13caa51354a19b2434ba2e76cf1f3" is_pre_election: true
I20250815 13:54:26.938112 4712 raft_consensus.cc:2466] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f86092b7fc7a46dfb843a939dd4fbc6a in term 2.
I20250815 13:54:26.939379 4495 leader_election.cc:304] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 90a13caa51354a19b2434ba2e76cf1f3, f86092b7fc7a46dfb843a939dd4fbc6a; no voters: 34201ba276c14ae6bc93bb059132d83f
I20250815 13:54:26.940021 4775 raft_consensus.cc:2802] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250815 13:54:26.940320 4775 raft_consensus.cc:491] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 13:54:26.940574 4775 raft_consensus.cc:3058] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 FOLLOWER]: Advancing to term 3
I20250815 13:54:26.945061 4775 raft_consensus.cc:513] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } } peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } attrs { promote: false } }
I20250815 13:54:26.946559 4775 leader_election.cc:290] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [CANDIDATE]: Term 3 election: Requested vote from peers 90a13caa51354a19b2434ba2e76cf1f3 (127.0.106.130:46037), 34201ba276c14ae6bc93bb059132d83f (127.0.106.131:36587)
I20250815 13:54:26.947449 4712 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9b39e98a33744b249184945f4451802d" candidate_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "90a13caa51354a19b2434ba2e76cf1f3"
I20250815 13:54:26.947983 4712 raft_consensus.cc:3058] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [term 2 FOLLOWER]: Advancing to term 3
W20250815 13:54:26.951431 4493 leader_election.cc:336] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 34201ba276c14ae6bc93bb059132d83f (127.0.106.131:36587): Network error: Client connection negotiation failed: client connection to 127.0.106.131:36587: connect: Connection refused (error 111)
I20250815 13:54:26.955870 4712 raft_consensus.cc:2466] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f86092b7fc7a46dfb843a939dd4fbc6a in term 3.
I20250815 13:54:26.956697 4495 leader_election.cc:304] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 90a13caa51354a19b2434ba2e76cf1f3, f86092b7fc7a46dfb843a939dd4fbc6a; no voters: 34201ba276c14ae6bc93bb059132d83f
I20250815 13:54:26.957269 4775 raft_consensus.cc:2802] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [term 3 FOLLOWER]: Leader election won for term 3
I20250815 13:54:26.958392 4775 raft_consensus.cc:695] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [term 3 LEADER]: Becoming Leader. State: Replica: f86092b7fc7a46dfb843a939dd4fbc6a, State: Running, Role: LEADER
I20250815 13:54:26.959235 4775 consensus_queue.cc:237] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } } peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } attrs { promote: false } }
I20250815 13:54:26.966864 4419 catalog_manager.cc:5582] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a reported cstate change: term changed from 2 to 3, leader changed from <none> to f86092b7fc7a46dfb843a939dd4fbc6a (127.0.106.129). New cstate: current_term: 3 leader_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250815 13:54:26.006528 4770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:54:26.011339 4772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:54:27.177810 4771 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1167 milliseconds
I20250815 13:54:27.177971 4761 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 13:54:27.179247 4761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:54:27.181885 4761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:54:27.183348 4761 hybrid_clock.cc:648] HybridClock initialized: now 1755266067183298 us; error 66 us; skew 500 ppm
I20250815 13:54:27.184165 4761 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:54:27.190708 4761 webserver.cc:480] Webserver started at http://127.0.106.131:36245/ using document root <none> and password file <none>
I20250815 13:54:27.191654 4761 fs_manager.cc:362] Metadata directory not provided
I20250815 13:54:27.191869 4761 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:54:27.199955 4761 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.001s
I20250815 13:54:27.205111 4790 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:54:27.206313 4761 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250815 13:54:27.206687 4761 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "34201ba276c14ae6bc93bb059132d83f"
format_stamp: "Formatted at 2025-08-15 13:53:57 on dist-test-slave-kt02"
I20250815 13:54:27.208763 4761 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:54:27.260525 4761 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:54:27.261960 4761 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:54:27.262377 4761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:54:27.264959 4761 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 13:54:27.270933 4797 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250815 13:54:27.285082 4761 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250815 13:54:27.285380 4761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.002s sys 0.000s
I20250815 13:54:27.285693 4761 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250815 13:54:27.287632 4775 raft_consensus.cc:491] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 13:54:27.288095 4775 raft_consensus.cc:513] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } } peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } attrs { promote: false } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } }
I20250815 13:54:27.289624 4775 leader_election.cc:290] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 34201ba276c14ae6bc93bb059132d83f (127.0.106.131:36587), 90a13caa51354a19b2434ba2e76cf1f3 (127.0.106.130:46037)
I20250815 13:54:27.290797 4712 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c4cc2a5449c34c269c21d85e6cb450c0" candidate_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "90a13caa51354a19b2434ba2e76cf1f3" is_pre_election: true
I20250815 13:54:27.291388 4712 raft_consensus.cc:2466] T c4cc2a5449c34c269c21d85e6cb450c0 P 90a13caa51354a19b2434ba2e76cf1f3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f86092b7fc7a46dfb843a939dd4fbc6a in term 2.
I20250815 13:54:27.292696 4797 tablet_bootstrap.cc:492] T 9b39e98a33744b249184945f4451802d P 34201ba276c14ae6bc93bb059132d83f: Bootstrap starting.
I20250815 13:54:27.292613 4495 leader_election.cc:304] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90a13caa51354a19b2434ba2e76cf1f3, f86092b7fc7a46dfb843a939dd4fbc6a; no voters:
I20250815 13:54:27.293522 4775 raft_consensus.cc:2802] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250815 13:54:27.293843 4775 raft_consensus.cc:491] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250815 13:54:27.294100 4493 leader_election.cc:336] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 34201ba276c14ae6bc93bb059132d83f (127.0.106.131:36587): Network error: Client connection negotiation failed: client connection to 127.0.106.131:36587: connect: Connection refused (error 111)
I20250815 13:54:27.294181 4775 raft_consensus.cc:3058] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a [term 2 FOLLOWER]: Advancing to term 3
I20250815 13:54:27.296475 4761 ts_tablet_manager.cc:610] Registered 2 tablets
I20250815 13:54:27.296713 4761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.011s user 0.011s sys 0.000s
I20250815 13:54:27.301369 4775 raft_consensus.cc:513] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } } peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } attrs { promote: false } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } }
W20250815 13:54:27.308171 4493 leader_election.cc:336] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 34201ba276c14ae6bc93bb059132d83f (127.0.106.131:36587): Network error: Client connection negotiation failed: client connection to 127.0.106.131:36587: connect: Connection refused (error 111)
I20250815 13:54:27.311664 4712 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c4cc2a5449c34c269c21d85e6cb450c0" candidate_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "90a13caa51354a19b2434ba2e76cf1f3"
I20250815 13:54:27.312310 4712 raft_consensus.cc:3058] T c4cc2a5449c34c269c21d85e6cb450c0 P 90a13caa51354a19b2434ba2e76cf1f3 [term 2 FOLLOWER]: Advancing to term 3
I20250815 13:54:27.319811 4712 raft_consensus.cc:2466] T c4cc2a5449c34c269c21d85e6cb450c0 P 90a13caa51354a19b2434ba2e76cf1f3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f86092b7fc7a46dfb843a939dd4fbc6a in term 3.
I20250815 13:54:27.321246 4495 leader_election.cc:304] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 90a13caa51354a19b2434ba2e76cf1f3, f86092b7fc7a46dfb843a939dd4fbc6a; no voters: 34201ba276c14ae6bc93bb059132d83f
I20250815 13:54:27.322335 4780 raft_consensus.cc:2802] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a [term 3 FOLLOWER]: Leader election won for term 3
I20250815 13:54:27.322350 4775 leader_election.cc:290] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a [CANDIDATE]: Term 3 election: Requested vote from peers 34201ba276c14ae6bc93bb059132d83f (127.0.106.131:36587), 90a13caa51354a19b2434ba2e76cf1f3 (127.0.106.130:46037)
I20250815 13:54:27.335695 4780 raft_consensus.cc:695] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a [term 3 LEADER]: Becoming Leader. State: Replica: f86092b7fc7a46dfb843a939dd4fbc6a, State: Running, Role: LEADER
I20250815 13:54:27.336915 4780 consensus_queue.cc:237] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } } peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } attrs { promote: false } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } }
I20250815 13:54:27.353556 4419 catalog_manager.cc:5582] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a reported cstate change: term changed from 2 to 3, leader changed from 34201ba276c14ae6bc93bb059132d83f (127.0.106.131) to f86092b7fc7a46dfb843a939dd4fbc6a (127.0.106.129). New cstate: current_term: 3 leader_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250815 13:54:27.366950 4797 log.cc:826] T 9b39e98a33744b249184945f4451802d P 34201ba276c14ae6bc93bb059132d83f: Log is configured to *not* fsync() on all Append() calls
I20250815 13:54:27.393046 4712 raft_consensus.cc:1273] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [term 3 FOLLOWER]: Refusing update from remote peer f86092b7fc7a46dfb843a939dd4fbc6a: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20250815 13:54:27.395356 4780 consensus_queue.cc:1035] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20250815 13:54:27.441634 4493 consensus_peers.cc:489] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a -> Peer 34201ba276c14ae6bc93bb059132d83f (127.0.106.131:36587): Couldn't send request to peer 34201ba276c14ae6bc93bb059132d83f. Status: Network error: Client connection negotiation failed: client connection to 127.0.106.131:36587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250815 13:54:27.476984 4562 consensus_queue.cc:237] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 13, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } }
I20250815 13:54:27.482571 4712 raft_consensus.cc:1273] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [term 3 FOLLOWER]: Refusing update from remote peer f86092b7fc7a46dfb843a939dd4fbc6a: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20250815 13:54:27.484019 4802 consensus_queue.cc:1035] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20250815 13:54:27.490072 4775 raft_consensus.cc:2953] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [term 3 LEADER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER 34201ba276c14ae6bc93bb059132d83f (127.0.106.131) evicted. New config: { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } } }
I20250815 13:54:27.497525 4712 raft_consensus.cc:2953] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [term 3 FOLLOWER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER 34201ba276c14ae6bc93bb059132d83f (127.0.106.131) evicted. New config: { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } } }
I20250815 13:54:27.500910 4405 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 9b39e98a33744b249184945f4451802d with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250815 13:54:27.504273 4419 catalog_manager.cc:5582] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a reported cstate change: config changed from index 13 to 15, VOTER 34201ba276c14ae6bc93bb059132d83f (127.0.106.131) evicted. New cstate: current_term: 3 leader_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" committed_config { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250815 13:54:27.513875 4419 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet 9b39e98a33744b249184945f4451802d on TS 34201ba276c14ae6bc93bb059132d83f: Not found: failed to reset TS proxy: Could not find TS for UUID 34201ba276c14ae6bc93bb059132d83f
I20250815 13:54:27.517835 4562 consensus_queue.cc:237] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 3.15, Last appended by leader: 13, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } }
I20250815 13:54:27.520480 4802 raft_consensus.cc:2953] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a [term 3 LEADER]: Committing config change with OpId 3.16: config changed from index 15 to 16, VOTER 90a13caa51354a19b2434ba2e76cf1f3 (127.0.106.130) evicted. New config: { opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } } }
I20250815 13:54:27.525417 4797 tablet_bootstrap.cc:492] T 9b39e98a33744b249184945f4451802d P 34201ba276c14ae6bc93bb059132d83f: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 13:54:27.526666 4797 tablet_bootstrap.cc:492] T 9b39e98a33744b249184945f4451802d P 34201ba276c14ae6bc93bb059132d83f: Bootstrap complete.
I20250815 13:54:27.527873 4405 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 9b39e98a33744b249184945f4451802d with cas_config_opid_index 15: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250815 13:54:27.528774 4797 ts_tablet_manager.cc:1397] T 9b39e98a33744b249184945f4451802d P 34201ba276c14ae6bc93bb059132d83f: Time spent bootstrapping tablet: real 0.237s user 0.201s sys 0.029s
I20250815 13:54:27.531425 4419 catalog_manager.cc:5582] T 9b39e98a33744b249184945f4451802d P f86092b7fc7a46dfb843a939dd4fbc6a reported cstate change: config changed from index 15 to 16, VOTER 90a13caa51354a19b2434ba2e76cf1f3 (127.0.106.130) evicted. New cstate: current_term: 3 leader_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" committed_config { opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250815 13:54:27.554072 4692 tablet_service.cc:1515] Processing DeleteTablet for tablet 9b39e98a33744b249184945f4451802d with delete_type TABLET_DATA_TOMBSTONED (TS 90a13caa51354a19b2434ba2e76cf1f3 not found in new config with opid_index 16) from {username='slave'} at 127.0.0.1:35010
I20250815 13:54:27.554190 4797 raft_consensus.cc:357] T 9b39e98a33744b249184945f4451802d P 34201ba276c14ae6bc93bb059132d83f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } } peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } attrs { promote: false } }
I20250815 13:54:27.558547 4797 raft_consensus.cc:738] T 9b39e98a33744b249184945f4451802d P 34201ba276c14ae6bc93bb059132d83f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34201ba276c14ae6bc93bb059132d83f, State: Initialized, Role: FOLLOWER
I20250815 13:54:27.560566 4797 consensus_queue.cc:260] T 9b39e98a33744b249184945f4451802d P 34201ba276c14ae6bc93bb059132d83f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } } peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } attrs { promote: false } }
I20250815 13:54:27.556993 4881 tablet_replica.cc:331] stopping tablet replica
I20250815 13:54:27.563709 4881 raft_consensus.cc:2241] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250815 13:54:27.564388 4881 raft_consensus.cc:2270] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3 [term 3 FOLLOWER]: Raft consensus is shut down!
W20250815 13:54:27.566529 4404 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 9b39e98a33744b249184945f4451802d on TS 34201ba276c14ae6bc93bb059132d83f failed: Not found: failed to reset TS proxy: Could not find TS for UUID 34201ba276c14ae6bc93bb059132d83f
I20250815 13:54:27.568403 4881 ts_tablet_manager.cc:1905] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250815 13:54:27.572475 4797 ts_tablet_manager.cc:1428] T 9b39e98a33744b249184945f4451802d P 34201ba276c14ae6bc93bb059132d83f: Time spent starting tablet: real 0.043s user 0.034s sys 0.004s
I20250815 13:54:27.573650 4797 tablet_bootstrap.cc:492] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f: Bootstrap starting.
I20250815 13:54:27.585354 4881 ts_tablet_manager.cc:1918] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.15
I20250815 13:54:27.585973 4881 log.cc:1199] T 9b39e98a33744b249184945f4451802d P 90a13caa51354a19b2434ba2e76cf1f3: Deleting WAL directory at /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal/wals/9b39e98a33744b249184945f4451802d
I20250815 13:54:27.587823 4405 catalog_manager.cc:4928] TS 90a13caa51354a19b2434ba2e76cf1f3 (127.0.106.130:46037): tablet 9b39e98a33744b249184945f4451802d (table TestTable [id=f593be896db44a1a99cfc5fd1ec98fd1]) successfully deleted
I20250815 13:54:27.678718 4761 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.131:36587
I20250815 13:54:27.678994 4916 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.131:36587 every 8 connection(s)
I20250815 13:54:27.681867 4761 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250815 13:54:27.682840 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 4761
I20250815 13:54:27.707875 4797 tablet_bootstrap.cc:492] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 13:54:27.708811 4797 tablet_bootstrap.cc:492] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f: Bootstrap complete.
I20250815 13:54:27.710040 4797 ts_tablet_manager.cc:1397] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f: Time spent bootstrapping tablet: real 0.137s user 0.110s sys 0.020s
I20250815 13:54:27.712026 4797 raft_consensus.cc:357] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } } peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } attrs { promote: false } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } }
I20250815 13:54:27.712515 4797 raft_consensus.cc:738] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34201ba276c14ae6bc93bb059132d83f, State: Initialized, Role: FOLLOWER
I20250815 13:54:27.712996 4797 consensus_queue.cc:260] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 } } peers { permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } attrs { promote: false } } peers { permanent_uuid: "f86092b7fc7a46dfb843a939dd4fbc6a" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 33117 } attrs { promote: false } }
I20250815 13:54:27.714596 4797 ts_tablet_manager.cc:1428] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250815 13:54:27.715974 4917 heartbeater.cc:344] Connected to a master server at 127.0.106.190:35951
I20250815 13:54:27.716449 4917 heartbeater.cc:461] Registering TS with master...
I20250815 13:54:27.717561 4917 heartbeater.cc:507] Master 127.0.106.190:35951 requested a full tablet report, sending...
I20250815 13:54:27.721485 4712 raft_consensus.cc:1273] T c4cc2a5449c34c269c21d85e6cb450c0 P 90a13caa51354a19b2434ba2e76cf1f3 [term 3 FOLLOWER]: Refusing update from remote peer f86092b7fc7a46dfb843a939dd4fbc6a: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20250815 13:54:27.722234 4419 ts_manager.cc:194] Registered new tserver with Master: 34201ba276c14ae6bc93bb059132d83f (127.0.106.131:36587)
I20250815 13:54:27.722756 4802 consensus_queue.cc:1035] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a13caa51354a19b2434ba2e76cf1f3" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 46037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20250815 13:54:27.725569 426 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 13:54:27.727480 4419 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.131:44769
I20250815 13:54:27.730199 426 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 13:54:27.732262 4917 heartbeater.cc:499] Master 127.0.106.190:35951 was elected leader, sending a full tablet report...
W20250815 13:54:27.734829 426 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9b39e98a33744b249184945f4451802d: tablet_id: "9b39e98a33744b249184945f4451802d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
I20250815 13:54:27.753999 4868 raft_consensus.cc:3058] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [term 2 FOLLOWER]: Advancing to term 3
I20250815 13:54:27.760593 4868 raft_consensus.cc:1273] T c4cc2a5449c34c269c21d85e6cb450c0 P 34201ba276c14ae6bc93bb059132d83f [term 3 FOLLOWER]: Refusing update from remote peer f86092b7fc7a46dfb843a939dd4fbc6a: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20250815 13:54:27.762534 4780 consensus_queue.cc:1035] T c4cc2a5449c34c269c21d85e6cb450c0 P f86092b7fc7a46dfb843a939dd4fbc6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "34201ba276c14ae6bc93bb059132d83f" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 36587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250815 13:54:27.894868 4848 tablet_service.cc:1515] Processing DeleteTablet for tablet 9b39e98a33744b249184945f4451802d with delete_type TABLET_DATA_TOMBSTONED (TS 34201ba276c14ae6bc93bb059132d83f not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:51338
I20250815 13:54:27.899626 4927 tablet_replica.cc:331] stopping tablet replica
I20250815 13:54:27.900312 4927 raft_consensus.cc:2241] T 9b39e98a33744b249184945f4451802d P 34201ba276c14ae6bc93bb059132d83f [term 2 FOLLOWER]: Raft consensus shutting down.
I20250815 13:54:27.900738 4927 raft_consensus.cc:2270] T 9b39e98a33744b249184945f4451802d P 34201ba276c14ae6bc93bb059132d83f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250815 13:54:27.903545 4927 ts_tablet_manager.cc:1905] T 9b39e98a33744b249184945f4451802d P 34201ba276c14ae6bc93bb059132d83f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250815 13:54:27.915561 4927 ts_tablet_manager.cc:1918] T 9b39e98a33744b249184945f4451802d P 34201ba276c14ae6bc93bb059132d83f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.13
I20250815 13:54:27.915935 4927 log.cc:1199] T 9b39e98a33744b249184945f4451802d P 34201ba276c14ae6bc93bb059132d83f: Deleting WAL directory at /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal/wals/9b39e98a33744b249184945f4451802d
I20250815 13:54:27.917425 4403 catalog_manager.cc:4928] TS 34201ba276c14ae6bc93bb059132d83f (127.0.106.131:36587): tablet 9b39e98a33744b249184945f4451802d (table TestTable [id=f593be896db44a1a99cfc5fd1ec98fd1]) successfully deleted
W20250815 13:54:28.739491 426 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9b39e98a33744b249184945f4451802d: tablet_id: "9b39e98a33744b249184945f4451802d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 13:54:29.744031 426 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9b39e98a33744b249184945f4451802d: tablet_id: "9b39e98a33744b249184945f4451802d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 13:54:30.748178 426 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9b39e98a33744b249184945f4451802d: tablet_id: "9b39e98a33744b249184945f4451802d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 13:54:31.752120 426 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9b39e98a33744b249184945f4451802d: tablet_id: "9b39e98a33744b249184945f4451802d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 13:54:32.756541 426 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9b39e98a33744b249184945f4451802d: tablet_id: "9b39e98a33744b249184945f4451802d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 13:54:33.761338 426 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9b39e98a33744b249184945f4451802d: tablet_id: "9b39e98a33744b249184945f4451802d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 13:54:34.765250 426 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9b39e98a33744b249184945f4451802d: tablet_id: "9b39e98a33744b249184945f4451802d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 13:54:35.769443 426 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9b39e98a33744b249184945f4451802d: tablet_id: "9b39e98a33744b249184945f4451802d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 13:54:36.773411 426 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9b39e98a33744b249184945f4451802d: tablet_id: "9b39e98a33744b249184945f4451802d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 13:54:37.777863 426 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9b39e98a33744b249184945f4451802d: tablet_id: "9b39e98a33744b249184945f4451802d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 13:54:38.782265 426 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9b39e98a33744b249184945f4451802d: tablet_id: "9b39e98a33744b249184945f4451802d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 13:54:39.786031 426 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9b39e98a33744b249184945f4451802d: tablet_id: "9b39e98a33744b249184945f4451802d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 13:54:40.790812 426 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9b39e98a33744b249184945f4451802d: tablet_id: "9b39e98a33744b249184945f4451802d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 13:54:41.795349 426 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9b39e98a33744b249184945f4451802d: tablet_id: "9b39e98a33744b249184945f4451802d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 13:54:42.800295 426 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9b39e98a33744b249184945f4451802d: tablet_id: "9b39e98a33744b249184945f4451802d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 13:54:43.804742 426 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9b39e98a33744b249184945f4451802d: tablet_id: "9b39e98a33744b249184945f4451802d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 13:54:44.808853 426 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9b39e98a33744b249184945f4451802d: tablet_id: "9b39e98a33744b249184945f4451802d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 13:54:45.812666 426 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9b39e98a33744b249184945f4451802d: tablet_id: "9b39e98a33744b249184945f4451802d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 13:54:46.816411 426 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9b39e98a33744b249184945f4451802d: tablet_id: "9b39e98a33744b249184945f4451802d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/kudu-admin-test.cc:3914: Failure
Failed
Bad status: Not found: not all replicas of tablets comprising table TestTable are registered yet
I20250815 13:54:47.819741 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 4456
I20250815 13:54:47.844470 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 4610
I20250815 13:54:47.867491 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 4761
I20250815 13:54:47.893165 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 4386
2025-08-15T13:54:47Z chronyd exiting
I20250815 13:54:47.946007 426 test_util.cc:183] -----------------------------------------------
I20250815 13:54:47.946208 426 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755265962145449-426-0
[ FAILED ] AdminCliTest.TestRebuildTables (58536 ms)
[----------] 5 tests from AdminCliTest (125724 ms total)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest
[ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4
I20250815 13:54:47.950011 426 test_util.cc:276] Using random seed: -1146579294
I20250815 13:54:47.954648 426 ts_itest-base.cc:115] Starting cluster with:
I20250815 13:54:47.954797 426 ts_itest-base.cc:116] --------------
I20250815 13:54:47.954962 426 ts_itest-base.cc:117] 5 tablet servers
I20250815 13:54:47.955155 426 ts_itest-base.cc:118] 3 replicas per TS
I20250815 13:54:47.955297 426 ts_itest-base.cc:119] --------------
2025-08-15T13:54:47Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-15T13:54:47Z Disabled control of system clock
I20250815 13:54:48.000746 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.190:45211
--webserver_interface=127.0.106.190
--webserver_port=0
--builtin_ntp_servers=127.0.106.148:40499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.106.190:45211
--raft_prepare_replacement_before_eviction=true with env {}
W20250815 13:54:48.298309 4948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:54:48.298874 4948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:54:48.299381 4948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:54:48.330224 4948 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250815 13:54:48.330578 4948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 13:54:48.330788 4948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:54:48.331001 4948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 13:54:48.331221 4948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 13:54:48.366329 4948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:40499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.106.190:45211
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.190:45211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.106.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:54:48.367614 4948 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:54:48.369163 4948 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:54:48.381268 4954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:54:48.382938 4955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:54:48.385097 4957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:54:49.525388 4956 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1139 milliseconds
I20250815 13:54:49.525543 4948 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 13:54:49.526988 4948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:54:49.530875 4948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:54:49.532459 4948 hybrid_clock.cc:648] HybridClock initialized: now 1755266089532417 us; error 92 us; skew 500 ppm
I20250815 13:54:49.533511 4948 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:54:49.541697 4948 webserver.cc:480] Webserver started at http://127.0.106.190:37685/ using document root <none> and password file <none>
I20250815 13:54:49.542847 4948 fs_manager.cc:362] Metadata directory not provided
I20250815 13:54:49.543144 4948 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:54:49.543663 4948 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 13:54:49.549959 4948 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "36828e7f515c42d88dfe72a1939564bf"
format_stamp: "Formatted at 2025-08-15 13:54:49 on dist-test-slave-kt02"
I20250815 13:54:49.551744 4948 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "36828e7f515c42d88dfe72a1939564bf"
format_stamp: "Formatted at 2025-08-15 13:54:49 on dist-test-slave-kt02"
I20250815 13:54:49.562054 4948 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.009s sys 0.000s
I20250815 13:54:49.568686 4964 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:54:49.570329 4948 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20250815 13:54:49.570832 4948 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
uuid: "36828e7f515c42d88dfe72a1939564bf"
format_stamp: "Formatted at 2025-08-15 13:54:49 on dist-test-slave-kt02"
I20250815 13:54:49.571301 4948 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:54:49.617786 4948 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:54:49.619249 4948 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:54:49.619644 4948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:54:49.692528 4948 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.190:45211
I20250815 13:54:49.692678 5015 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.190:45211 every 8 connection(s)
I20250815 13:54:49.695339 4948 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250815 13:54:49.701488 5016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:54:49.706008 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 4948
I20250815 13:54:49.706588 426 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250815 13:54:49.727687 5016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36828e7f515c42d88dfe72a1939564bf: Bootstrap starting.
I20250815 13:54:49.734977 5016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36828e7f515c42d88dfe72a1939564bf: Neither blocks nor log segments found. Creating new log.
I20250815 13:54:49.737546 5016 log.cc:826] T 00000000000000000000000000000000 P 36828e7f515c42d88dfe72a1939564bf: Log is configured to *not* fsync() on all Append() calls
I20250815 13:54:49.743497 5016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36828e7f515c42d88dfe72a1939564bf: No bootstrap required, opened a new log
I20250815 13:54:49.763893 5016 raft_consensus.cc:357] T 00000000000000000000000000000000 P 36828e7f515c42d88dfe72a1939564bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36828e7f515c42d88dfe72a1939564bf" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 45211 } }
I20250815 13:54:49.764712 5016 raft_consensus.cc:383] T 00000000000000000000000000000000 P 36828e7f515c42d88dfe72a1939564bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 13:54:49.764920 5016 raft_consensus.cc:738] T 00000000000000000000000000000000 P 36828e7f515c42d88dfe72a1939564bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36828e7f515c42d88dfe72a1939564bf, State: Initialized, Role: FOLLOWER
I20250815 13:54:49.765820 5016 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36828e7f515c42d88dfe72a1939564bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36828e7f515c42d88dfe72a1939564bf" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 45211 } }
I20250815 13:54:49.766469 5016 raft_consensus.cc:397] T 00000000000000000000000000000000 P 36828e7f515c42d88dfe72a1939564bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 13:54:49.766831 5016 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36828e7f515c42d88dfe72a1939564bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 13:54:49.767284 5016 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 36828e7f515c42d88dfe72a1939564bf [term 0 FOLLOWER]: Advancing to term 1
I20250815 13:54:49.772110 5016 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36828e7f515c42d88dfe72a1939564bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36828e7f515c42d88dfe72a1939564bf" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 45211 } }
I20250815 13:54:49.773031 5016 leader_election.cc:304] T 00000000000000000000000000000000 P 36828e7f515c42d88dfe72a1939564bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36828e7f515c42d88dfe72a1939564bf; no voters:
I20250815 13:54:49.774920 5016 leader_election.cc:290] T 00000000000000000000000000000000 P 36828e7f515c42d88dfe72a1939564bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 13:54:49.775970 5021 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 36828e7f515c42d88dfe72a1939564bf [term 1 FOLLOWER]: Leader election won for term 1
I20250815 13:54:49.778802 5021 raft_consensus.cc:695] T 00000000000000000000000000000000 P 36828e7f515c42d88dfe72a1939564bf [term 1 LEADER]: Becoming Leader. State: Replica: 36828e7f515c42d88dfe72a1939564bf, State: Running, Role: LEADER
I20250815 13:54:49.779671 5021 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36828e7f515c42d88dfe72a1939564bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36828e7f515c42d88dfe72a1939564bf" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 45211 } }
I20250815 13:54:49.780053 5016 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36828e7f515c42d88dfe72a1939564bf [sys.catalog]: configured and running, proceeding with master startup.
I20250815 13:54:49.790966 5022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36828e7f515c42d88dfe72a1939564bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36828e7f515c42d88dfe72a1939564bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36828e7f515c42d88dfe72a1939564bf" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 45211 } } }
I20250815 13:54:49.791862 5022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36828e7f515c42d88dfe72a1939564bf [sys.catalog]: This master's current role is: LEADER
I20250815 13:54:49.792536 5023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36828e7f515c42d88dfe72a1939564bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36828e7f515c42d88dfe72a1939564bf. Latest consensus state: current_term: 1 leader_uuid: "36828e7f515c42d88dfe72a1939564bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36828e7f515c42d88dfe72a1939564bf" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 45211 } } }
I20250815 13:54:49.793488 5023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36828e7f515c42d88dfe72a1939564bf [sys.catalog]: This master's current role is: LEADER
I20250815 13:54:49.795495 5030 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 13:54:49.808513 5030 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 13:54:49.826467 5030 catalog_manager.cc:1349] Generated new cluster ID: d154450f3c2b4dafb4aa20d0c7ff24a1
I20250815 13:54:49.826777 5030 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 13:54:49.850178 5030 catalog_manager.cc:1372] Generated new certificate authority record
I20250815 13:54:49.852007 5030 catalog_manager.cc:1506] Loading token signing keys...
I20250815 13:54:49.868574 5030 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 36828e7f515c42d88dfe72a1939564bf: Generated new TSK 0
I20250815 13:54:49.869606 5030 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250815 13:54:49.897914 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.129:0
--local_ip_for_outbound_sockets=127.0.106.129
--webserver_interface=127.0.106.129
--webserver_port=0
--tserver_master_addrs=127.0.106.190:45211
--builtin_ntp_servers=127.0.106.148:40499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250815 13:54:50.232414 5040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:54:50.233048 5040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:54:50.233587 5040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:54:50.268194 5040 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250815 13:54:50.268652 5040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:54:50.269557 5040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.129
I20250815 13:54:50.308699 5040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:40499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.106.129
--webserver_port=0
--tserver_master_addrs=127.0.106.190:45211
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.129
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:54:50.310299 5040 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:54:50.312417 5040 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:54:50.327148 5046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:54:50.328670 5047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:54:51.694653 5048 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1363 milliseconds
W20250815 13:54:51.695355 5049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 13:54:51.696050 5040 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 13:54:51.700429 5040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:54:51.703519 5040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:54:51.704982 5040 hybrid_clock.cc:648] HybridClock initialized: now 1755266091704872 us; error 110 us; skew 500 ppm
I20250815 13:54:51.706139 5040 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:54:51.714159 5040 webserver.cc:480] Webserver started at http://127.0.106.129:36243/ using document root <none> and password file <none>
I20250815 13:54:51.715226 5040 fs_manager.cc:362] Metadata directory not provided
I20250815 13:54:51.715454 5040 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:54:51.715945 5040 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 13:54:51.720445 5040 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "186cf437a2684003b201f1bafca3eb37"
format_stamp: "Formatted at 2025-08-15 13:54:51 on dist-test-slave-kt02"
I20250815 13:54:51.721518 5040 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "186cf437a2684003b201f1bafca3eb37"
format_stamp: "Formatted at 2025-08-15 13:54:51 on dist-test-slave-kt02"
I20250815 13:54:51.728781 5040 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.000s sys 0.008s
I20250815 13:54:51.735015 5056 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:54:51.736156 5040 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250815 13:54:51.736457 5040 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "186cf437a2684003b201f1bafca3eb37"
format_stamp: "Formatted at 2025-08-15 13:54:51 on dist-test-slave-kt02"
I20250815 13:54:51.736774 5040 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:54:51.788527 5040 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:54:51.790005 5040 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:54:51.790424 5040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:54:51.793332 5040 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 13:54:51.797909 5040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 13:54:51.798120 5040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 13:54:51.798365 5040 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 13:54:51.798508 5040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 13:54:51.959295 5040 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.129:43939
I20250815 13:54:51.959621 5168 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.129:43939 every 8 connection(s)
I20250815 13:54:51.961860 5040 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250815 13:54:51.966636 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 5040
I20250815 13:54:51.967310 426 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250815 13:54:51.975895 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.130:0
--local_ip_for_outbound_sockets=127.0.106.130
--webserver_interface=127.0.106.130
--webserver_port=0
--tserver_master_addrs=127.0.106.190:45211
--builtin_ntp_servers=127.0.106.148:40499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250815 13:54:52.001654 5169 heartbeater.cc:344] Connected to a master server at 127.0.106.190:45211
I20250815 13:54:52.002398 5169 heartbeater.cc:461] Registering TS with master...
I20250815 13:54:52.004119 5169 heartbeater.cc:507] Master 127.0.106.190:45211 requested a full tablet report, sending...
I20250815 13:54:52.008136 4981 ts_manager.cc:194] Registered new tserver with Master: 186cf437a2684003b201f1bafca3eb37 (127.0.106.129:43939)
I20250815 13:54:52.011592 4981 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.129:59851
W20250815 13:54:52.292798 5173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:54:52.293292 5173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:54:52.293738 5173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:54:52.324587 5173 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250815 13:54:52.324959 5173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:54:52.325682 5173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.130
I20250815 13:54:52.360106 5173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:40499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.106.130
--webserver_port=0
--tserver_master_addrs=127.0.106.190:45211
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.130
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:54:52.361416 5173 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:54:52.362962 5173 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:54:52.375347 5179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 13:54:53.017102 5169 heartbeater.cc:499] Master 127.0.106.190:45211 was elected leader, sending a full tablet report...
W20250815 13:54:52.376439 5180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:54:53.517391 5181 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1139 milliseconds
W20250815 13:54:53.519446 5182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:54:53.522717 5173 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.147s user 0.004s sys 0.004s
W20250815 13:54:53.523200 5173 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.147s user 0.004s sys 0.004s
I20250815 13:54:53.523476 5173 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 13:54:53.524610 5173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:54:53.531776 5173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:54:53.533263 5173 hybrid_clock.cc:648] HybridClock initialized: now 1755266093533199 us; error 90 us; skew 500 ppm
I20250815 13:54:53.534343 5173 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:54:53.543514 5173 webserver.cc:480] Webserver started at http://127.0.106.130:41979/ using document root <none> and password file <none>
I20250815 13:54:53.544736 5173 fs_manager.cc:362] Metadata directory not provided
I20250815 13:54:53.544976 5173 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:54:53.545511 5173 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 13:54:53.551080 5173 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "7f32c2504c9a4178a8a02457f39ec955"
format_stamp: "Formatted at 2025-08-15 13:54:53 on dist-test-slave-kt02"
I20250815 13:54:53.552487 5173 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "7f32c2504c9a4178a8a02457f39ec955"
format_stamp: "Formatted at 2025-08-15 13:54:53 on dist-test-slave-kt02"
I20250815 13:54:53.561432 5173 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s
I20250815 13:54:53.569044 5190 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:54:53.570374 5173 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.003s
I20250815 13:54:53.570719 5173 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "7f32c2504c9a4178a8a02457f39ec955"
format_stamp: "Formatted at 2025-08-15 13:54:53 on dist-test-slave-kt02"
I20250815 13:54:53.571136 5173 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:54:53.641697 5173 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:54:53.643370 5173 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:54:53.643843 5173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:54:53.646723 5173 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 13:54:53.651279 5173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 13:54:53.651506 5173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 13:54:53.651723 5173 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 13:54:53.651863 5173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 13:54:53.805214 5173 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.130:42303
I20250815 13:54:53.805342 5302 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.130:42303 every 8 connection(s)
I20250815 13:54:53.808072 5173 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250815 13:54:53.811089 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 5173
I20250815 13:54:53.811900 426 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250815 13:54:53.819593 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.131:0
--local_ip_for_outbound_sockets=127.0.106.131
--webserver_interface=127.0.106.131
--webserver_port=0
--tserver_master_addrs=127.0.106.190:45211
--builtin_ntp_servers=127.0.106.148:40499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250815 13:54:53.830085 5303 heartbeater.cc:344] Connected to a master server at 127.0.106.190:45211
I20250815 13:54:53.830634 5303 heartbeater.cc:461] Registering TS with master...
I20250815 13:54:53.831750 5303 heartbeater.cc:507] Master 127.0.106.190:45211 requested a full tablet report, sending...
I20250815 13:54:53.833923 4981 ts_manager.cc:194] Registered new tserver with Master: 7f32c2504c9a4178a8a02457f39ec955 (127.0.106.130:42303)
I20250815 13:54:53.835152 4981 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.130:57595
W20250815 13:54:54.154901 5307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:54:54.155550 5307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:54:54.156071 5307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:54:54.189574 5307 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250815 13:54:54.190050 5307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:54:54.190896 5307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.131
I20250815 13:54:54.233050 5307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:40499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.106.131
--webserver_port=0
--tserver_master_addrs=127.0.106.190:45211
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.131
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:54:54.235204 5307 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:54:54.237918 5307 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:54:54.254070 5313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 13:54:54.838405 5303 heartbeater.cc:499] Master 127.0.106.190:45211 was elected leader, sending a full tablet report...
W20250815 13:54:54.255458 5314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:54:55.387497 5316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:54:55.390226 5315 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1129 milliseconds
I20250815 13:54:55.390357 5307 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 13:54:55.391724 5307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:54:55.394002 5307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:54:55.395399 5307 hybrid_clock.cc:648] HybridClock initialized: now 1755266095395377 us; error 45 us; skew 500 ppm
I20250815 13:54:55.396287 5307 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:54:55.402608 5307 webserver.cc:480] Webserver started at http://127.0.106.131:44139/ using document root <none> and password file <none>
I20250815 13:54:55.403597 5307 fs_manager.cc:362] Metadata directory not provided
I20250815 13:54:55.403815 5307 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:54:55.404326 5307 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 13:54:55.408831 5307 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "858e90bf84374fd4be75e6806c4c025e"
format_stamp: "Formatted at 2025-08-15 13:54:55 on dist-test-slave-kt02"
I20250815 13:54:55.409961 5307 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "858e90bf84374fd4be75e6806c4c025e"
format_stamp: "Formatted at 2025-08-15 13:54:55 on dist-test-slave-kt02"
I20250815 13:54:55.417573 5307 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20250815 13:54:55.423527 5323 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:54:55.424778 5307 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20250815 13:54:55.425125 5307 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "858e90bf84374fd4be75e6806c4c025e"
format_stamp: "Formatted at 2025-08-15 13:54:55 on dist-test-slave-kt02"
I20250815 13:54:55.425449 5307 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:54:55.473739 5307 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:54:55.475390 5307 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:54:55.475893 5307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:54:55.478564 5307 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 13:54:55.482807 5307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 13:54:55.483016 5307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 13:54:55.483304 5307 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 13:54:55.483456 5307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 13:54:55.625496 5307 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.131:44177
I20250815 13:54:55.625646 5435 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.131:44177 every 8 connection(s)
I20250815 13:54:55.628093 5307 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250815 13:54:55.632361 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 5307
I20250815 13:54:55.632939 426 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250815 13:54:55.640470 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.132:0
--local_ip_for_outbound_sockets=127.0.106.132
--webserver_interface=127.0.106.132
--webserver_port=0
--tserver_master_addrs=127.0.106.190:45211
--builtin_ntp_servers=127.0.106.148:40499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250815 13:54:55.654811 5436 heartbeater.cc:344] Connected to a master server at 127.0.106.190:45211
I20250815 13:54:55.655488 5436 heartbeater.cc:461] Registering TS with master...
I20250815 13:54:55.657037 5436 heartbeater.cc:507] Master 127.0.106.190:45211 requested a full tablet report, sending...
I20250815 13:54:55.659757 4981 ts_manager.cc:194] Registered new tserver with Master: 858e90bf84374fd4be75e6806c4c025e (127.0.106.131:44177)
I20250815 13:54:55.661372 4981 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.131:34301
W20250815 13:54:55.975873 5440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:54:55.976336 5440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:54:55.976778 5440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:54:56.008011 5440 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250815 13:54:56.008370 5440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:54:56.009112 5440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.132
I20250815 13:54:56.045631 5440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:40499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.106.132
--webserver_port=0
--tserver_master_addrs=127.0.106.190:45211
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.132
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:54:56.046881 5440 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:54:56.048444 5440 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:54:56.060885 5446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 13:54:56.665434 5436 heartbeater.cc:499] Master 127.0.106.190:45211 was elected leader, sending a full tablet report...
W20250815 13:54:56.061545 5447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:54:57.186959 5449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:54:57.191363 5448 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1130 milliseconds
I20250815 13:54:57.191491 5440 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 13:54:57.192842 5440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:54:57.195407 5440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:54:57.196831 5440 hybrid_clock.cc:648] HybridClock initialized: now 1755266097196779 us; error 60 us; skew 500 ppm
I20250815 13:54:57.197817 5440 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:54:57.204933 5440 webserver.cc:480] Webserver started at http://127.0.106.132:37021/ using document root <none> and password file <none>
I20250815 13:54:57.205946 5440 fs_manager.cc:362] Metadata directory not provided
I20250815 13:54:57.206151 5440 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:54:57.206655 5440 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 13:54:57.211565 5440 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "e9e0bbe41c4745e5949a0cd701e87391"
format_stamp: "Formatted at 2025-08-15 13:54:57 on dist-test-slave-kt02"
I20250815 13:54:57.212738 5440 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "e9e0bbe41c4745e5949a0cd701e87391"
format_stamp: "Formatted at 2025-08-15 13:54:57 on dist-test-slave-kt02"
I20250815 13:54:57.220434 5440 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20250815 13:54:57.226940 5456 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:54:57.228163 5440 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250815 13:54:57.228535 5440 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "e9e0bbe41c4745e5949a0cd701e87391"
format_stamp: "Formatted at 2025-08-15 13:54:57 on dist-test-slave-kt02"
I20250815 13:54:57.228881 5440 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:54:57.281149 5440 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:54:57.282868 5440 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:54:57.283377 5440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:54:57.286243 5440 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 13:54:57.290830 5440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 13:54:57.291088 5440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 13:54:57.291368 5440 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 13:54:57.291520 5440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 13:54:57.444288 5440 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.132:45837
I20250815 13:54:57.444397 5568 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.132:45837 every 8 connection(s)
I20250815 13:54:57.447158 5440 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250815 13:54:57.454715 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 5440
I20250815 13:54:57.455202 426 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250815 13:54:57.462719 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.133:0
--local_ip_for_outbound_sockets=127.0.106.133
--webserver_interface=127.0.106.133
--webserver_port=0
--tserver_master_addrs=127.0.106.190:45211
--builtin_ntp_servers=127.0.106.148:40499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250815 13:54:57.469899 5569 heartbeater.cc:344] Connected to a master server at 127.0.106.190:45211
I20250815 13:54:57.470355 5569 heartbeater.cc:461] Registering TS with master...
I20250815 13:54:57.471450 5569 heartbeater.cc:507] Master 127.0.106.190:45211 requested a full tablet report, sending...
I20250815 13:54:57.473762 4981 ts_manager.cc:194] Registered new tserver with Master: e9e0bbe41c4745e5949a0cd701e87391 (127.0.106.132:45837)
I20250815 13:54:57.475237 4981 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.132:48477
W20250815 13:54:57.805672 5573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:54:57.806259 5573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:54:57.806769 5573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:54:57.840323 5573 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250815 13:54:57.840793 5573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:54:57.841627 5573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.133
I20250815 13:54:57.880983 5573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:40499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.106.133
--webserver_port=0
--tserver_master_addrs=127.0.106.190:45211
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.133
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:54:57.882565 5573 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:54:57.884443 5573 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:54:57.897787 5579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 13:54:58.478447 5569 heartbeater.cc:499] Master 127.0.106.190:45211 was elected leader, sending a full tablet report...
W20250815 13:54:57.898778 5580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:54:59.242058 5573 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.344s user 0.506s sys 0.830s
W20250815 13:54:59.242506 5573 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.345s user 0.506s sys 0.830s
W20250815 13:54:59.246464 5581 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1347 milliseconds
W20250815 13:54:59.247402 5582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 13:54:59.247509 5573 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 13:54:59.249224 5573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:54:59.252377 5573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:54:59.253885 5573 hybrid_clock.cc:648] HybridClock initialized: now 1755266099253801 us; error 78 us; skew 500 ppm
I20250815 13:54:59.255136 5573 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:54:59.264086 5573 webserver.cc:480] Webserver started at http://127.0.106.133:42699/ using document root <none> and password file <none>
I20250815 13:54:59.265101 5573 fs_manager.cc:362] Metadata directory not provided
I20250815 13:54:59.265329 5573 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:54:59.265810 5573 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 13:54:59.272243 5573 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "f22647fe3ebd4f5f91f5c846c8d5b52c"
format_stamp: "Formatted at 2025-08-15 13:54:59 on dist-test-slave-kt02"
I20250815 13:54:59.273535 5573 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "f22647fe3ebd4f5f91f5c846c8d5b52c"
format_stamp: "Formatted at 2025-08-15 13:54:59 on dist-test-slave-kt02"
I20250815 13:54:59.282368 5573 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.009s sys 0.001s
I20250815 13:54:59.288937 5590 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:54:59.290313 5573 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.002s
I20250815 13:54:59.290663 5573 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "f22647fe3ebd4f5f91f5c846c8d5b52c"
format_stamp: "Formatted at 2025-08-15 13:54:59 on dist-test-slave-kt02"
I20250815 13:54:59.290997 5573 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:54:59.370060 5573 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:54:59.371649 5573 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:54:59.372095 5573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:54:59.374764 5573 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 13:54:59.378975 5573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 13:54:59.379240 5573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250815 13:54:59.379513 5573 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 13:54:59.379665 5573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 13:54:59.563357 5573 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.133:46859
I20250815 13:54:59.563482 5702 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.133:46859 every 8 connection(s)
I20250815 13:54:59.566339 5573 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250815 13:54:59.576359 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 5573
I20250815 13:54:59.576851 426 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755265962145449-426-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250815 13:54:59.591537 5703 heartbeater.cc:344] Connected to a master server at 127.0.106.190:45211
I20250815 13:54:59.591972 5703 heartbeater.cc:461] Registering TS with master...
I20250815 13:54:59.593502 5703 heartbeater.cc:507] Master 127.0.106.190:45211 requested a full tablet report, sending...
I20250815 13:54:59.596143 4981 ts_manager.cc:194] Registered new tserver with Master: f22647fe3ebd4f5f91f5c846c8d5b52c (127.0.106.133:46859)
I20250815 13:54:59.598098 4981 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.133:53661
I20250815 13:54:59.599820 426 external_mini_cluster.cc:949] 5 TS(s) registered with all masters
I20250815 13:54:59.650964 4981 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32988:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250815 13:54:59.752846 5504 tablet_service.cc:1468] Processing CreateTablet for tablet 256e0b5db90c4386b743a6b82a4ac1b7 (DEFAULT_TABLE table=TestTable [id=8be0825be571494face22b92bfa77f32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 13:54:59.755223 5504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 256e0b5db90c4386b743a6b82a4ac1b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:54:59.757736 5238 tablet_service.cc:1468] Processing CreateTablet for tablet 256e0b5db90c4386b743a6b82a4ac1b7 (DEFAULT_TABLE table=TestTable [id=8be0825be571494face22b92bfa77f32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 13:54:59.762444 5238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 256e0b5db90c4386b743a6b82a4ac1b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:54:59.760315 5638 tablet_service.cc:1468] Processing CreateTablet for tablet 256e0b5db90c4386b743a6b82a4ac1b7 (DEFAULT_TABLE table=TestTable [id=8be0825be571494face22b92bfa77f32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 13:54:59.764199 5638 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 256e0b5db90c4386b743a6b82a4ac1b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:54:59.783291 5722 tablet_bootstrap.cc:492] T 256e0b5db90c4386b743a6b82a4ac1b7 P e9e0bbe41c4745e5949a0cd701e87391: Bootstrap starting.
I20250815 13:54:59.792075 5722 tablet_bootstrap.cc:654] T 256e0b5db90c4386b743a6b82a4ac1b7 P e9e0bbe41c4745e5949a0cd701e87391: Neither blocks nor log segments found. Creating new log.
I20250815 13:54:59.795557 5722 log.cc:826] T 256e0b5db90c4386b743a6b82a4ac1b7 P e9e0bbe41c4745e5949a0cd701e87391: Log is configured to *not* fsync() on all Append() calls
I20250815 13:54:59.795850 5723 tablet_bootstrap.cc:492] T 256e0b5db90c4386b743a6b82a4ac1b7 P 7f32c2504c9a4178a8a02457f39ec955: Bootstrap starting.
I20250815 13:54:59.802242 5724 tablet_bootstrap.cc:492] T 256e0b5db90c4386b743a6b82a4ac1b7 P f22647fe3ebd4f5f91f5c846c8d5b52c: Bootstrap starting.
I20250815 13:54:59.805178 5722 tablet_bootstrap.cc:492] T 256e0b5db90c4386b743a6b82a4ac1b7 P e9e0bbe41c4745e5949a0cd701e87391: No bootstrap required, opened a new log
I20250815 13:54:59.805753 5722 ts_tablet_manager.cc:1397] T 256e0b5db90c4386b743a6b82a4ac1b7 P e9e0bbe41c4745e5949a0cd701e87391: Time spent bootstrapping tablet: real 0.023s user 0.013s sys 0.008s
I20250815 13:54:59.805822 5723 tablet_bootstrap.cc:654] T 256e0b5db90c4386b743a6b82a4ac1b7 P 7f32c2504c9a4178a8a02457f39ec955: Neither blocks nor log segments found. Creating new log.
I20250815 13:54:59.808794 5723 log.cc:826] T 256e0b5db90c4386b743a6b82a4ac1b7 P 7f32c2504c9a4178a8a02457f39ec955: Log is configured to *not* fsync() on all Append() calls
I20250815 13:54:59.810482 5724 tablet_bootstrap.cc:654] T 256e0b5db90c4386b743a6b82a4ac1b7 P f22647fe3ebd4f5f91f5c846c8d5b52c: Neither blocks nor log segments found. Creating new log.
I20250815 13:54:59.813982 5724 log.cc:826] T 256e0b5db90c4386b743a6b82a4ac1b7 P f22647fe3ebd4f5f91f5c846c8d5b52c: Log is configured to *not* fsync() on all Append() calls
I20250815 13:54:59.826202 5724 tablet_bootstrap.cc:492] T 256e0b5db90c4386b743a6b82a4ac1b7 P f22647fe3ebd4f5f91f5c846c8d5b52c: No bootstrap required, opened a new log
I20250815 13:54:59.826905 5724 ts_tablet_manager.cc:1397] T 256e0b5db90c4386b743a6b82a4ac1b7 P f22647fe3ebd4f5f91f5c846c8d5b52c: Time spent bootstrapping tablet: real 0.025s user 0.014s sys 0.009s
I20250815 13:54:59.830116 5723 tablet_bootstrap.cc:492] T 256e0b5db90c4386b743a6b82a4ac1b7 P 7f32c2504c9a4178a8a02457f39ec955: No bootstrap required, opened a new log
I20250815 13:54:59.830732 5723 ts_tablet_manager.cc:1397] T 256e0b5db90c4386b743a6b82a4ac1b7 P 7f32c2504c9a4178a8a02457f39ec955: Time spent bootstrapping tablet: real 0.036s user 0.006s sys 0.026s
I20250815 13:54:59.838367 5722 raft_consensus.cc:357] T 256e0b5db90c4386b743a6b82a4ac1b7 P e9e0bbe41c4745e5949a0cd701e87391 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f32c2504c9a4178a8a02457f39ec955" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 42303 } } peers { permanent_uuid: "f22647fe3ebd4f5f91f5c846c8d5b52c" member_type: VOTER last_known_addr { host: "127.0.106.133" port: 46859 } } peers { permanent_uuid: "e9e0bbe41c4745e5949a0cd701e87391" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 45837 } }
I20250815 13:54:59.839288 5722 raft_consensus.cc:383] T 256e0b5db90c4386b743a6b82a4ac1b7 P e9e0bbe41c4745e5949a0cd701e87391 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 13:54:59.839625 5722 raft_consensus.cc:738] T 256e0b5db90c4386b743a6b82a4ac1b7 P e9e0bbe41c4745e5949a0cd701e87391 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9e0bbe41c4745e5949a0cd701e87391, State: Initialized, Role: FOLLOWER
I20250815 13:54:59.840718 5722 consensus_queue.cc:260] T 256e0b5db90c4386b743a6b82a4ac1b7 P e9e0bbe41c4745e5949a0cd701e87391 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f32c2504c9a4178a8a02457f39ec955" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 42303 } } peers { permanent_uuid: "f22647fe3ebd4f5f91f5c846c8d5b52c" member_type: VOTER last_known_addr { host: "127.0.106.133" port: 46859 } } peers { permanent_uuid: "e9e0bbe41c4745e5949a0cd701e87391" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 45837 } }
I20250815 13:54:59.845513 5722 ts_tablet_manager.cc:1428] T 256e0b5db90c4386b743a6b82a4ac1b7 P e9e0bbe41c4745e5949a0cd701e87391: Time spent starting tablet: real 0.039s user 0.024s sys 0.010s
I20250815 13:54:59.851866 5724 raft_consensus.cc:357] T 256e0b5db90c4386b743a6b82a4ac1b7 P f22647fe3ebd4f5f91f5c846c8d5b52c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f32c2504c9a4178a8a02457f39ec955" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 42303 } } peers { permanent_uuid: "f22647fe3ebd4f5f91f5c846c8d5b52c" member_type: VOTER last_known_addr { host: "127.0.106.133" port: 46859 } } peers { permanent_uuid: "e9e0bbe41c4745e5949a0cd701e87391" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 45837 } }
I20250815 13:54:59.852684 5724 raft_consensus.cc:383] T 256e0b5db90c4386b743a6b82a4ac1b7 P f22647fe3ebd4f5f91f5c846c8d5b52c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 13:54:59.852937 5724 raft_consensus.cc:738] T 256e0b5db90c4386b743a6b82a4ac1b7 P f22647fe3ebd4f5f91f5c846c8d5b52c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f22647fe3ebd4f5f91f5c846c8d5b52c, State: Initialized, Role: FOLLOWER
I20250815 13:54:59.854074 5724 consensus_queue.cc:260] T 256e0b5db90c4386b743a6b82a4ac1b7 P f22647fe3ebd4f5f91f5c846c8d5b52c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f32c2504c9a4178a8a02457f39ec955" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 42303 } } peers { permanent_uuid: "f22647fe3ebd4f5f91f5c846c8d5b52c" member_type: VOTER last_known_addr { host: "127.0.106.133" port: 46859 } } peers { permanent_uuid: "e9e0bbe41c4745e5949a0cd701e87391" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 45837 } }
I20250815 13:54:59.858392 5703 heartbeater.cc:499] Master 127.0.106.190:45211 was elected leader, sending a full tablet report...
I20250815 13:54:59.859498 5724 ts_tablet_manager.cc:1428] T 256e0b5db90c4386b743a6b82a4ac1b7 P f22647fe3ebd4f5f91f5c846c8d5b52c: Time spent starting tablet: real 0.032s user 0.028s sys 0.003s
I20250815 13:54:59.860059 5723 raft_consensus.cc:357] T 256e0b5db90c4386b743a6b82a4ac1b7 P 7f32c2504c9a4178a8a02457f39ec955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f32c2504c9a4178a8a02457f39ec955" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 42303 } } peers { permanent_uuid: "f22647fe3ebd4f5f91f5c846c8d5b52c" member_type: VOTER last_known_addr { host: "127.0.106.133" port: 46859 } } peers { permanent_uuid: "e9e0bbe41c4745e5949a0cd701e87391" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 45837 } }
I20250815 13:54:59.861794 5723 raft_consensus.cc:383] T 256e0b5db90c4386b743a6b82a4ac1b7 P 7f32c2504c9a4178a8a02457f39ec955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 13:54:59.862149 5723 raft_consensus.cc:738] T 256e0b5db90c4386b743a6b82a4ac1b7 P 7f32c2504c9a4178a8a02457f39ec955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f32c2504c9a4178a8a02457f39ec955, State: Initialized, Role: FOLLOWER
I20250815 13:54:59.863673 5723 consensus_queue.cc:260] T 256e0b5db90c4386b743a6b82a4ac1b7 P 7f32c2504c9a4178a8a02457f39ec955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f32c2504c9a4178a8a02457f39ec955" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 42303 } } peers { permanent_uuid: "f22647fe3ebd4f5f91f5c846c8d5b52c" member_type: VOTER last_known_addr { host: "127.0.106.133" port: 46859 } } peers { permanent_uuid: "e9e0bbe41c4745e5949a0cd701e87391" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 45837 } }
I20250815 13:54:59.869200 5723 ts_tablet_manager.cc:1428] T 256e0b5db90c4386b743a6b82a4ac1b7 P 7f32c2504c9a4178a8a02457f39ec955: Time spent starting tablet: real 0.038s user 0.029s sys 0.003s
I20250815 13:54:59.911513 5729 raft_consensus.cc:491] T 256e0b5db90c4386b743a6b82a4ac1b7 P f22647fe3ebd4f5f91f5c846c8d5b52c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 13:54:59.912106 5729 raft_consensus.cc:513] T 256e0b5db90c4386b743a6b82a4ac1b7 P f22647fe3ebd4f5f91f5c846c8d5b52c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f32c2504c9a4178a8a02457f39ec955" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 42303 } } peers { permanent_uuid: "f22647fe3ebd4f5f91f5c846c8d5b52c" member_type: VOTER last_known_addr { host: "127.0.106.133" port: 46859 } } peers { permanent_uuid: "e9e0bbe41c4745e5949a0cd701e87391" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 45837 } }
I20250815 13:54:59.914815 5729 leader_election.cc:290] T 256e0b5db90c4386b743a6b82a4ac1b7 P f22647fe3ebd4f5f91f5c846c8d5b52c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f32c2504c9a4178a8a02457f39ec955 (127.0.106.130:42303), e9e0bbe41c4745e5949a0cd701e87391 (127.0.106.132:45837)
I20250815 13:54:59.924662 5258 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "256e0b5db90c4386b743a6b82a4ac1b7" candidate_uuid: "f22647fe3ebd4f5f91f5c846c8d5b52c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f32c2504c9a4178a8a02457f39ec955" is_pre_election: true
I20250815 13:54:59.925621 5258 raft_consensus.cc:2466] T 256e0b5db90c4386b743a6b82a4ac1b7 P 7f32c2504c9a4178a8a02457f39ec955 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f22647fe3ebd4f5f91f5c846c8d5b52c in term 0.
I20250815 13:54:59.927469 5591 leader_election.cc:304] T 256e0b5db90c4386b743a6b82a4ac1b7 P f22647fe3ebd4f5f91f5c846c8d5b52c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f32c2504c9a4178a8a02457f39ec955, f22647fe3ebd4f5f91f5c846c8d5b52c; no voters:
I20250815 13:54:59.928318 5729 raft_consensus.cc:2802] T 256e0b5db90c4386b743a6b82a4ac1b7 P f22647fe3ebd4f5f91f5c846c8d5b52c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250815 13:54:59.928658 5729 raft_consensus.cc:491] T 256e0b5db90c4386b743a6b82a4ac1b7 P f22647fe3ebd4f5f91f5c846c8d5b52c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 13:54:59.928956 5729 raft_consensus.cc:3058] T 256e0b5db90c4386b743a6b82a4ac1b7 P f22647fe3ebd4f5f91f5c846c8d5b52c [term 0 FOLLOWER]: Advancing to term 1
I20250815 13:54:59.928913 5524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "256e0b5db90c4386b743a6b82a4ac1b7" candidate_uuid: "f22647fe3ebd4f5f91f5c846c8d5b52c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9e0bbe41c4745e5949a0cd701e87391" is_pre_election: true
I20250815 13:54:59.929622 5524 raft_consensus.cc:2466] T 256e0b5db90c4386b743a6b82a4ac1b7 P e9e0bbe41c4745e5949a0cd701e87391 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f22647fe3ebd4f5f91f5c846c8d5b52c in term 0.
I20250815 13:54:59.935184 5729 raft_consensus.cc:513] T 256e0b5db90c4386b743a6b82a4ac1b7 P f22647fe3ebd4f5f91f5c846c8d5b52c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f32c2504c9a4178a8a02457f39ec955" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 42303 } } peers { permanent_uuid: "f22647fe3ebd4f5f91f5c846c8d5b52c" member_type: VOTER last_known_addr { host: "127.0.106.133" port: 46859 } } peers { permanent_uuid: "e9e0bbe41c4745e5949a0cd701e87391" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 45837 } }
I20250815 13:54:59.936628 5729 leader_election.cc:290] T 256e0b5db90c4386b743a6b82a4ac1b7 P f22647fe3ebd4f5f91f5c846c8d5b52c [CANDIDATE]: Term 1 election: Requested vote from peers 7f32c2504c9a4178a8a02457f39ec955 (127.0.106.130:42303), e9e0bbe41c4745e5949a0cd701e87391 (127.0.106.132:45837)
I20250815 13:54:59.937553 5258 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "256e0b5db90c4386b743a6b82a4ac1b7" candidate_uuid: "f22647fe3ebd4f5f91f5c846c8d5b52c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f32c2504c9a4178a8a02457f39ec955"
I20250815 13:54:59.937593 5524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "256e0b5db90c4386b743a6b82a4ac1b7" candidate_uuid: "f22647fe3ebd4f5f91f5c846c8d5b52c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9e0bbe41c4745e5949a0cd701e87391"
I20250815 13:54:59.938006 5258 raft_consensus.cc:3058] T 256e0b5db90c4386b743a6b82a4ac1b7 P 7f32c2504c9a4178a8a02457f39ec955 [term 0 FOLLOWER]: Advancing to term 1
I20250815 13:54:59.938019 5524 raft_consensus.cc:3058] T 256e0b5db90c4386b743a6b82a4ac1b7 P e9e0bbe41c4745e5949a0cd701e87391 [term 0 FOLLOWER]: Advancing to term 1
I20250815 13:54:59.942881 5524 raft_consensus.cc:2466] T 256e0b5db90c4386b743a6b82a4ac1b7 P e9e0bbe41c4745e5949a0cd701e87391 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f22647fe3ebd4f5f91f5c846c8d5b52c in term 1.
I20250815 13:54:59.943122 5258 raft_consensus.cc:2466] T 256e0b5db90c4386b743a6b82a4ac1b7 P 7f32c2504c9a4178a8a02457f39ec955 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f22647fe3ebd4f5f91f5c846c8d5b52c in term 1.
I20250815 13:54:59.944568 5593 leader_election.cc:304] T 256e0b5db90c4386b743a6b82a4ac1b7 P f22647fe3ebd4f5f91f5c846c8d5b52c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e9e0bbe41c4745e5949a0cd701e87391, f22647fe3ebd4f5f91f5c846c8d5b52c; no voters:
I20250815 13:54:59.945269 5729 raft_consensus.cc:2802] T 256e0b5db90c4386b743a6b82a4ac1b7 P f22647fe3ebd4f5f91f5c846c8d5b52c [term 1 FOLLOWER]: Leader election won for term 1
I20250815 13:54:59.946930 5729 raft_consensus.cc:695] T 256e0b5db90c4386b743a6b82a4ac1b7 P f22647fe3ebd4f5f91f5c846c8d5b52c [term 1 LEADER]: Becoming Leader. State: Replica: f22647fe3ebd4f5f91f5c846c8d5b52c, State: Running, Role: LEADER
I20250815 13:54:59.947885 5729 consensus_queue.cc:237] T 256e0b5db90c4386b743a6b82a4ac1b7 P f22647fe3ebd4f5f91f5c846c8d5b52c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f32c2504c9a4178a8a02457f39ec955" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 42303 } } peers { permanent_uuid: "f22647fe3ebd4f5f91f5c846c8d5b52c" member_type: VOTER last_known_addr { host: "127.0.106.133" port: 46859 } } peers { permanent_uuid: "e9e0bbe41c4745e5949a0cd701e87391" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 45837 } }
W20250815 13:54:59.954736 5570 tablet.cc:2378] T 256e0b5db90c4386b743a6b82a4ac1b7 P e9e0bbe41c4745e5949a0cd701e87391: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 13:54:59.959679 4979 catalog_manager.cc:5582] T 256e0b5db90c4386b743a6b82a4ac1b7 P f22647fe3ebd4f5f91f5c846c8d5b52c reported cstate change: term changed from 0 to 1, leader changed from <none> to f22647fe3ebd4f5f91f5c846c8d5b52c (127.0.106.133). New cstate: current_term: 1 leader_uuid: "f22647fe3ebd4f5f91f5c846c8d5b52c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f32c2504c9a4178a8a02457f39ec955" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 42303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f22647fe3ebd4f5f91f5c846c8d5b52c" member_type: VOTER last_known_addr { host: "127.0.106.133" port: 46859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9e0bbe41c4745e5949a0cd701e87391" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 45837 } health_report { overall_health: UNKNOWN } } }
I20250815 13:54:59.990890 426 external_mini_cluster.cc:949] 5 TS(s) registered with all masters
I20250815 13:54:59.995610 426 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 7f32c2504c9a4178a8a02457f39ec955 to finish bootstrapping
I20250815 13:55:00.009826 426 ts_itest-base.cc:246] Waiting for 1 tablets on tserver e9e0bbe41c4745e5949a0cd701e87391 to finish bootstrapping
I20250815 13:55:00.021615 426 ts_itest-base.cc:246] Waiting for 1 tablets on tserver f22647fe3ebd4f5f91f5c846c8d5b52c to finish bootstrapping
I20250815 13:55:00.033694 426 test_util.cc:276] Using random seed: -1134495610
I20250815 13:55:00.061478 426 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20250815 13:55:00.062434 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 5573
W20250815 13:55:00.070013 5304 tablet.cc:2378] T 256e0b5db90c4386b743a6b82a4ac1b7 P 7f32c2504c9a4178a8a02457f39ec955: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250815 13:55:00.087615 5739 negotiation.cc:337] Failed RPC negotiation. Trace:
0815 13:55:00.076498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.106.133:46859 (local address 127.0.0.1:38418)
0815 13:55:00.077086 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0815 13:55:00.077123 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0815 13:55:00.077362 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0815 13:55:00.086168 (+ 8806us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.106.133:46859: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":89}
W20250815 13:55:00.088404 5735 meta_cache.cc:302] tablet 256e0b5db90c4386b743a6b82a4ac1b7: replica f22647fe3ebd4f5f91f5c846c8d5b52c (127.0.106.133:46859) has failed: Network error: Client connection negotiation failed: client connection to 127.0.106.133:46859: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250815 13:55:00.123885 5218 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:55530: Illegal state: replica 7f32c2504c9a4178a8a02457f39ec955 is not leader of this config: current role FOLLOWER
W20250815 13:55:00.148236 5484 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35582: Illegal state: replica e9e0bbe41c4745e5949a0cd701e87391 is not leader of this config: current role FOLLOWER
W20250815 13:55:00.174852 5218 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:55530: Illegal state: replica 7f32c2504c9a4178a8a02457f39ec955 is not leader of this config: current role FOLLOWER
W20250815 13:55:00.188302 5484 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35582: Illegal state: replica e9e0bbe41c4745e5949a0cd701e87391 is not leader of this config: current role FOLLOWER
W20250815 13:55:00.212051 5218 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:55530: Illegal state: replica 7f32c2504c9a4178a8a02457f39ec955 is not leader of this config: current role FOLLOWER
W20250815 13:55:00.226439 5484 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35582: Illegal state: replica e9e0bbe41c4745e5949a0cd701e87391 is not leader of this config: current role FOLLOWER
W20250815 13:55:00.257299 5218 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:55530: Illegal state: replica 7f32c2504c9a4178a8a02457f39ec955 is not leader of this config: current role FOLLOWER
W20250815 13:55:00.276661 5484 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35582: Illegal state: replica e9e0bbe41c4745e5949a0cd701e87391 is not leader of this config: current role FOLLOWER
W20250815 13:55:00.316797 5218 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:55530: Illegal state: replica 7f32c2504c9a4178a8a02457f39ec955 is not leader of this config: current role FOLLOWER
W20250815 13:55:00.335158 5484 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35582: Illegal state: replica e9e0bbe41c4745e5949a0cd701e87391 is not leader of this config: current role FOLLOWER
W20250815 13:55:00.382112 5218 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:55530: Illegal state: replica 7f32c2504c9a4178a8a02457f39ec955 is not leader of this config: current role FOLLOWER
W20250815 13:55:00.405612 5484 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35582: Illegal state: replica e9e0bbe41c4745e5949a0cd701e87391 is not leader of this config: current role FOLLOWER
W20250815 13:55:00.458835 5218 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:55530: Illegal state: replica 7f32c2504c9a4178a8a02457f39ec955 is not leader of this config: current role FOLLOWER
W20250815 13:55:00.483736 5484 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35582: Illegal state: replica e9e0bbe41c4745e5949a0cd701e87391 is not leader of this config: current role FOLLOWER
W20250815 13:55:00.544492 5218 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:55530: Illegal state: replica 7f32c2504c9a4178a8a02457f39ec955 is not leader of this config: current role FOLLOWER
W20250815 13:55:00.573505 5484 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35582: Illegal state: replica e9e0bbe41c4745e5949a0cd701e87391 is not leader of this config: current role FOLLOWER
W20250815 13:55:00.639319 5218 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:55530: Illegal state: replica 7f32c2504c9a4178a8a02457f39ec955 is not leader of this config: current role FOLLOWER
W20250815 13:55:00.674304 5484 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35582: Illegal state: replica e9e0bbe41c4745e5949a0cd701e87391 is not leader of this config: current role FOLLOWER
W20250815 13:55:00.748061 5218 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:55530: Illegal state: replica 7f32c2504c9a4178a8a02457f39ec955 is not leader of this config: current role FOLLOWER
W20250815 13:55:00.785985 5484 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35582: Illegal state: replica e9e0bbe41c4745e5949a0cd701e87391 is not leader of this config: current role FOLLOWER
W20250815 13:55:00.864352 5218 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:55530: Illegal state: replica 7f32c2504c9a4178a8a02457f39ec955 is not leader of this config: current role FOLLOWER
W20250815 13:55:00.904101 5484 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35582: Illegal state: replica e9e0bbe41c4745e5949a0cd701e87391 is not leader of this config: current role FOLLOWER
W20250815 13:55:00.991227 5218 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:55530: Illegal state: replica 7f32c2504c9a4178a8a02457f39ec955 is not leader of this config: current role FOLLOWER
W20250815 13:55:01.034859 5484 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35582: Illegal state: replica e9e0bbe41c4745e5949a0cd701e87391 is not leader of this config: current role FOLLOWER
W20250815 13:55:01.124702 5218 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:55530: Illegal state: replica 7f32c2504c9a4178a8a02457f39ec955 is not leader of this config: current role FOLLOWER
W20250815 13:55:01.172355 5484 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35582: Illegal state: replica e9e0bbe41c4745e5949a0cd701e87391 is not leader of this config: current role FOLLOWER
W20250815 13:55:01.222666 5735 meta_cache.cc:302] tablet 256e0b5db90c4386b743a6b82a4ac1b7: replica f22647fe3ebd4f5f91f5c846c8d5b52c (127.0.106.133:46859) has failed: Network error: Client connection negotiation failed: client connection to 127.0.106.133:46859: connect: Connection refused (error 111)
W20250815 13:55:01.271246 5218 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:55530: Illegal state: replica 7f32c2504c9a4178a8a02457f39ec955 is not leader of this config: current role FOLLOWER
W20250815 13:55:01.322098 5484 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35582: Illegal state: replica e9e0bbe41c4745e5949a0cd701e87391 is not leader of this config: current role FOLLOWER
W20250815 13:55:01.428283 5218 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:55530: Illegal state: replica 7f32c2504c9a4178a8a02457f39ec955 is not leader of this config: current role FOLLOWER
W20250815 13:55:01.477746 5484 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35582: Illegal state: replica e9e0bbe41c4745e5949a0cd701e87391 is not leader of this config: current role FOLLOWER
I20250815 13:55:01.576742 5753 raft_consensus.cc:491] T 256e0b5db90c4386b743a6b82a4ac1b7 P 7f32c2504c9a4178a8a02457f39ec955 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 13:55:01.577368 5753 raft_consensus.cc:513] T 256e0b5db90c4386b743a6b82a4ac1b7 P 7f32c2504c9a4178a8a02457f39ec955 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f32c2504c9a4178a8a02457f39ec955" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 42303 } } peers { permanent_uuid: "f22647fe3ebd4f5f91f5c846c8d5b52c" member_type: VOTER last_known_addr { host: "127.0.106.133" port: 46859 } } peers { permanent_uuid: "e9e0bbe41c4745e5949a0cd701e87391" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 45837 } }
I20250815 13:55:01.580583 5753 leader_election.cc:290] T 256e0b5db90c4386b743a6b82a4ac1b7 P 7f32c2504c9a4178a8a02457f39ec955 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f22647fe3ebd4f5f91f5c846c8d5b52c (127.0.106.133:46859), e9e0bbe41c4745e5949a0cd701e87391 (127.0.106.132:45837)
W20250815 13:55:01.591785 5218 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:55530: Illegal state: replica 7f32c2504c9a4178a8a02457f39ec955 is not leader of this config: current role FOLLOWER
W20250815 13:55:01.598920 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.106.133:46859: connect: Connection refused (error 111)
W20250815 13:55:01.605419 5191 leader_election.cc:336] T 256e0b5db90c4386b743a6b82a4ac1b7 P 7f32c2504c9a4178a8a02457f39ec955 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f22647fe3ebd4f5f91f5c846c8d5b52c (127.0.106.133:46859): Network error: Client connection negotiation failed: client connection to 127.0.106.133:46859: connect: Connection refused (error 111)
I20250815 13:55:01.611707 5524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "256e0b5db90c4386b743a6b82a4ac1b7" candidate_uuid: "7f32c2504c9a4178a8a02457f39ec955" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9e0bbe41c4745e5949a0cd701e87391" is_pre_election: true
I20250815 13:55:01.612289 5524 raft_consensus.cc:2466] T 256e0b5db90c4386b743a6b82a4ac1b7 P e9e0bbe41c4745e5949a0cd701e87391 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f32c2504c9a4178a8a02457f39ec955 in term 1.
I20250815 13:55:01.613520 5193 leader_election.cc:304] T 256e0b5db90c4386b743a6b82a4ac1b7 P 7f32c2504c9a4178a8a02457f39ec955 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7f32c2504c9a4178a8a02457f39ec955, e9e0bbe41c4745e5949a0cd701e87391; no voters: f22647fe3ebd4f5f91f5c846c8d5b52c
I20250815 13:55:01.614282 5753 raft_consensus.cc:2802] T 256e0b5db90c4386b743a6b82a4ac1b7 P 7f32c2504c9a4178a8a02457f39ec955 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250815 13:55:01.614656 5753 raft_consensus.cc:491] T 256e0b5db90c4386b743a6b82a4ac1b7 P 7f32c2504c9a4178a8a02457f39ec955 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 13:55:01.614926 5753 raft_consensus.cc:3058] T 256e0b5db90c4386b743a6b82a4ac1b7 P 7f32c2504c9a4178a8a02457f39ec955 [term 1 FOLLOWER]: Advancing to term 2
I20250815 13:55:01.620442 5753 raft_consensus.cc:513] T 256e0b5db90c4386b743a6b82a4ac1b7 P 7f32c2504c9a4178a8a02457f39ec955 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f32c2504c9a4178a8a02457f39ec955" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 42303 } } peers { permanent_uuid: "f22647fe3ebd4f5f91f5c846c8d5b52c" member_type: VOTER last_known_addr { host: "127.0.106.133" port: 46859 } } peers { permanent_uuid: "e9e0bbe41c4745e5949a0cd701e87391" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 45837 } }
I20250815 13:55:01.622483 5753 leader_election.cc:290] T 256e0b5db90c4386b743a6b82a4ac1b7 P 7f32c2504c9a4178a8a02457f39ec955 [CANDIDATE]: Term 2 election: Requested vote from peers f22647fe3ebd4f5f91f5c846c8d5b52c (127.0.106.133:46859), e9e0bbe41c4745e5949a0cd701e87391 (127.0.106.132:45837)
W20250815 13:55:01.627771 5191 leader_election.cc:336] T 256e0b5db90c4386b743a6b82a4ac1b7 P 7f32c2504c9a4178a8a02457f39ec955 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f22647fe3ebd4f5f91f5c846c8d5b52c (127.0.106.133:46859): Network error: Client connection negotiation failed: client connection to 127.0.106.133:46859: connect: Connection refused (error 111)
I20250815 13:55:01.627741 5524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "256e0b5db90c4386b743a6b82a4ac1b7" candidate_uuid: "7f32c2504c9a4178a8a02457f39ec955" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9e0bbe41c4745e5949a0cd701e87391"
I20250815 13:55:01.628350 5524 raft_consensus.cc:3058] T 256e0b5db90c4386b743a6b82a4ac1b7 P e9e0bbe41c4745e5949a0cd701e87391 [term 1 FOLLOWER]: Advancing to term 2
I20250815 13:55:01.639819 5524 raft_consensus.cc:2466] T 256e0b5db90c4386b743a6b82a4ac1b7 P e9e0bbe41c4745e5949a0cd701e87391 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f32c2504c9a4178a8a02457f39ec955 in term 2.
I20250815 13:55:01.641119 5193 leader_election.cc:304] T 256e0b5db90c4386b743a6b82a4ac1b7 P 7f32c2504c9a4178a8a02457f39ec955 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7f32c2504c9a4178a8a02457f39ec955, e9e0bbe41c4745e5949a0cd701e87391; no voters: f22647fe3ebd4f5f91f5c846c8d5b52c
I20250815 13:55:01.642239 5753 raft_consensus.cc:2802] T 256e0b5db90c4386b743a6b82a4ac1b7 P 7f32c2504c9a4178a8a02457f39ec955 [term 2 FOLLOWER]: Leader election won for term 2
I20250815 13:55:01.644830 5753 raft_consensus.cc:695] T 256e0b5db90c4386b743a6b82a4ac1b7 P 7f32c2504c9a4178a8a02457f39ec955 [term 2 LEADER]: Becoming Leader. State: Replica: 7f32c2504c9a4178a8a02457f39ec955, State: Running, Role: LEADER
I20250815 13:55:01.646932 5753 consensus_queue.cc:237] T 256e0b5db90c4386b743a6b82a4ac1b7 P 7f32c2504c9a4178a8a02457f39ec955 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f32c2504c9a4178a8a02457f39ec955" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 42303 } } peers { permanent_uuid: "f22647fe3ebd4f5f91f5c846c8d5b52c" member_type: VOTER last_known_addr { host: "127.0.106.133" port: 46859 } } peers { permanent_uuid: "e9e0bbe41c4745e5949a0cd701e87391" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 45837 } }
W20250815 13:55:01.649433 5484 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35582: Illegal state: replica e9e0bbe41c4745e5949a0cd701e87391 is not leader of this config: current role FOLLOWER
I20250815 13:55:01.661857 4981 catalog_manager.cc:5582] T 256e0b5db90c4386b743a6b82a4ac1b7 P 7f32c2504c9a4178a8a02457f39ec955 reported cstate change: term changed from 1 to 2, leader changed from f22647fe3ebd4f5f91f5c846c8d5b52c (127.0.106.133) to 7f32c2504c9a4178a8a02457f39ec955 (127.0.106.130). New cstate: current_term: 2 leader_uuid: "7f32c2504c9a4178a8a02457f39ec955" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f32c2504c9a4178a8a02457f39ec955" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 42303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f22647fe3ebd4f5f91f5c846c8d5b52c" member_type: VOTER last_known_addr { host: "127.0.106.133" port: 46859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9e0bbe41c4745e5949a0cd701e87391" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 45837 } health_report { overall_health: UNKNOWN } } }
I20250815 13:55:01.727434 5524 raft_consensus.cc:1273] T 256e0b5db90c4386b743a6b82a4ac1b7 P e9e0bbe41c4745e5949a0cd701e87391 [term 2 FOLLOWER]: Refusing update from remote peer 7f32c2504c9a4178a8a02457f39ec955: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250815 13:55:01.729447 5753 consensus_queue.cc:1035] T 256e0b5db90c4386b743a6b82a4ac1b7 P 7f32c2504c9a4178a8a02457f39ec955 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9e0bbe41c4745e5949a0cd701e87391" member_type: VOTER last_known_addr { host: "127.0.106.132" port: 45837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250815 13:55:01.741348 5191 consensus_peers.cc:489] T 256e0b5db90c4386b743a6b82a4ac1b7 P 7f32c2504c9a4178a8a02457f39ec955 -> Peer f22647fe3ebd4f5f91f5c846c8d5b52c (127.0.106.133:46859): Couldn't send request to peer f22647fe3ebd4f5f91f5c846c8d5b52c. Status: Network error: Client connection negotiation failed: client connection to 127.0.106.133:46859: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250815 13:55:01.766857 5764 mvcc.cc:204] Tried to move back new op lower bound from 7189569952656875520 to 7189569952358240256. Current Snapshot: MvccSnapshot[applied={T|T < 7189569952656875520}]
I20250815 13:55:02.190884 5104 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250815 13:55:02.213723 5238 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250815 13:55:02.233857 5371 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250815 13:55:02.255539 5504 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250815 13:55:04.052253 5104 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250815 13:55:04.054852 5238 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250815 13:55:04.090385 5371 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250815 13:55:04.103264 5504 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250815 13:55:04.112725 5191 consensus_peers.cc:489] T 256e0b5db90c4386b743a6b82a4ac1b7 P 7f32c2504c9a4178a8a02457f39ec955 -> Peer f22647fe3ebd4f5f91f5c846c8d5b52c (127.0.106.133:46859): Couldn't send request to peer f22647fe3ebd4f5f91f5c846c8d5b52c. Status: Network error: Client connection negotiation failed: client connection to 127.0.106.133:46859: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250815 13:55:06.574039 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 5040
I20250815 13:55:06.600971 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 5173
I20250815 13:55:06.636029 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 5307
I20250815 13:55:06.660296 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 5440
I20250815 13:55:06.692394 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 4948
2025-08-15T13:55:06Z chronyd exiting
[ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4 (18811 ms)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest (18812 ms total)
[----------] 1 test from ListTableCliSimpleParamTest
[ RUN ] ListTableCliSimpleParamTest.TestListTables/2
I20250815 13:55:06.763661 426 test_util.cc:276] Using random seed: -1127765655
I20250815 13:55:06.768792 426 ts_itest-base.cc:115] Starting cluster with:
I20250815 13:55:06.769004 426 ts_itest-base.cc:116] --------------
I20250815 13:55:06.769135 426 ts_itest-base.cc:117] 1 tablet servers
I20250815 13:55:06.769248 426 ts_itest-base.cc:118] 1 replicas per TS
I20250815 13:55:06.769363 426 ts_itest-base.cc:119] --------------
2025-08-15T13:55:06Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-15T13:55:06Z Disabled control of system clock
I20250815 13:55:06.822342 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.190:34937
--webserver_interface=127.0.106.190
--webserver_port=0
--builtin_ntp_servers=127.0.106.148:36293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.106.190:34937 with env {}
W20250815 13:55:07.125159 5850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:55:07.125751 5850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:55:07.126233 5850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:55:07.157869 5850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 13:55:07.158186 5850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:55:07.158471 5850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 13:55:07.158712 5850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 13:55:07.194959 5850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:36293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.106.190:34937
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.190:34937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.106.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:55:07.196416 5850 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:55:07.198028 5850 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:55:07.209982 5857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:55:07.209982 5856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:55:08.341428 5859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:55:08.344372 5858 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1128 milliseconds
I20250815 13:55:08.344556 5850 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 13:55:08.346021 5850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:55:08.349076 5850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:55:08.350524 5850 hybrid_clock.cc:648] HybridClock initialized: now 1755266108350463 us; error 76 us; skew 500 ppm
I20250815 13:55:08.351464 5850 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:55:08.358291 5850 webserver.cc:480] Webserver started at http://127.0.106.190:43337/ using document root <none> and password file <none>
I20250815 13:55:08.359346 5850 fs_manager.cc:362] Metadata directory not provided
I20250815 13:55:08.359586 5850 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:55:08.360172 5850 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 13:55:08.365276 5850 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "553ac83d932242fe9a8df7c1c9c4705a"
format_stamp: "Formatted at 2025-08-15 13:55:08 on dist-test-slave-kt02"
I20250815 13:55:08.366593 5850 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "553ac83d932242fe9a8df7c1c9c4705a"
format_stamp: "Formatted at 2025-08-15 13:55:08 on dist-test-slave-kt02"
I20250815 13:55:08.375025 5850 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20250815 13:55:08.381649 5866 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:55:08.382863 5850 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20250815 13:55:08.383291 5850 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
uuid: "553ac83d932242fe9a8df7c1c9c4705a"
format_stamp: "Formatted at 2025-08-15 13:55:08 on dist-test-slave-kt02"
I20250815 13:55:08.383709 5850 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:55:08.458336 5850 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:55:08.460081 5850 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:55:08.460572 5850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:55:08.544169 5850 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.190:34937
I20250815 13:55:08.544235 5917 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.190:34937 every 8 connection(s)
I20250815 13:55:08.547183 5850 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250815 13:55:08.552812 5918 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:55:08.553951 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 5850
I20250815 13:55:08.554472 426 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755265962145449-426-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250815 13:55:08.576200 5918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 553ac83d932242fe9a8df7c1c9c4705a: Bootstrap starting.
I20250815 13:55:08.581882 5918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 553ac83d932242fe9a8df7c1c9c4705a: Neither blocks nor log segments found. Creating new log.
I20250815 13:55:08.583838 5918 log.cc:826] T 00000000000000000000000000000000 P 553ac83d932242fe9a8df7c1c9c4705a: Log is configured to *not* fsync() on all Append() calls
I20250815 13:55:08.589049 5918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 553ac83d932242fe9a8df7c1c9c4705a: No bootstrap required, opened a new log
I20250815 13:55:08.607321 5918 raft_consensus.cc:357] T 00000000000000000000000000000000 P 553ac83d932242fe9a8df7c1c9c4705a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "553ac83d932242fe9a8df7c1c9c4705a" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 34937 } }
I20250815 13:55:08.608022 5918 raft_consensus.cc:383] T 00000000000000000000000000000000 P 553ac83d932242fe9a8df7c1c9c4705a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 13:55:08.608227 5918 raft_consensus.cc:738] T 00000000000000000000000000000000 P 553ac83d932242fe9a8df7c1c9c4705a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 553ac83d932242fe9a8df7c1c9c4705a, State: Initialized, Role: FOLLOWER
I20250815 13:55:08.608873 5918 consensus_queue.cc:260] T 00000000000000000000000000000000 P 553ac83d932242fe9a8df7c1c9c4705a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "553ac83d932242fe9a8df7c1c9c4705a" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 34937 } }
I20250815 13:55:08.609476 5918 raft_consensus.cc:397] T 00000000000000000000000000000000 P 553ac83d932242fe9a8df7c1c9c4705a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 13:55:08.609830 5918 raft_consensus.cc:491] T 00000000000000000000000000000000 P 553ac83d932242fe9a8df7c1c9c4705a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 13:55:08.610174 5918 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 553ac83d932242fe9a8df7c1c9c4705a [term 0 FOLLOWER]: Advancing to term 1
I20250815 13:55:08.614526 5918 raft_consensus.cc:513] T 00000000000000000000000000000000 P 553ac83d932242fe9a8df7c1c9c4705a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "553ac83d932242fe9a8df7c1c9c4705a" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 34937 } }
I20250815 13:55:08.615247 5918 leader_election.cc:304] T 00000000000000000000000000000000 P 553ac83d932242fe9a8df7c1c9c4705a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 553ac83d932242fe9a8df7c1c9c4705a; no voters:
I20250815 13:55:08.617264 5918 leader_election.cc:290] T 00000000000000000000000000000000 P 553ac83d932242fe9a8df7c1c9c4705a [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 13:55:08.618346 5923 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 553ac83d932242fe9a8df7c1c9c4705a [term 1 FOLLOWER]: Leader election won for term 1
I20250815 13:55:08.620755 5923 raft_consensus.cc:695] T 00000000000000000000000000000000 P 553ac83d932242fe9a8df7c1c9c4705a [term 1 LEADER]: Becoming Leader. State: Replica: 553ac83d932242fe9a8df7c1c9c4705a, State: Running, Role: LEADER
I20250815 13:55:08.621403 5918 sys_catalog.cc:564] T 00000000000000000000000000000000 P 553ac83d932242fe9a8df7c1c9c4705a [sys.catalog]: configured and running, proceeding with master startup.
I20250815 13:55:08.621616 5923 consensus_queue.cc:237] T 00000000000000000000000000000000 P 553ac83d932242fe9a8df7c1c9c4705a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "553ac83d932242fe9a8df7c1c9c4705a" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 34937 } }
I20250815 13:55:08.635244 5925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 553ac83d932242fe9a8df7c1c9c4705a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 553ac83d932242fe9a8df7c1c9c4705a. Latest consensus state: current_term: 1 leader_uuid: "553ac83d932242fe9a8df7c1c9c4705a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "553ac83d932242fe9a8df7c1c9c4705a" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 34937 } } }
I20250815 13:55:08.635795 5924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 553ac83d932242fe9a8df7c1c9c4705a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "553ac83d932242fe9a8df7c1c9c4705a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "553ac83d932242fe9a8df7c1c9c4705a" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 34937 } } }
I20250815 13:55:08.636751 5925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 553ac83d932242fe9a8df7c1c9c4705a [sys.catalog]: This master's current role is: LEADER
I20250815 13:55:08.637176 5924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 553ac83d932242fe9a8df7c1c9c4705a [sys.catalog]: This master's current role is: LEADER
I20250815 13:55:08.642030 5933 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 13:55:08.652973 5933 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 13:55:08.670966 5933 catalog_manager.cc:1349] Generated new cluster ID: ccb5443752524fc48c46e5f3b40f1939
I20250815 13:55:08.671310 5933 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 13:55:08.695158 5933 catalog_manager.cc:1372] Generated new certificate authority record
I20250815 13:55:08.696841 5933 catalog_manager.cc:1506] Loading token signing keys...
I20250815 13:55:08.710980 5933 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 553ac83d932242fe9a8df7c1c9c4705a: Generated new TSK 0
I20250815 13:55:08.712126 5933 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250815 13:55:08.726606 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.129:0
--local_ip_for_outbound_sockets=127.0.106.129
--webserver_interface=127.0.106.129
--webserver_port=0
--tserver_master_addrs=127.0.106.190:34937
--builtin_ntp_servers=127.0.106.148:36293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250815 13:55:09.061924 5942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:55:09.062559 5942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:55:09.063341 5942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:55:09.099524 5942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:55:09.100522 5942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.129
I20250815 13:55:09.140239 5942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:36293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.106.129
--webserver_port=0
--tserver_master_addrs=127.0.106.190:34937
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.129
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:55:09.141994 5942 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:55:09.144016 5942 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:55:09.158442 5948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:55:09.161506 5949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:55:10.532171 5950 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1369 milliseconds
W20250815 13:55:10.532691 5951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 13:55:10.532763 5942 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 13:55:10.538103 5942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:55:10.540843 5942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:55:10.542382 5942 hybrid_clock.cc:648] HybridClock initialized: now 1755266110542316 us; error 80 us; skew 500 ppm
I20250815 13:55:10.543373 5942 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:55:10.551015 5942 webserver.cc:480] Webserver started at http://127.0.106.129:44335/ using document root <none> and password file <none>
I20250815 13:55:10.552114 5942 fs_manager.cc:362] Metadata directory not provided
I20250815 13:55:10.552357 5942 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:55:10.553128 5942 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 13:55:10.557915 5942 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "13f8db352fd24e90b54e18f8ce570afc"
format_stamp: "Formatted at 2025-08-15 13:55:10 on dist-test-slave-kt02"
I20250815 13:55:10.559139 5942 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "13f8db352fd24e90b54e18f8ce570afc"
format_stamp: "Formatted at 2025-08-15 13:55:10 on dist-test-slave-kt02"
I20250815 13:55:10.567551 5942 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20250815 13:55:10.574195 5958 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:55:10.575434 5942 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.004s
I20250815 13:55:10.575763 5942 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "13f8db352fd24e90b54e18f8ce570afc"
format_stamp: "Formatted at 2025-08-15 13:55:10 on dist-test-slave-kt02"
I20250815 13:55:10.576102 5942 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:55:10.636299 5942 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:55:10.637705 5942 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:55:10.638144 5942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:55:10.640885 5942 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 13:55:10.645460 5942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 13:55:10.645663 5942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 13:55:10.645907 5942 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 13:55:10.646057 5942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 13:55:10.814747 5942 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.129:46249
I20250815 13:55:10.815006 6070 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.129:46249 every 8 connection(s)
I20250815 13:55:10.817482 5942 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250815 13:55:10.824497 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 5942
I20250815 13:55:10.824929 426 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755265962145449-426-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250815 13:55:10.856786 6071 heartbeater.cc:344] Connected to a master server at 127.0.106.190:34937
I20250815 13:55:10.857321 6071 heartbeater.cc:461] Registering TS with master...
I20250815 13:55:10.858592 6071 heartbeater.cc:507] Master 127.0.106.190:34937 requested a full tablet report, sending...
I20250815 13:55:10.861986 5883 ts_manager.cc:194] Registered new tserver with Master: 13f8db352fd24e90b54e18f8ce570afc (127.0.106.129:46249)
I20250815 13:55:10.864395 5883 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.129:36317
I20250815 13:55:10.872856 426 external_mini_cluster.cc:949] 1 TS(s) registered with all masters
I20250815 13:55:10.908026 5883 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59020:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250815 13:55:10.983685 6006 tablet_service.cc:1468] Processing CreateTablet for tablet a42565f71a774d27aa7ea96034f8f0bb (DEFAULT_TABLE table=TestTable [id=96fc6d57616b4fd1bd7d81d3d8c7af16]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 13:55:10.985652 6006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a42565f71a774d27aa7ea96034f8f0bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:55:11.011601 6086 tablet_bootstrap.cc:492] T a42565f71a774d27aa7ea96034f8f0bb P 13f8db352fd24e90b54e18f8ce570afc: Bootstrap starting.
I20250815 13:55:11.017860 6086 tablet_bootstrap.cc:654] T a42565f71a774d27aa7ea96034f8f0bb P 13f8db352fd24e90b54e18f8ce570afc: Neither blocks nor log segments found. Creating new log.
I20250815 13:55:11.019856 6086 log.cc:826] T a42565f71a774d27aa7ea96034f8f0bb P 13f8db352fd24e90b54e18f8ce570afc: Log is configured to *not* fsync() on all Append() calls
I20250815 13:55:11.025478 6086 tablet_bootstrap.cc:492] T a42565f71a774d27aa7ea96034f8f0bb P 13f8db352fd24e90b54e18f8ce570afc: No bootstrap required, opened a new log
I20250815 13:55:11.025918 6086 ts_tablet_manager.cc:1397] T a42565f71a774d27aa7ea96034f8f0bb P 13f8db352fd24e90b54e18f8ce570afc: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20250815 13:55:11.044867 6086 raft_consensus.cc:357] T a42565f71a774d27aa7ea96034f8f0bb P 13f8db352fd24e90b54e18f8ce570afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f8db352fd24e90b54e18f8ce570afc" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 46249 } }
I20250815 13:55:11.045491 6086 raft_consensus.cc:383] T a42565f71a774d27aa7ea96034f8f0bb P 13f8db352fd24e90b54e18f8ce570afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 13:55:11.045754 6086 raft_consensus.cc:738] T a42565f71a774d27aa7ea96034f8f0bb P 13f8db352fd24e90b54e18f8ce570afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13f8db352fd24e90b54e18f8ce570afc, State: Initialized, Role: FOLLOWER
I20250815 13:55:11.046473 6086 consensus_queue.cc:260] T a42565f71a774d27aa7ea96034f8f0bb P 13f8db352fd24e90b54e18f8ce570afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f8db352fd24e90b54e18f8ce570afc" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 46249 } }
I20250815 13:55:11.046999 6086 raft_consensus.cc:397] T a42565f71a774d27aa7ea96034f8f0bb P 13f8db352fd24e90b54e18f8ce570afc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 13:55:11.047325 6086 raft_consensus.cc:491] T a42565f71a774d27aa7ea96034f8f0bb P 13f8db352fd24e90b54e18f8ce570afc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 13:55:11.047652 6086 raft_consensus.cc:3058] T a42565f71a774d27aa7ea96034f8f0bb P 13f8db352fd24e90b54e18f8ce570afc [term 0 FOLLOWER]: Advancing to term 1
I20250815 13:55:11.052111 6086 raft_consensus.cc:513] T a42565f71a774d27aa7ea96034f8f0bb P 13f8db352fd24e90b54e18f8ce570afc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f8db352fd24e90b54e18f8ce570afc" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 46249 } }
I20250815 13:55:11.052865 6086 leader_election.cc:304] T a42565f71a774d27aa7ea96034f8f0bb P 13f8db352fd24e90b54e18f8ce570afc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13f8db352fd24e90b54e18f8ce570afc; no voters:
I20250815 13:55:11.055233 6086 leader_election.cc:290] T a42565f71a774d27aa7ea96034f8f0bb P 13f8db352fd24e90b54e18f8ce570afc [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 13:55:11.055760 6088 raft_consensus.cc:2802] T a42565f71a774d27aa7ea96034f8f0bb P 13f8db352fd24e90b54e18f8ce570afc [term 1 FOLLOWER]: Leader election won for term 1
I20250815 13:55:11.058454 6088 raft_consensus.cc:695] T a42565f71a774d27aa7ea96034f8f0bb P 13f8db352fd24e90b54e18f8ce570afc [term 1 LEADER]: Becoming Leader. State: Replica: 13f8db352fd24e90b54e18f8ce570afc, State: Running, Role: LEADER
I20250815 13:55:11.058705 6071 heartbeater.cc:499] Master 127.0.106.190:34937 was elected leader, sending a full tablet report...
I20250815 13:55:11.059554 6086 ts_tablet_manager.cc:1428] T a42565f71a774d27aa7ea96034f8f0bb P 13f8db352fd24e90b54e18f8ce570afc: Time spent starting tablet: real 0.033s user 0.031s sys 0.004s
I20250815 13:55:11.059557 6088 consensus_queue.cc:237] T a42565f71a774d27aa7ea96034f8f0bb P 13f8db352fd24e90b54e18f8ce570afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f8db352fd24e90b54e18f8ce570afc" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 46249 } }
I20250815 13:55:11.074378 5883 catalog_manager.cc:5582] T a42565f71a774d27aa7ea96034f8f0bb P 13f8db352fd24e90b54e18f8ce570afc reported cstate change: term changed from 0 to 1, leader changed from <none> to 13f8db352fd24e90b54e18f8ce570afc (127.0.106.129). New cstate: current_term: 1 leader_uuid: "13f8db352fd24e90b54e18f8ce570afc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f8db352fd24e90b54e18f8ce570afc" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 46249 } health_report { overall_health: HEALTHY } } }
I20250815 13:55:11.112875 426 external_mini_cluster.cc:949] 1 TS(s) registered with all masters
I20250815 13:55:11.116787 426 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 13f8db352fd24e90b54e18f8ce570afc to finish bootstrapping
I20250815 13:55:13.845810 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 5942
I20250815 13:55:13.872954 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 5850
2025-08-15T13:55:13Z chronyd exiting
[ OK ] ListTableCliSimpleParamTest.TestListTables/2 (7170 ms)
[----------] 1 test from ListTableCliSimpleParamTest (7170 ms total)
[----------] 1 test from ListTableCliParamTest
[ RUN ] ListTableCliParamTest.ListTabletWithPartitionInfo/4
I20250815 13:55:13.932984 426 test_util.cc:276] Using random seed: -1120596325
[ OK ] ListTableCliParamTest.ListTabletWithPartitionInfo/4 (13 ms)
[----------] 1 test from ListTableCliParamTest (13 ms total)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest
[ RUN ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0
2025-08-15T13:55:13Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-15T13:55:13Z Disabled control of system clock
I20250815 13:55:13.988384 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.190:45437
--webserver_interface=127.0.106.190
--webserver_port=0
--builtin_ntp_servers=127.0.106.148:35301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.106.190:45437 with env {}
W20250815 13:55:14.287698 6115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:55:14.288251 6115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:55:14.288676 6115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:55:14.319762 6115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 13:55:14.320055 6115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:55:14.320261 6115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 13:55:14.320457 6115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 13:55:14.355448 6115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:35301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.106.190:45437
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.190:45437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.106.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:55:14.356760 6115 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:55:14.358307 6115 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:55:14.369428 6121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:55:14.370509 6122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:55:14.374573 6124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:55:15.526023 6123 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1151 milliseconds
I20250815 13:55:15.526151 6115 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 13:55:15.527426 6115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:55:15.529969 6115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:55:15.531332 6115 hybrid_clock.cc:648] HybridClock initialized: now 1755266115531275 us; error 77 us; skew 500 ppm
I20250815 13:55:15.532152 6115 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:55:15.538671 6115 webserver.cc:480] Webserver started at http://127.0.106.190:36121/ using document root <none> and password file <none>
I20250815 13:55:15.539624 6115 fs_manager.cc:362] Metadata directory not provided
I20250815 13:55:15.539822 6115 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:55:15.540285 6115 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 13:55:15.544732 6115 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/master-0/data/instance:
uuid: "157a7243b4f54d869fe682447ba17a09"
format_stamp: "Formatted at 2025-08-15 13:55:15 on dist-test-slave-kt02"
I20250815 13:55:15.545881 6115 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/master-0/wal/instance:
uuid: "157a7243b4f54d869fe682447ba17a09"
format_stamp: "Formatted at 2025-08-15 13:55:15 on dist-test-slave-kt02"
I20250815 13:55:15.553674 6115 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250815 13:55:15.559775 6131 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:55:15.561079 6115 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20250815 13:55:15.561491 6115 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/master-0/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/master-0/wal
uuid: "157a7243b4f54d869fe682447ba17a09"
format_stamp: "Formatted at 2025-08-15 13:55:15 on dist-test-slave-kt02"
I20250815 13:55:15.561852 6115 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:55:15.608192 6115 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:55:15.609634 6115 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:55:15.610064 6115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:55:15.680717 6115 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.190:45437
I20250815 13:55:15.680838 6182 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.190:45437 every 8 connection(s)
I20250815 13:55:15.683386 6115 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/master-0/data/info.pb
I20250815 13:55:15.689009 6183 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:55:15.689301 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 6115
I20250815 13:55:15.689704 426 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/master-0/wal/instance
I20250815 13:55:15.713459 6183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 157a7243b4f54d869fe682447ba17a09: Bootstrap starting.
I20250815 13:55:15.718971 6183 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 157a7243b4f54d869fe682447ba17a09: Neither blocks nor log segments found. Creating new log.
I20250815 13:55:15.720662 6183 log.cc:826] T 00000000000000000000000000000000 P 157a7243b4f54d869fe682447ba17a09: Log is configured to *not* fsync() on all Append() calls
I20250815 13:55:15.725155 6183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 157a7243b4f54d869fe682447ba17a09: No bootstrap required, opened a new log
I20250815 13:55:15.742419 6183 raft_consensus.cc:357] T 00000000000000000000000000000000 P 157a7243b4f54d869fe682447ba17a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "157a7243b4f54d869fe682447ba17a09" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 45437 } }
I20250815 13:55:15.743104 6183 raft_consensus.cc:383] T 00000000000000000000000000000000 P 157a7243b4f54d869fe682447ba17a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 13:55:15.743322 6183 raft_consensus.cc:738] T 00000000000000000000000000000000 P 157a7243b4f54d869fe682447ba17a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 157a7243b4f54d869fe682447ba17a09, State: Initialized, Role: FOLLOWER
I20250815 13:55:15.743979 6183 consensus_queue.cc:260] T 00000000000000000000000000000000 P 157a7243b4f54d869fe682447ba17a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "157a7243b4f54d869fe682447ba17a09" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 45437 } }
I20250815 13:55:15.744480 6183 raft_consensus.cc:397] T 00000000000000000000000000000000 P 157a7243b4f54d869fe682447ba17a09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 13:55:15.744736 6183 raft_consensus.cc:491] T 00000000000000000000000000000000 P 157a7243b4f54d869fe682447ba17a09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 13:55:15.745033 6183 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 157a7243b4f54d869fe682447ba17a09 [term 0 FOLLOWER]: Advancing to term 1
I20250815 13:55:15.749002 6183 raft_consensus.cc:513] T 00000000000000000000000000000000 P 157a7243b4f54d869fe682447ba17a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "157a7243b4f54d869fe682447ba17a09" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 45437 } }
I20250815 13:55:15.749679 6183 leader_election.cc:304] T 00000000000000000000000000000000 P 157a7243b4f54d869fe682447ba17a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 157a7243b4f54d869fe682447ba17a09; no voters:
I20250815 13:55:15.751556 6183 leader_election.cc:290] T 00000000000000000000000000000000 P 157a7243b4f54d869fe682447ba17a09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 13:55:15.752684 6188 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 157a7243b4f54d869fe682447ba17a09 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 13:55:15.754992 6188 raft_consensus.cc:695] T 00000000000000000000000000000000 P 157a7243b4f54d869fe682447ba17a09 [term 1 LEADER]: Becoming Leader. State: Replica: 157a7243b4f54d869fe682447ba17a09, State: Running, Role: LEADER
I20250815 13:55:15.755672 6183 sys_catalog.cc:564] T 00000000000000000000000000000000 P 157a7243b4f54d869fe682447ba17a09 [sys.catalog]: configured and running, proceeding with master startup.
I20250815 13:55:15.755779 6188 consensus_queue.cc:237] T 00000000000000000000000000000000 P 157a7243b4f54d869fe682447ba17a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "157a7243b4f54d869fe682447ba17a09" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 45437 } }
I20250815 13:55:15.767751 6189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 157a7243b4f54d869fe682447ba17a09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "157a7243b4f54d869fe682447ba17a09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "157a7243b4f54d869fe682447ba17a09" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 45437 } } }
I20250815 13:55:15.767992 6190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 157a7243b4f54d869fe682447ba17a09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 157a7243b4f54d869fe682447ba17a09. Latest consensus state: current_term: 1 leader_uuid: "157a7243b4f54d869fe682447ba17a09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "157a7243b4f54d869fe682447ba17a09" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 45437 } } }
I20250815 13:55:15.768676 6189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 157a7243b4f54d869fe682447ba17a09 [sys.catalog]: This master's current role is: LEADER
I20250815 13:55:15.768926 6190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 157a7243b4f54d869fe682447ba17a09 [sys.catalog]: This master's current role is: LEADER
I20250815 13:55:15.772979 6197 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 13:55:15.784783 6197 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 13:55:15.802428 6197 catalog_manager.cc:1349] Generated new cluster ID: 6803aee756ac40ab9416ec06a3472fa5
I20250815 13:55:15.802755 6197 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 13:55:15.856060 6197 catalog_manager.cc:1372] Generated new certificate authority record
I20250815 13:55:15.857860 6197 catalog_manager.cc:1506] Loading token signing keys...
I20250815 13:55:15.871330 6197 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 157a7243b4f54d869fe682447ba17a09: Generated new TSK 0
I20250815 13:55:15.872437 6197 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250815 13:55:15.885394 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.129:0
--local_ip_for_outbound_sockets=127.0.106.129
--webserver_interface=127.0.106.129
--webserver_port=0
--tserver_master_addrs=127.0.106.190:45437
--builtin_ntp_servers=127.0.106.148:35301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20250815 13:55:16.216389 6207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:55:16.217016 6207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:55:16.217546 6207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:55:16.252820 6207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:55:16.253926 6207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.129
I20250815 13:55:16.293394 6207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:35301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.106.129
--webserver_port=0
--tserver_master_addrs=127.0.106.190:45437
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.129
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:55:16.295085 6207 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:55:16.297049 6207 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:55:16.311784 6213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:55:16.318993 6214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:55:16.321436 6216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 13:55:17.413338 6207 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250815 13:55:17.414026 6215 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1098 milliseconds
I20250815 13:55:17.418061 6207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:55:17.421095 6207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:55:17.422561 6207 hybrid_clock.cc:648] HybridClock initialized: now 1755266117422480 us; error 86 us; skew 500 ppm
I20250815 13:55:17.423703 6207 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:55:17.431761 6207 webserver.cc:480] Webserver started at http://127.0.106.129:38405/ using document root <none> and password file <none>
I20250815 13:55:17.433089 6207 fs_manager.cc:362] Metadata directory not provided
I20250815 13:55:17.433378 6207 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:55:17.434077 6207 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 13:55:17.440999 6207 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-0/data/instance:
uuid: "bb218cc976d246bb887cc4fa3bbde42b"
format_stamp: "Formatted at 2025-08-15 13:55:17 on dist-test-slave-kt02"
I20250815 13:55:17.442620 6207 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-0/wal/instance:
uuid: "bb218cc976d246bb887cc4fa3bbde42b"
format_stamp: "Formatted at 2025-08-15 13:55:17 on dist-test-slave-kt02"
I20250815 13:55:17.452633 6207 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.007s sys 0.001s
I20250815 13:55:17.460887 6223 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:55:17.462144 6207 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.003s
I20250815 13:55:17.462505 6207 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-0/wal
uuid: "bb218cc976d246bb887cc4fa3bbde42b"
format_stamp: "Formatted at 2025-08-15 13:55:17 on dist-test-slave-kt02"
I20250815 13:55:17.462883 6207 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:55:17.526836 6207 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:55:17.528415 6207 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:55:17.528856 6207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:55:17.531492 6207 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 13:55:17.535702 6207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 13:55:17.535915 6207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 13:55:17.536180 6207 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 13:55:17.536336 6207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 13:55:17.672537 6207 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.129:40827
I20250815 13:55:17.672806 6335 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.129:40827 every 8 connection(s)
I20250815 13:55:17.675187 6207 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-0/data/info.pb
I20250815 13:55:17.677575 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 6207
I20250815 13:55:17.678156 426 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-0/wal/instance
I20250815 13:55:17.693322 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.130:0
--local_ip_for_outbound_sockets=127.0.106.130
--webserver_interface=127.0.106.130
--webserver_port=0
--tserver_master_addrs=127.0.106.190:45437
--builtin_ntp_servers=127.0.106.148:35301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250815 13:55:17.708485 6336 heartbeater.cc:344] Connected to a master server at 127.0.106.190:45437
I20250815 13:55:17.709050 6336 heartbeater.cc:461] Registering TS with master...
I20250815 13:55:17.710464 6336 heartbeater.cc:507] Master 127.0.106.190:45437 requested a full tablet report, sending...
I20250815 13:55:17.713703 6148 ts_manager.cc:194] Registered new tserver with Master: bb218cc976d246bb887cc4fa3bbde42b (127.0.106.129:40827)
I20250815 13:55:17.716084 6148 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.129:34763
W20250815 13:55:17.998265 6340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:55:17.998775 6340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:55:17.999303 6340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:55:18.030203 6340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:55:18.031132 6340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.130
I20250815 13:55:18.065201 6340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:35301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.106.130
--webserver_port=0
--tserver_master_addrs=127.0.106.190:45437
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.130
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:55:18.066514 6340 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:55:18.068105 6340 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:55:18.080770 6346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 13:55:18.720747 6336 heartbeater.cc:499] Master 127.0.106.190:45437 was elected leader, sending a full tablet report...
W20250815 13:55:18.081465 6347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:55:19.183255 6348 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250815 13:55:19.185271 6349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:55:19.188639 6340 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.108s user 0.442s sys 0.664s
W20250815 13:55:19.189013 6340 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.109s user 0.442s sys 0.664s
I20250815 13:55:19.189257 6340 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 13:55:19.190439 6340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:55:19.193567 6340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:55:19.195040 6340 hybrid_clock.cc:648] HybridClock initialized: now 1755266119194973 us; error 75 us; skew 500 ppm
I20250815 13:55:19.196039 6340 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:55:19.205119 6340 webserver.cc:480] Webserver started at http://127.0.106.130:34725/ using document root <none> and password file <none>
I20250815 13:55:19.206240 6340 fs_manager.cc:362] Metadata directory not provided
I20250815 13:55:19.206483 6340 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:55:19.206984 6340 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 13:55:19.212221 6340 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-1/data/instance:
uuid: "6194ad8a8134493d97a72ac9cc40a705"
format_stamp: "Formatted at 2025-08-15 13:55:19 on dist-test-slave-kt02"
I20250815 13:55:19.213565 6340 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-1/wal/instance:
uuid: "6194ad8a8134493d97a72ac9cc40a705"
format_stamp: "Formatted at 2025-08-15 13:55:19 on dist-test-slave-kt02"
I20250815 13:55:19.222648 6340 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.010s sys 0.000s
I20250815 13:55:19.229741 6356 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:55:19.231036 6340 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20250815 13:55:19.231415 6340 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-1/wal
uuid: "6194ad8a8134493d97a72ac9cc40a705"
format_stamp: "Formatted at 2025-08-15 13:55:19 on dist-test-slave-kt02"
I20250815 13:55:19.231770 6340 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:55:19.303767 6340 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:55:19.305415 6340 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:55:19.306057 6340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:55:19.309010 6340 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 13:55:19.313679 6340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 13:55:19.313915 6340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 13:55:19.314169 6340 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 13:55:19.314334 6340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 13:55:19.461942 6340 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.130:33295
I20250815 13:55:19.462069 6468 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.130:33295 every 8 connection(s)
I20250815 13:55:19.464943 6340 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-1/data/info.pb
I20250815 13:55:19.466511 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 6340
I20250815 13:55:19.466943 426 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-1/wal/instance
I20250815 13:55:19.474936 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.131:0
--local_ip_for_outbound_sockets=127.0.106.131
--webserver_interface=127.0.106.131
--webserver_port=0
--tserver_master_addrs=127.0.106.190:45437
--builtin_ntp_servers=127.0.106.148:35301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250815 13:55:19.491870 6469 heartbeater.cc:344] Connected to a master server at 127.0.106.190:45437
I20250815 13:55:19.492522 6469 heartbeater.cc:461] Registering TS with master...
I20250815 13:55:19.493995 6469 heartbeater.cc:507] Master 127.0.106.190:45437 requested a full tablet report, sending...
I20250815 13:55:19.496784 6148 ts_manager.cc:194] Registered new tserver with Master: 6194ad8a8134493d97a72ac9cc40a705 (127.0.106.130:33295)
I20250815 13:55:19.498064 6148 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.130:59307
W20250815 13:55:19.820413 6473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:55:19.821017 6473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:55:19.821554 6473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:55:19.855587 6473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:55:19.856585 6473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.131
I20250815 13:55:19.895283 6473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:35301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.106.131
--webserver_port=0
--tserver_master_addrs=127.0.106.190:45437
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.131
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:55:19.896904 6473 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:55:19.898703 6473 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:55:19.913173 6479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 13:55:20.502617 6469 heartbeater.cc:499] Master 127.0.106.190:45437 was elected leader, sending a full tablet report...
W20250815 13:55:19.914923 6480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 13:55:19.923590 6473 server_base.cc:1047] running on GCE node
W20250815 13:55:19.923594 6482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 13:55:21.016201 6473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:55:21.018944 6473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:55:21.020454 6473 hybrid_clock.cc:648] HybridClock initialized: now 1755266121020390 us; error 41 us; skew 500 ppm
I20250815 13:55:21.021360 6473 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:55:21.028800 6473 webserver.cc:480] Webserver started at http://127.0.106.131:35129/ using document root <none> and password file <none>
I20250815 13:55:21.029868 6473 fs_manager.cc:362] Metadata directory not provided
I20250815 13:55:21.030099 6473 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:55:21.030663 6473 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 13:55:21.035604 6473 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-2/data/instance:
uuid: "efd284df86ed4e399d6c6dcfa5ecafc7"
format_stamp: "Formatted at 2025-08-15 13:55:21 on dist-test-slave-kt02"
I20250815 13:55:21.036821 6473 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-2/wal/instance:
uuid: "efd284df86ed4e399d6c6dcfa5ecafc7"
format_stamp: "Formatted at 2025-08-15 13:55:21 on dist-test-slave-kt02"
I20250815 13:55:21.045109 6473 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.003s
I20250815 13:55:21.051929 6489 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:55:21.053166 6473 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.000s sys 0.004s
I20250815 13:55:21.053613 6473 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-2/wal
uuid: "efd284df86ed4e399d6c6dcfa5ecafc7"
format_stamp: "Formatted at 2025-08-15 13:55:21 on dist-test-slave-kt02"
I20250815 13:55:21.054018 6473 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:55:21.107546 6473 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:55:21.109215 6473 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:55:21.109829 6473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:55:21.112782 6473 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 13:55:21.117448 6473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 13:55:21.117702 6473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 13:55:21.117957 6473 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 13:55:21.118109 6473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 13:55:21.271863 6473 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.131:45697
I20250815 13:55:21.271994 6601 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.131:45697 every 8 connection(s)
I20250815 13:55:21.274799 6473 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-2/data/info.pb
I20250815 13:55:21.277274 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 6473
I20250815 13:55:21.277793 426 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-2/wal/instance
I20250815 13:55:21.299732 6602 heartbeater.cc:344] Connected to a master server at 127.0.106.190:45437
I20250815 13:55:21.300295 6602 heartbeater.cc:461] Registering TS with master...
I20250815 13:55:21.301482 6602 heartbeater.cc:507] Master 127.0.106.190:45437 requested a full tablet report, sending...
I20250815 13:55:21.303879 6148 ts_manager.cc:194] Registered new tserver with Master: efd284df86ed4e399d6c6dcfa5ecafc7 (127.0.106.131:45697)
I20250815 13:55:21.305295 6148 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.131:50491
I20250815 13:55:21.316087 426 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 13:55:21.343608 426 test_util.cc:276] Using random seed: -1113185695
I20250815 13:55:21.388842 6148 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33792:
name: "pre_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250815 13:55:21.391559 6148 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table pre_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250815 13:55:21.445940 6404 tablet_service.cc:1468] Processing CreateTablet for tablet d1b55d7048ac490bb56a174acd73c669 (DEFAULT_TABLE table=pre_rebuild [id=76b8cf822bd24aedb45dd230c8d8f7f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 13:55:21.448618 6404 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1b55d7048ac490bb56a174acd73c669. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:55:21.455384 6537 tablet_service.cc:1468] Processing CreateTablet for tablet d1b55d7048ac490bb56a174acd73c669 (DEFAULT_TABLE table=pre_rebuild [id=76b8cf822bd24aedb45dd230c8d8f7f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 13:55:21.457441 6537 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1b55d7048ac490bb56a174acd73c669. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:55:21.463739 6271 tablet_service.cc:1468] Processing CreateTablet for tablet d1b55d7048ac490bb56a174acd73c669 (DEFAULT_TABLE table=pre_rebuild [id=76b8cf822bd24aedb45dd230c8d8f7f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 13:55:21.466255 6271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1b55d7048ac490bb56a174acd73c669. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:55:21.493275 6626 tablet_bootstrap.cc:492] T d1b55d7048ac490bb56a174acd73c669 P efd284df86ed4e399d6c6dcfa5ecafc7: Bootstrap starting.
I20250815 13:55:21.496114 6627 tablet_bootstrap.cc:492] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705: Bootstrap starting.
I20250815 13:55:21.499779 6628 tablet_bootstrap.cc:492] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b: Bootstrap starting.
I20250815 13:55:21.500728 6626 tablet_bootstrap.cc:654] T d1b55d7048ac490bb56a174acd73c669 P efd284df86ed4e399d6c6dcfa5ecafc7: Neither blocks nor log segments found. Creating new log.
I20250815 13:55:21.503072 6626 log.cc:826] T d1b55d7048ac490bb56a174acd73c669 P efd284df86ed4e399d6c6dcfa5ecafc7: Log is configured to *not* fsync() on all Append() calls
I20250815 13:55:21.503754 6627 tablet_bootstrap.cc:654] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705: Neither blocks nor log segments found. Creating new log.
I20250815 13:55:21.506058 6627 log.cc:826] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705: Log is configured to *not* fsync() on all Append() calls
I20250815 13:55:21.507671 6628 tablet_bootstrap.cc:654] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b: Neither blocks nor log segments found. Creating new log.
I20250815 13:55:21.509682 6626 tablet_bootstrap.cc:492] T d1b55d7048ac490bb56a174acd73c669 P efd284df86ed4e399d6c6dcfa5ecafc7: No bootstrap required, opened a new log
I20250815 13:55:21.510272 6628 log.cc:826] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b: Log is configured to *not* fsync() on all Append() calls
I20250815 13:55:21.510412 6626 ts_tablet_manager.cc:1397] T d1b55d7048ac490bb56a174acd73c669 P efd284df86ed4e399d6c6dcfa5ecafc7: Time spent bootstrapping tablet: real 0.018s user 0.016s sys 0.000s
I20250815 13:55:21.515381 6627 tablet_bootstrap.cc:492] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705: No bootstrap required, opened a new log
I20250815 13:55:21.516532 6627 ts_tablet_manager.cc:1397] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705: Time spent bootstrapping tablet: real 0.021s user 0.015s sys 0.002s
I20250815 13:55:21.519994 6628 tablet_bootstrap.cc:492] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b: No bootstrap required, opened a new log
I20250815 13:55:21.520434 6628 ts_tablet_manager.cc:1397] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.011s
I20250815 13:55:21.538702 6626 raft_consensus.cc:357] T d1b55d7048ac490bb56a174acd73c669 P efd284df86ed4e399d6c6dcfa5ecafc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6194ad8a8134493d97a72ac9cc40a705" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 33295 } } peers { permanent_uuid: "efd284df86ed4e399d6c6dcfa5ecafc7" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 45697 } } peers { permanent_uuid: "bb218cc976d246bb887cc4fa3bbde42b" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 40827 } }
I20250815 13:55:21.540000 6626 raft_consensus.cc:383] T d1b55d7048ac490bb56a174acd73c669 P efd284df86ed4e399d6c6dcfa5ecafc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 13:55:21.539759 6628 raft_consensus.cc:357] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6194ad8a8134493d97a72ac9cc40a705" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 33295 } } peers { permanent_uuid: "efd284df86ed4e399d6c6dcfa5ecafc7" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 45697 } } peers { permanent_uuid: "bb218cc976d246bb887cc4fa3bbde42b" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 40827 } }
I20250815 13:55:21.540336 6626 raft_consensus.cc:738] T d1b55d7048ac490bb56a174acd73c669 P efd284df86ed4e399d6c6dcfa5ecafc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efd284df86ed4e399d6c6dcfa5ecafc7, State: Initialized, Role: FOLLOWER
I20250815 13:55:21.540604 6628 raft_consensus.cc:383] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 13:55:21.540867 6628 raft_consensus.cc:738] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb218cc976d246bb887cc4fa3bbde42b, State: Initialized, Role: FOLLOWER
I20250815 13:55:21.541390 6626 consensus_queue.cc:260] T d1b55d7048ac490bb56a174acd73c669 P efd284df86ed4e399d6c6dcfa5ecafc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6194ad8a8134493d97a72ac9cc40a705" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 33295 } } peers { permanent_uuid: "efd284df86ed4e399d6c6dcfa5ecafc7" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 45697 } } peers { permanent_uuid: "bb218cc976d246bb887cc4fa3bbde42b" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 40827 } }
I20250815 13:55:21.541697 6628 consensus_queue.cc:260] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6194ad8a8134493d97a72ac9cc40a705" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 33295 } } peers { permanent_uuid: "efd284df86ed4e399d6c6dcfa5ecafc7" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 45697 } } peers { permanent_uuid: "bb218cc976d246bb887cc4fa3bbde42b" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 40827 } }
I20250815 13:55:21.545024 6627 raft_consensus.cc:357] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6194ad8a8134493d97a72ac9cc40a705" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 33295 } } peers { permanent_uuid: "efd284df86ed4e399d6c6dcfa5ecafc7" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 45697 } } peers { permanent_uuid: "bb218cc976d246bb887cc4fa3bbde42b" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 40827 } }
I20250815 13:55:21.546278 6627 raft_consensus.cc:383] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 13:55:21.546679 6627 raft_consensus.cc:738] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6194ad8a8134493d97a72ac9cc40a705, State: Initialized, Role: FOLLOWER
I20250815 13:55:21.547909 6627 consensus_queue.cc:260] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6194ad8a8134493d97a72ac9cc40a705" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 33295 } } peers { permanent_uuid: "efd284df86ed4e399d6c6dcfa5ecafc7" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 45697 } } peers { permanent_uuid: "bb218cc976d246bb887cc4fa3bbde42b" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 40827 } }
I20250815 13:55:21.551100 6602 heartbeater.cc:499] Master 127.0.106.190:45437 was elected leader, sending a full tablet report...
I20250815 13:55:21.552255 6626 ts_tablet_manager.cc:1428] T d1b55d7048ac490bb56a174acd73c669 P efd284df86ed4e399d6c6dcfa5ecafc7: Time spent starting tablet: real 0.041s user 0.024s sys 0.016s
I20250815 13:55:21.556529 6627 ts_tablet_manager.cc:1428] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705: Time spent starting tablet: real 0.040s user 0.034s sys 0.006s
I20250815 13:55:21.557368 6628 ts_tablet_manager.cc:1428] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b: Time spent starting tablet: real 0.037s user 0.025s sys 0.004s
I20250815 13:55:21.563078 6634 raft_consensus.cc:491] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 13:55:21.563719 6634 raft_consensus.cc:513] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6194ad8a8134493d97a72ac9cc40a705" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 33295 } } peers { permanent_uuid: "efd284df86ed4e399d6c6dcfa5ecafc7" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 45697 } } peers { permanent_uuid: "bb218cc976d246bb887cc4fa3bbde42b" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 40827 } }
I20250815 13:55:21.567106 6634 leader_election.cc:290] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efd284df86ed4e399d6c6dcfa5ecafc7 (127.0.106.131:45697), bb218cc976d246bb887cc4fa3bbde42b (127.0.106.129:40827)
I20250815 13:55:21.578644 6557 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1b55d7048ac490bb56a174acd73c669" candidate_uuid: "6194ad8a8134493d97a72ac9cc40a705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efd284df86ed4e399d6c6dcfa5ecafc7" is_pre_election: true
I20250815 13:55:21.579491 6557 raft_consensus.cc:2466] T d1b55d7048ac490bb56a174acd73c669 P efd284df86ed4e399d6c6dcfa5ecafc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6194ad8a8134493d97a72ac9cc40a705 in term 0.
I20250815 13:55:21.580566 6358 leader_election.cc:304] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6194ad8a8134493d97a72ac9cc40a705, efd284df86ed4e399d6c6dcfa5ecafc7; no voters:
I20250815 13:55:21.581358 6634 raft_consensus.cc:2802] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250815 13:55:21.581264 6291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1b55d7048ac490bb56a174acd73c669" candidate_uuid: "6194ad8a8134493d97a72ac9cc40a705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb218cc976d246bb887cc4fa3bbde42b" is_pre_election: true
I20250815 13:55:21.581759 6634 raft_consensus.cc:491] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 13:55:21.582077 6634 raft_consensus.cc:3058] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705 [term 0 FOLLOWER]: Advancing to term 1
I20250815 13:55:21.582262 6291 raft_consensus.cc:2466] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6194ad8a8134493d97a72ac9cc40a705 in term 0.
I20250815 13:55:21.587230 6634 raft_consensus.cc:513] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6194ad8a8134493d97a72ac9cc40a705" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 33295 } } peers { permanent_uuid: "efd284df86ed4e399d6c6dcfa5ecafc7" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 45697 } } peers { permanent_uuid: "bb218cc976d246bb887cc4fa3bbde42b" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 40827 } }
I20250815 13:55:21.588682 6634 leader_election.cc:290] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705 [CANDIDATE]: Term 1 election: Requested vote from peers efd284df86ed4e399d6c6dcfa5ecafc7 (127.0.106.131:45697), bb218cc976d246bb887cc4fa3bbde42b (127.0.106.129:40827)
I20250815 13:55:21.589478 6557 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1b55d7048ac490bb56a174acd73c669" candidate_uuid: "6194ad8a8134493d97a72ac9cc40a705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efd284df86ed4e399d6c6dcfa5ecafc7"
I20250815 13:55:21.590022 6557 raft_consensus.cc:3058] T d1b55d7048ac490bb56a174acd73c669 P efd284df86ed4e399d6c6dcfa5ecafc7 [term 0 FOLLOWER]: Advancing to term 1
I20250815 13:55:21.589871 6291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1b55d7048ac490bb56a174acd73c669" candidate_uuid: "6194ad8a8134493d97a72ac9cc40a705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb218cc976d246bb887cc4fa3bbde42b"
I20250815 13:55:21.590492 6291 raft_consensus.cc:3058] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b [term 0 FOLLOWER]: Advancing to term 1
I20250815 13:55:21.597002 6557 raft_consensus.cc:2466] T d1b55d7048ac490bb56a174acd73c669 P efd284df86ed4e399d6c6dcfa5ecafc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6194ad8a8134493d97a72ac9cc40a705 in term 1.
I20250815 13:55:21.597133 6291 raft_consensus.cc:2466] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6194ad8a8134493d97a72ac9cc40a705 in term 1.
I20250815 13:55:21.598016 6358 leader_election.cc:304] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6194ad8a8134493d97a72ac9cc40a705, efd284df86ed4e399d6c6dcfa5ecafc7; no voters:
I20250815 13:55:21.598693 6634 raft_consensus.cc:2802] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 13:55:21.600531 6634 raft_consensus.cc:695] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705 [term 1 LEADER]: Becoming Leader. State: Replica: 6194ad8a8134493d97a72ac9cc40a705, State: Running, Role: LEADER
I20250815 13:55:21.601367 6634 consensus_queue.cc:237] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6194ad8a8134493d97a72ac9cc40a705" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 33295 } } peers { permanent_uuid: "efd284df86ed4e399d6c6dcfa5ecafc7" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 45697 } } peers { permanent_uuid: "bb218cc976d246bb887cc4fa3bbde42b" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 40827 } }
I20250815 13:55:21.612178 6147 catalog_manager.cc:5582] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6194ad8a8134493d97a72ac9cc40a705 (127.0.106.130). New cstate: current_term: 1 leader_uuid: "6194ad8a8134493d97a72ac9cc40a705" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6194ad8a8134493d97a72ac9cc40a705" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 33295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efd284df86ed4e399d6c6dcfa5ecafc7" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 45697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb218cc976d246bb887cc4fa3bbde42b" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 40827 } health_report { overall_health: UNKNOWN } } }
W20250815 13:55:21.692819 6337 tablet.cc:2378] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250815 13:55:21.722949 6470 tablet.cc:2378] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 13:55:21.753577 6291 raft_consensus.cc:1273] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b [term 1 FOLLOWER]: Refusing update from remote peer 6194ad8a8134493d97a72ac9cc40a705: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250815 13:55:21.754047 6557 raft_consensus.cc:1273] T d1b55d7048ac490bb56a174acd73c669 P efd284df86ed4e399d6c6dcfa5ecafc7 [term 1 FOLLOWER]: Refusing update from remote peer 6194ad8a8134493d97a72ac9cc40a705: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250815 13:55:21.755997 6637 consensus_queue.cc:1035] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb218cc976d246bb887cc4fa3bbde42b" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 40827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250815 13:55:21.756907 6634 consensus_queue.cc:1035] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd284df86ed4e399d6c6dcfa5ecafc7" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 45697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250815 13:55:21.780261 6603 tablet.cc:2378] T d1b55d7048ac490bb56a174acd73c669 P efd284df86ed4e399d6c6dcfa5ecafc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 13:55:21.792706 6644 mvcc.cc:204] Tried to move back new op lower bound from 7189570034684317696 to 7189570034087780352. Current Snapshot: MvccSnapshot[applied={T|T < 7189570034684317696}]
I20250815 13:55:21.804893 6646 mvcc.cc:204] Tried to move back new op lower bound from 7189570034684317696 to 7189570034087780352. Current Snapshot: MvccSnapshot[applied={T|T < 7189570034684317696}]
I20250815 13:55:27.063959 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 6115
W20250815 13:55:27.409351 6678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:55:27.409947 6678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:55:27.441602 6678 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250815 13:55:27.878643 6336 heartbeater.cc:646] Failed to heartbeat to 127.0.106.190:45437 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.106.190:45437: connect: Connection refused (error 111)
W20250815 13:55:27.885386 6469 heartbeater.cc:646] Failed to heartbeat to 127.0.106.190:45437 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.106.190:45437: connect: Connection refused (error 111)
W20250815 13:55:27.919364 6602 heartbeater.cc:646] Failed to heartbeat to 127.0.106.190:45437 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.106.190:45437: connect: Connection refused (error 111)
W20250815 13:55:28.686044 6678 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.201s user 0.002s sys 0.002s
W20250815 13:55:28.686456 6678 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.202s user 0.002s sys 0.002s
I20250815 13:55:28.834278 6678 minidump.cc:252] Setting minidump size limit to 20M
I20250815 13:55:28.836990 6678 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:55:28.838256 6678 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:55:28.851408 6709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:55:28.853699 6710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:55:28.993230 6712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 13:55:28.993770 6678 server_base.cc:1047] running on GCE node
I20250815 13:55:28.994987 6678 hybrid_clock.cc:584] initializing the hybrid clock with 'system' time source
I20250815 13:55:28.995692 6678 hybrid_clock.cc:648] HybridClock initialized: now 1755266128995658 us; error 130516 us; skew 500 ppm
I20250815 13:55:28.996546 6678 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:55:29.001869 6678 webserver.cc:480] Webserver started at http://0.0.0.0:34505/ using document root <none> and password file <none>
I20250815 13:55:29.002826 6678 fs_manager.cc:362] Metadata directory not provided
I20250815 13:55:29.003104 6678 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:55:29.003562 6678 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 13:55:29.008433 6678 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/master-0/data/instance:
uuid: "6741580a9c154f2b9587bd3c456696e0"
format_stamp: "Formatted at 2025-08-15 13:55:29 on dist-test-slave-kt02"
I20250815 13:55:29.009759 6678 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/master-0/wal/instance:
uuid: "6741580a9c154f2b9587bd3c456696e0"
format_stamp: "Formatted at 2025-08-15 13:55:29 on dist-test-slave-kt02"
I20250815 13:55:29.016532 6678 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20250815 13:55:29.022027 6720 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:55:29.023133 6678 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250815 13:55:29.023505 6678 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/master-0/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/master-0/wal
uuid: "6741580a9c154f2b9587bd3c456696e0"
format_stamp: "Formatted at 2025-08-15 13:55:29 on dist-test-slave-kt02"
I20250815 13:55:29.023866 6678 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:55:29.164955 6678 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:55:29.166541 6678 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:55:29.167069 6678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:55:29.172363 6678 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:55:29.188802 6678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0: Bootstrap starting.
I20250815 13:55:29.194450 6678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0: Neither blocks nor log segments found. Creating new log.
I20250815 13:55:29.196705 6678 log.cc:826] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0: Log is configured to *not* fsync() on all Append() calls
I20250815 13:55:29.201726 6678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0: No bootstrap required, opened a new log
I20250815 13:55:29.220139 6678 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6741580a9c154f2b9587bd3c456696e0" member_type: VOTER }
I20250815 13:55:29.220741 6678 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 13:55:29.220952 6678 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6741580a9c154f2b9587bd3c456696e0, State: Initialized, Role: FOLLOWER
I20250815 13:55:29.221778 6678 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6741580a9c154f2b9587bd3c456696e0" member_type: VOTER }
I20250815 13:55:29.222297 6678 raft_consensus.cc:397] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 13:55:29.222539 6678 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 13:55:29.222836 6678 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0 [term 0 FOLLOWER]: Advancing to term 1
I20250815 13:55:29.227698 6678 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6741580a9c154f2b9587bd3c456696e0" member_type: VOTER }
I20250815 13:55:29.228485 6678 leader_election.cc:304] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6741580a9c154f2b9587bd3c456696e0; no voters:
I20250815 13:55:29.230479 6678 leader_election.cc:290] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 13:55:29.230820 6727 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 13:55:29.233561 6727 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0 [term 1 LEADER]: Becoming Leader. State: Replica: 6741580a9c154f2b9587bd3c456696e0, State: Running, Role: LEADER
I20250815 13:55:29.234606 6727 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6741580a9c154f2b9587bd3c456696e0" member_type: VOTER }
I20250815 13:55:29.243110 6728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6741580a9c154f2b9587bd3c456696e0. Latest consensus state: current_term: 1 leader_uuid: "6741580a9c154f2b9587bd3c456696e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6741580a9c154f2b9587bd3c456696e0" member_type: VOTER } }
I20250815 13:55:29.243721 6728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0 [sys.catalog]: This master's current role is: LEADER
I20250815 13:55:29.244699 6729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6741580a9c154f2b9587bd3c456696e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6741580a9c154f2b9587bd3c456696e0" member_type: VOTER } }
I20250815 13:55:29.245169 6729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0 [sys.catalog]: This master's current role is: LEADER
I20250815 13:55:29.256134 6678 tablet_replica.cc:331] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0: stopping tablet replica
I20250815 13:55:29.256701 6678 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0 [term 1 LEADER]: Raft consensus shutting down.
I20250815 13:55:29.257161 6678 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250815 13:55:29.259589 6678 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250815 13:55:29.260015 6678 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250815 13:55:29.287724 6678 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
I20250815 13:55:30.319268 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 6207
I20250815 13:55:30.353018 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 6340
I20250815 13:55:30.392238 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 6473
I20250815 13:55:30.423285 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.190:45437
--webserver_interface=127.0.106.190
--webserver_port=36121
--builtin_ntp_servers=127.0.106.148:35301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.106.190:45437 with env {}
W20250815 13:55:30.728690 6737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:55:30.729307 6737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:55:30.729802 6737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:55:30.764824 6737 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 13:55:30.765143 6737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:55:30.765400 6737 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 13:55:30.765636 6737 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 13:55:30.803040 6737 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:35301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.106.190:45437
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.190:45437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.106.190
--webserver_port=36121
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:55:30.804458 6737 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:55:30.806120 6737 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:55:30.818289 6743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:55:30.818553 6744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:55:31.955598 6746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:55:31.959635 6737 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.142s user 0.362s sys 0.774s
W20250815 13:55:31.960054 6737 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.143s user 0.362s sys 0.774s
W20250815 13:55:31.960391 6745 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1140 milliseconds
I20250815 13:55:31.960386 6737 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 13:55:31.962018 6737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:55:31.965075 6737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:55:31.966610 6737 hybrid_clock.cc:648] HybridClock initialized: now 1755266131966570 us; error 73 us; skew 500 ppm
I20250815 13:55:31.967540 6737 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:55:31.975584 6737 webserver.cc:480] Webserver started at http://127.0.106.190:36121/ using document root <none> and password file <none>
I20250815 13:55:31.976655 6737 fs_manager.cc:362] Metadata directory not provided
I20250815 13:55:31.976897 6737 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:55:31.985827 6737 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.000s sys 0.005s
I20250815 13:55:31.991806 6753 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:55:31.993269 6737 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20250815 13:55:31.993724 6737 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/master-0/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/master-0/wal
uuid: "6741580a9c154f2b9587bd3c456696e0"
format_stamp: "Formatted at 2025-08-15 13:55:29 on dist-test-slave-kt02"
I20250815 13:55:31.995852 6737 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:55:32.068909 6737 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:55:32.070374 6737 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:55:32.070817 6737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:55:32.145608 6737 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.190:45437
I20250815 13:55:32.145717 6804 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.190:45437 every 8 connection(s)
I20250815 13:55:32.149029 6737 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/master-0/data/info.pb
I20250815 13:55:32.150377 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 6737
I20250815 13:55:32.152163 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.129:40827
--local_ip_for_outbound_sockets=127.0.106.129
--tserver_master_addrs=127.0.106.190:45437
--webserver_port=38405
--webserver_interface=127.0.106.129
--builtin_ntp_servers=127.0.106.148:35301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250815 13:55:32.160511 6805 sys_catalog.cc:263] Verifying existing consensus state
I20250815 13:55:32.172400 6805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0: Bootstrap starting.
I20250815 13:55:32.184001 6805 log.cc:826] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0: Log is configured to *not* fsync() on all Append() calls
I20250815 13:55:32.197247 6805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 13:55:32.198071 6805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0: Bootstrap complete.
I20250815 13:55:32.220693 6805 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6741580a9c154f2b9587bd3c456696e0" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 45437 } }
I20250815 13:55:32.221444 6805 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6741580a9c154f2b9587bd3c456696e0, State: Initialized, Role: FOLLOWER
I20250815 13:55:32.222290 6805 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6741580a9c154f2b9587bd3c456696e0" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 45437 } }
I20250815 13:55:32.222815 6805 raft_consensus.cc:397] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 13:55:32.223115 6805 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 13:55:32.223438 6805 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0 [term 1 FOLLOWER]: Advancing to term 2
I20250815 13:55:32.227911 6805 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6741580a9c154f2b9587bd3c456696e0" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 45437 } }
I20250815 13:55:32.228753 6805 leader_election.cc:304] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6741580a9c154f2b9587bd3c456696e0; no voters:
I20250815 13:55:32.230841 6805 leader_election.cc:290] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250815 13:55:32.231441 6809 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0 [term 2 FOLLOWER]: Leader election won for term 2
I20250815 13:55:32.234859 6809 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0 [term 2 LEADER]: Becoming Leader. State: Replica: 6741580a9c154f2b9587bd3c456696e0, State: Running, Role: LEADER
I20250815 13:55:32.235931 6809 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6741580a9c154f2b9587bd3c456696e0" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 45437 } }
I20250815 13:55:32.236598 6805 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0 [sys.catalog]: configured and running, proceeding with master startup.
I20250815 13:55:32.245580 6810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6741580a9c154f2b9587bd3c456696e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6741580a9c154f2b9587bd3c456696e0" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 45437 } } }
I20250815 13:55:32.246335 6810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0 [sys.catalog]: This master's current role is: LEADER
I20250815 13:55:32.247807 6811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6741580a9c154f2b9587bd3c456696e0. Latest consensus state: current_term: 2 leader_uuid: "6741580a9c154f2b9587bd3c456696e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6741580a9c154f2b9587bd3c456696e0" member_type: VOTER last_known_addr { host: "127.0.106.190" port: 45437 } } }
I20250815 13:55:32.249297 6811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0 [sys.catalog]: This master's current role is: LEADER
I20250815 13:55:32.263223 6817 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 13:55:32.275426 6817 catalog_manager.cc:671] Loaded metadata for table pre_rebuild [id=caa7228106b94ee8b1bf1432d67e47bb]
I20250815 13:55:32.282428 6817 tablet_loader.cc:96] loaded metadata for tablet d1b55d7048ac490bb56a174acd73c669 (table pre_rebuild [id=caa7228106b94ee8b1bf1432d67e47bb])
I20250815 13:55:32.284785 6817 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 13:55:32.315250 6817 catalog_manager.cc:1349] Generated new cluster ID: 473ea61bd4e0427dabb9e811a048fe46
I20250815 13:55:32.315637 6817 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 13:55:32.349751 6817 catalog_manager.cc:1372] Generated new certificate authority record
I20250815 13:55:32.352290 6817 catalog_manager.cc:1506] Loading token signing keys...
I20250815 13:55:32.372043 6817 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0: Generated new TSK 0
I20250815 13:55:32.373238 6817 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250815 13:55:32.522125 6807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:55:32.522799 6807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:55:32.523372 6807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:55:32.556337 6807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:55:32.557262 6807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.129
I20250815 13:55:32.595872 6807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:35301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.129:40827
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.106.129
--webserver_port=38405
--tserver_master_addrs=127.0.106.190:45437
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.129
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:55:32.597601 6807 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:55:32.599649 6807 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:55:32.616191 6833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 13:55:32.622532 6807 server_base.cc:1047] running on GCE node
W20250815 13:55:32.620287 6836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:55:32.620689 6834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 13:55:33.758955 6807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:55:33.762010 6807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:55:33.763583 6807 hybrid_clock.cc:648] HybridClock initialized: now 1755266133763475 us; error 101 us; skew 500 ppm
I20250815 13:55:33.764570 6807 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:55:33.772639 6807 webserver.cc:480] Webserver started at http://127.0.106.129:38405/ using document root <none> and password file <none>
I20250815 13:55:33.774037 6807 fs_manager.cc:362] Metadata directory not provided
I20250815 13:55:33.774359 6807 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:55:33.786048 6807 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.001s sys 0.004s
I20250815 13:55:33.793040 6844 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:55:33.794929 6807 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.002s sys 0.003s
I20250815 13:55:33.795446 6807 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-0/wal
uuid: "bb218cc976d246bb887cc4fa3bbde42b"
format_stamp: "Formatted at 2025-08-15 13:55:17 on dist-test-slave-kt02"
I20250815 13:55:33.798518 6807 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:55:33.881150 6807 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:55:33.883287 6807 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:55:33.883891 6807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:55:33.887334 6807 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 13:55:33.895328 6851 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250815 13:55:33.904558 6807 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250815 13:55:33.904953 6807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.012s user 0.002s sys 0.001s
I20250815 13:55:33.905315 6807 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250815 13:55:33.913352 6807 ts_tablet_manager.cc:610] Registered 1 tablets
I20250815 13:55:33.913682 6807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.008s user 0.003s sys 0.004s
I20250815 13:55:33.914291 6851 tablet_bootstrap.cc:492] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b: Bootstrap starting.
I20250815 13:55:34.099035 6807 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.129:40827
I20250815 13:55:34.099287 6957 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.129:40827 every 8 connection(s)
I20250815 13:55:34.103006 6807 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-0/data/info.pb
I20250815 13:55:34.106691 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 6807
I20250815 13:55:34.109005 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.130:33295
--local_ip_for_outbound_sockets=127.0.106.130
--tserver_master_addrs=127.0.106.190:45437
--webserver_port=34725
--webserver_interface=127.0.106.130
--builtin_ntp_servers=127.0.106.148:35301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250815 13:55:34.171207 6958 heartbeater.cc:344] Connected to a master server at 127.0.106.190:45437
I20250815 13:55:34.171788 6958 heartbeater.cc:461] Registering TS with master...
I20250815 13:55:34.173231 6958 heartbeater.cc:507] Master 127.0.106.190:45437 requested a full tablet report, sending...
I20250815 13:55:34.179352 6770 ts_manager.cc:194] Registered new tserver with Master: bb218cc976d246bb887cc4fa3bbde42b (127.0.106.129:40827)
I20250815 13:55:34.188709 6770 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.129:48199
I20250815 13:55:34.228751 6851 log.cc:826] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b: Log is configured to *not* fsync() on all Append() calls
W20250815 13:55:34.509081 6962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:55:34.509696 6962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:55:34.510249 6962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:55:34.546003 6962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:55:34.547246 6962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.130
I20250815 13:55:34.597761 6962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:35301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.130:33295
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.106.130
--webserver_port=34725
--tserver_master_addrs=127.0.106.190:45437
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.130
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:55:34.599974 6962 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:55:34.602541 6962 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:55:34.620371 6969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 13:55:35.194720 6958 heartbeater.cc:499] Master 127.0.106.190:45437 was elected leader, sending a full tablet report...
W20250815 13:55:36.022514 6968 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 6962
W20250815 13:55:36.371431 6968 kernel_stack_watchdog.cc:198] Thread 6962 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 398ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250815 13:55:34.621466 6970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:55:36.372176 6962 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.750s user 0.711s sys 1.008s
W20250815 13:55:36.372607 6962 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.751s user 0.711s sys 1.008s
W20250815 13:55:36.375082 6972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:55:36.380257 6971 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1756 milliseconds
I20250815 13:55:36.380360 6962 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 13:55:36.382103 6962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:55:36.385191 6962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:55:36.386868 6962 hybrid_clock.cc:648] HybridClock initialized: now 1755266136386796 us; error 69 us; skew 500 ppm
I20250815 13:55:36.388124 6962 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:55:36.396734 6962 webserver.cc:480] Webserver started at http://127.0.106.130:34725/ using document root <none> and password file <none>
I20250815 13:55:36.398237 6962 fs_manager.cc:362] Metadata directory not provided
I20250815 13:55:36.398567 6962 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:55:36.408474 6962 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.006s sys 0.001s
I20250815 13:55:36.414180 6979 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:55:36.415722 6962 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20250815 13:55:36.416115 6962 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-1/wal
uuid: "6194ad8a8134493d97a72ac9cc40a705"
format_stamp: "Formatted at 2025-08-15 13:55:19 on dist-test-slave-kt02"
I20250815 13:55:36.418241 6962 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:55:36.467000 6962 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:55:36.468618 6962 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:55:36.469131 6962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:55:36.471942 6962 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 13:55:36.478600 6986 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250815 13:55:36.490168 6962 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250815 13:55:36.490432 6962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.001s sys 0.001s
I20250815 13:55:36.490731 6962 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250815 13:55:36.496090 6962 ts_tablet_manager.cc:610] Registered 1 tablets
I20250815 13:55:36.496376 6962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s user 0.005s sys 0.000s
I20250815 13:55:36.496816 6986 tablet_bootstrap.cc:492] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705: Bootstrap starting.
I20250815 13:55:36.695276 6962 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.130:33295
I20250815 13:55:36.695520 7092 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.130:33295 every 8 connection(s)
I20250815 13:55:36.699131 6962 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-1/data/info.pb
I20250815 13:55:36.708564 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 6962
I20250815 13:55:36.710580 426 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
/tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.131:45697
--local_ip_for_outbound_sockets=127.0.106.131
--tserver_master_addrs=127.0.106.190:45437
--webserver_port=35129
--webserver_interface=127.0.106.131
--builtin_ntp_servers=127.0.106.148:35301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250815 13:55:36.757661 7093 heartbeater.cc:344] Connected to a master server at 127.0.106.190:45437
I20250815 13:55:36.758234 7093 heartbeater.cc:461] Registering TS with master...
I20250815 13:55:36.759593 7093 heartbeater.cc:507] Master 127.0.106.190:45437 requested a full tablet report, sending...
I20250815 13:55:36.764081 6770 ts_manager.cc:194] Registered new tserver with Master: 6194ad8a8134493d97a72ac9cc40a705 (127.0.106.130:33295)
I20250815 13:55:36.767236 6770 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.130:43999
I20250815 13:55:36.808519 6986 log.cc:826] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705: Log is configured to *not* fsync() on all Append() calls
W20250815 13:55:37.234572 7097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 13:55:37.235296 7097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 13:55:37.236114 7097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 13:55:37.293236 7097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 13:55:37.294739 7097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.131
I20250815 13:55:37.299343 6851 tablet_bootstrap.cc:492] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 13:55:37.300242 6851 tablet_bootstrap.cc:492] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b: Bootstrap complete.
I20250815 13:55:37.301615 6851 ts_tablet_manager.cc:1397] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b: Time spent bootstrapping tablet: real 3.388s user 3.228s sys 0.132s
I20250815 13:55:37.314210 6851 raft_consensus.cc:357] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6194ad8a8134493d97a72ac9cc40a705" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 33295 } } peers { permanent_uuid: "efd284df86ed4e399d6c6dcfa5ecafc7" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 45697 } } peers { permanent_uuid: "bb218cc976d246bb887cc4fa3bbde42b" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 40827 } }
I20250815 13:55:37.316634 6851 raft_consensus.cc:738] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb218cc976d246bb887cc4fa3bbde42b, State: Initialized, Role: FOLLOWER
I20250815 13:55:37.317495 6851 consensus_queue.cc:260] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6194ad8a8134493d97a72ac9cc40a705" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 33295 } } peers { permanent_uuid: "efd284df86ed4e399d6c6dcfa5ecafc7" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 45697 } } peers { permanent_uuid: "bb218cc976d246bb887cc4fa3bbde42b" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 40827 } }
I20250815 13:55:37.321244 6851 ts_tablet_manager.cc:1428] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b: Time spent starting tablet: real 0.019s user 0.013s sys 0.008s
I20250815 13:55:37.356412 7097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.148:35301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.131:45697
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.106.131
--webserver_port=35129
--tserver_master_addrs=127.0.106.190:45437
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.131
--log_dir=/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 7581
TSAN enabled
I20250815 13:55:37.358130 7097 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 13:55:37.360292 7097 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 13:55:37.377154 7105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 13:55:37.771675 7093 heartbeater.cc:499] Master 127.0.106.190:45437 was elected leader, sending a full tablet report...
I20250815 13:55:38.693421 7111 raft_consensus.cc:491] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 13:55:38.695458 7111 raft_consensus.cc:513] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6194ad8a8134493d97a72ac9cc40a705" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 33295 } } peers { permanent_uuid: "efd284df86ed4e399d6c6dcfa5ecafc7" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 45697 } } peers { permanent_uuid: "bb218cc976d246bb887cc4fa3bbde42b" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 40827 } }
I20250815 13:55:38.717567 7111 leader_election.cc:290] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6194ad8a8134493d97a72ac9cc40a705 (127.0.106.130:33295), efd284df86ed4e399d6c6dcfa5ecafc7 (127.0.106.131:45697)
W20250815 13:55:38.722676 6846 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.106.131:45697: connect: Connection refused (error 111)
W20250815 13:55:38.738862 6846 leader_election.cc:336] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer efd284df86ed4e399d6c6dcfa5ecafc7 (127.0.106.131:45697): Network error: Client connection negotiation failed: client connection to 127.0.106.131:45697: connect: Connection refused (error 111)
I20250815 13:55:38.757264 7048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1b55d7048ac490bb56a174acd73c669" candidate_uuid: "bb218cc976d246bb887cc4fa3bbde42b" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "6194ad8a8134493d97a72ac9cc40a705" is_pre_election: true
W20250815 13:55:38.811290 6846 leader_election.cc:343] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6194ad8a8134493d97a72ac9cc40a705 (127.0.106.130:33295): Illegal state: must be running to vote when last-logged opid is not known
I20250815 13:55:38.817034 6846 leader_election.cc:304] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb218cc976d246bb887cc4fa3bbde42b; no voters: 6194ad8a8134493d97a72ac9cc40a705, efd284df86ed4e399d6c6dcfa5ecafc7
I20250815 13:55:38.818151 7111 raft_consensus.cc:2747] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250815 13:55:37.378690 7106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:55:38.780020 7104 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 7097
W20250815 13:55:39.091491 7104 kernel_stack_watchdog.cc:198] Thread 7097 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 399ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250815 13:55:39.092197 7097 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.714s user 0.617s sys 1.023s
W20250815 13:55:39.092585 7097 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.714s user 0.617s sys 1.023s
W20250815 13:55:39.093869 7108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 13:55:39.096710 7107 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1717 milliseconds
I20250815 13:55:39.096745 7097 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 13:55:39.098428 7097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 13:55:39.101263 7097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 13:55:39.102850 7097 hybrid_clock.cc:648] HybridClock initialized: now 1755266139102812 us; error 56 us; skew 500 ppm
I20250815 13:55:39.104043 7097 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 13:55:39.112109 7097 webserver.cc:480] Webserver started at http://127.0.106.131:35129/ using document root <none> and password file <none>
I20250815 13:55:39.113516 7097 fs_manager.cc:362] Metadata directory not provided
I20250815 13:55:39.113857 7097 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 13:55:39.125785 7097 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.005s sys 0.004s
I20250815 13:55:39.132301 7120 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 13:55:39.133790 7097 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20250815 13:55:39.134263 7097 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-2/wal
uuid: "efd284df86ed4e399d6c6dcfa5ecafc7"
format_stamp: "Formatted at 2025-08-15 13:55:21 on dist-test-slave-kt02"
I20250815 13:55:39.137132 7097 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 13:55:39.210255 7097 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 13:55:39.212361 7097 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 13:55:39.212941 7097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 13:55:39.216323 7097 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 13:55:39.224532 7127 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250815 13:55:39.233583 7097 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250815 13:55:39.233950 7097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.012s user 0.000s sys 0.003s
I20250815 13:55:39.234297 7097 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250815 13:55:39.241935 7097 ts_tablet_manager.cc:610] Registered 1 tablets
I20250815 13:55:39.242244 7097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.008s user 0.006s sys 0.000s
I20250815 13:55:39.242811 7127 tablet_bootstrap.cc:492] T d1b55d7048ac490bb56a174acd73c669 P efd284df86ed4e399d6c6dcfa5ecafc7: Bootstrap starting.
I20250815 13:55:39.443419 7097 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.131:45697
I20250815 13:55:39.443786 7233 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.131:45697 every 8 connection(s)
I20250815 13:55:39.447417 7097 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-2/data/info.pb
I20250815 13:55:39.457120 426 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu as pid 7097
I20250815 13:55:39.505039 7234 heartbeater.cc:344] Connected to a master server at 127.0.106.190:45437
I20250815 13:55:39.505661 7234 heartbeater.cc:461] Registering TS with master...
I20250815 13:55:39.507193 7234 heartbeater.cc:507] Master 127.0.106.190:45437 requested a full tablet report, sending...
I20250815 13:55:39.511765 6770 ts_manager.cc:194] Registered new tserver with Master: efd284df86ed4e399d6c6dcfa5ecafc7 (127.0.106.131:45697)
I20250815 13:55:39.515585 6770 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.131:60909
I20250815 13:55:39.517853 426 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 13:55:39.559875 7127 log.cc:826] T d1b55d7048ac490bb56a174acd73c669 P efd284df86ed4e399d6c6dcfa5ecafc7: Log is configured to *not* fsync() on all Append() calls
I20250815 13:55:39.734829 6986 tablet_bootstrap.cc:492] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 13:55:39.735796 6986 tablet_bootstrap.cc:492] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705: Bootstrap complete.
I20250815 13:55:39.736953 6986 ts_tablet_manager.cc:1397] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705: Time spent bootstrapping tablet: real 3.241s user 3.104s sys 0.091s
I20250815 13:55:39.742247 6986 raft_consensus.cc:357] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6194ad8a8134493d97a72ac9cc40a705" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 33295 } } peers { permanent_uuid: "efd284df86ed4e399d6c6dcfa5ecafc7" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 45697 } } peers { permanent_uuid: "bb218cc976d246bb887cc4fa3bbde42b" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 40827 } }
I20250815 13:55:39.744226 6986 raft_consensus.cc:738] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6194ad8a8134493d97a72ac9cc40a705, State: Initialized, Role: FOLLOWER
I20250815 13:55:39.744980 6986 consensus_queue.cc:260] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6194ad8a8134493d97a72ac9cc40a705" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 33295 } } peers { permanent_uuid: "efd284df86ed4e399d6c6dcfa5ecafc7" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 45697 } } peers { permanent_uuid: "bb218cc976d246bb887cc4fa3bbde42b" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 40827 } }
I20250815 13:55:39.747960 6986 ts_tablet_manager.cc:1428] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20250815 13:55:40.466053 7247 raft_consensus.cc:491] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 13:55:40.466455 7247 raft_consensus.cc:513] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6194ad8a8134493d97a72ac9cc40a705" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 33295 } } peers { permanent_uuid: "efd284df86ed4e399d6c6dcfa5ecafc7" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 45697 } } peers { permanent_uuid: "bb218cc976d246bb887cc4fa3bbde42b" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 40827 } }
I20250815 13:55:40.468116 7247 leader_election.cc:290] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6194ad8a8134493d97a72ac9cc40a705 (127.0.106.130:33295), efd284df86ed4e399d6c6dcfa5ecafc7 (127.0.106.131:45697)
I20250815 13:55:40.469516 7048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1b55d7048ac490bb56a174acd73c669" candidate_uuid: "bb218cc976d246bb887cc4fa3bbde42b" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "6194ad8a8134493d97a72ac9cc40a705" is_pre_election: true
I20250815 13:55:40.470322 7048 raft_consensus.cc:2466] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb218cc976d246bb887cc4fa3bbde42b in term 1.
I20250815 13:55:40.479425 6846 leader_election.cc:304] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6194ad8a8134493d97a72ac9cc40a705, bb218cc976d246bb887cc4fa3bbde42b; no voters:
I20250815 13:55:40.481686 7247 raft_consensus.cc:2802] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250815 13:55:40.482071 7247 raft_consensus.cc:491] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 13:55:40.482395 7247 raft_consensus.cc:3058] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b [term 1 FOLLOWER]: Advancing to term 2
I20250815 13:55:40.490365 7247 raft_consensus.cc:513] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6194ad8a8134493d97a72ac9cc40a705" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 33295 } } peers { permanent_uuid: "efd284df86ed4e399d6c6dcfa5ecafc7" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 45697 } } peers { permanent_uuid: "bb218cc976d246bb887cc4fa3bbde42b" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 40827 } }
I20250815 13:55:40.492256 7247 leader_election.cc:290] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b [CANDIDATE]: Term 2 election: Requested vote from peers 6194ad8a8134493d97a72ac9cc40a705 (127.0.106.130:33295), efd284df86ed4e399d6c6dcfa5ecafc7 (127.0.106.131:45697)
I20250815 13:55:40.493038 7048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1b55d7048ac490bb56a174acd73c669" candidate_uuid: "bb218cc976d246bb887cc4fa3bbde42b" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "6194ad8a8134493d97a72ac9cc40a705"
I20250815 13:55:40.493468 7048 raft_consensus.cc:3058] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705 [term 1 FOLLOWER]: Advancing to term 2
I20250815 13:55:40.501515 7048 raft_consensus.cc:2466] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb218cc976d246bb887cc4fa3bbde42b in term 2.
I20250815 13:55:40.502748 6846 leader_election.cc:304] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6194ad8a8134493d97a72ac9cc40a705, bb218cc976d246bb887cc4fa3bbde42b; no voters:
I20250815 13:55:40.493562 7189 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1b55d7048ac490bb56a174acd73c669" candidate_uuid: "bb218cc976d246bb887cc4fa3bbde42b" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "efd284df86ed4e399d6c6dcfa5ecafc7" is_pre_election: true
I20250815 13:55:40.503667 7247 raft_consensus.cc:2802] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b [term 2 FOLLOWER]: Leader election won for term 2
I20250815 13:55:40.499310 7188 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1b55d7048ac490bb56a174acd73c669" candidate_uuid: "bb218cc976d246bb887cc4fa3bbde42b" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "efd284df86ed4e399d6c6dcfa5ecafc7"
I20250815 13:55:40.505721 7247 raft_consensus.cc:695] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b [term 2 LEADER]: Becoming Leader. State: Replica: bb218cc976d246bb887cc4fa3bbde42b, State: Running, Role: LEADER
I20250815 13:55:40.506826 7247 consensus_queue.cc:237] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6194ad8a8134493d97a72ac9cc40a705" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 33295 } } peers { permanent_uuid: "efd284df86ed4e399d6c6dcfa5ecafc7" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 45697 } } peers { permanent_uuid: "bb218cc976d246bb887cc4fa3bbde42b" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 40827 } }
W20250815 13:55:40.507218 6846 leader_election.cc:343] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer efd284df86ed4e399d6c6dcfa5ecafc7 (127.0.106.131:45697): Illegal state: must be running to vote when last-logged opid is not known
I20250815 13:55:40.519997 6769 catalog_manager.cc:5582] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b reported cstate change: term changed from 0 to 2, leader changed from <none> to bb218cc976d246bb887cc4fa3bbde42b (127.0.106.129), VOTER 6194ad8a8134493d97a72ac9cc40a705 (127.0.106.130) added, VOTER bb218cc976d246bb887cc4fa3bbde42b (127.0.106.129) added, VOTER efd284df86ed4e399d6c6dcfa5ecafc7 (127.0.106.131) added. New cstate: current_term: 2 leader_uuid: "bb218cc976d246bb887cc4fa3bbde42b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6194ad8a8134493d97a72ac9cc40a705" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 33295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efd284df86ed4e399d6c6dcfa5ecafc7" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 45697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb218cc976d246bb887cc4fa3bbde42b" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 40827 } health_report { overall_health: HEALTHY } } }
I20250815 13:55:40.520408 7234 heartbeater.cc:499] Master 127.0.106.190:45437 was elected leader, sending a full tablet report...
I20250815 13:55:40.899096 7048 raft_consensus.cc:1273] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705 [term 2 FOLLOWER]: Refusing update from remote peer bb218cc976d246bb887cc4fa3bbde42b: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250815 13:55:40.901278 7247 consensus_queue.cc:1035] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6194ad8a8134493d97a72ac9cc40a705" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 33295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.001s
W20250815 13:55:40.963253 6846 consensus_peers.cc:489] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b -> Peer efd284df86ed4e399d6c6dcfa5ecafc7 (127.0.106.131:45697): Couldn't send request to peer efd284df86ed4e399d6c6dcfa5ecafc7. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20250815 13:55:41.003513 426 scanner-internal.cc:458] Time spent opening tablet: real 1.439s user 0.004s sys 0.004s
I20250815 13:55:41.004653 6913 consensus_queue.cc:237] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 2.206, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "6194ad8a8134493d97a72ac9cc40a705" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 33295 } } peers { permanent_uuid: "bb218cc976d246bb887cc4fa3bbde42b" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 40827 } }
I20250815 13:55:41.009709 7048 raft_consensus.cc:1273] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705 [term 2 FOLLOWER]: Refusing update from remote peer bb218cc976d246bb887cc4fa3bbde42b: Log matching property violated. Preceding OpId in replica: term: 2 index: 206. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
I20250815 13:55:41.011235 7247 consensus_queue.cc:1035] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6194ad8a8134493d97a72ac9cc40a705" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 33295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.000s
I20250815 13:55:41.019524 7250 raft_consensus.cc:2953] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b [term 2 LEADER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER efd284df86ed4e399d6c6dcfa5ecafc7 (127.0.106.131) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "6194ad8a8134493d97a72ac9cc40a705" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 33295 } } peers { permanent_uuid: "bb218cc976d246bb887cc4fa3bbde42b" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 40827 } } }
I20250815 13:55:41.031237 7048 raft_consensus.cc:2953] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705 [term 2 FOLLOWER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER efd284df86ed4e399d6c6dcfa5ecafc7 (127.0.106.131) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "6194ad8a8134493d97a72ac9cc40a705" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 33295 } } peers { permanent_uuid: "bb218cc976d246bb887cc4fa3bbde42b" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 40827 } } }
I20250815 13:55:41.040381 6755 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet d1b55d7048ac490bb56a174acd73c669 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250815 13:55:41.046151 6770 catalog_manager.cc:5582] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b reported cstate change: config changed from index -1 to 207, VOTER efd284df86ed4e399d6c6dcfa5ecafc7 (127.0.106.131) evicted. New cstate: current_term: 2 leader_uuid: "bb218cc976d246bb887cc4fa3bbde42b" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "6194ad8a8134493d97a72ac9cc40a705" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 33295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb218cc976d246bb887cc4fa3bbde42b" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 40827 } health_report { overall_health: HEALTHY } } }
I20250815 13:55:41.103156 6913 consensus_queue.cc:237] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 2.207, Last appended by leader: 205, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "bb218cc976d246bb887cc4fa3bbde42b" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 40827 } }
I20250815 13:55:41.110422 7250 raft_consensus.cc:2953] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b [term 2 LEADER]: Committing config change with OpId 2.208: config changed from index 207 to 208, VOTER 6194ad8a8134493d97a72ac9cc40a705 (127.0.106.130) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "bb218cc976d246bb887cc4fa3bbde42b" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 40827 } } }
I20250815 13:55:41.133960 6755 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet d1b55d7048ac490bb56a174acd73c669 with cas_config_opid_index 207: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250815 13:55:41.139027 6769 catalog_manager.cc:5582] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b reported cstate change: config changed from index 207 to 208, VOTER 6194ad8a8134493d97a72ac9cc40a705 (127.0.106.130) evicted. New cstate: current_term: 2 leader_uuid: "bb218cc976d246bb887cc4fa3bbde42b" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "bb218cc976d246bb887cc4fa3bbde42b" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 40827 } health_report { overall_health: HEALTHY } } }
I20250815 13:55:41.202925 7028 tablet_service.cc:1515] Processing DeleteTablet for tablet d1b55d7048ac490bb56a174acd73c669 with delete_type TABLET_DATA_TOMBSTONED (TS 6194ad8a8134493d97a72ac9cc40a705 not found in new config with opid_index 208) from {username='slave'} at 127.0.0.1:47946
I20250815 13:55:41.206257 7169 tablet_service.cc:1515] Processing DeleteTablet for tablet d1b55d7048ac490bb56a174acd73c669 with delete_type TABLET_DATA_TOMBSTONED (TS efd284df86ed4e399d6c6dcfa5ecafc7 not found in new config with opid_index 207) from {username='slave'} at 127.0.0.1:41168
W20250815 13:55:41.217833 6755 catalog_manager.cc:4908] TS efd284df86ed4e399d6c6dcfa5ecafc7 (127.0.106.131:45697): delete failed for tablet d1b55d7048ac490bb56a174acd73c669 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1b55d7048ac490bb56a174acd73c669 already in progress: opening tablet
I20250815 13:55:41.221086 7272 tablet_replica.cc:331] stopping tablet replica
I20250815 13:55:41.222111 7272 raft_consensus.cc:2241] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250815 13:55:41.223037 7272 raft_consensus.cc:2270] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250815 13:55:41.256779 7272 ts_tablet_manager.cc:1905] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250815 13:55:41.274433 7272 ts_tablet_manager.cc:1918] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.207
I20250815 13:55:41.275110 7272 log.cc:1199] T d1b55d7048ac490bb56a174acd73c669 P 6194ad8a8134493d97a72ac9cc40a705: Deleting WAL directory at /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-1/wal/wals/d1b55d7048ac490bb56a174acd73c669
I20250815 13:55:41.277246 6755 catalog_manager.cc:4928] TS 6194ad8a8134493d97a72ac9cc40a705 (127.0.106.130:33295): tablet d1b55d7048ac490bb56a174acd73c669 (table pre_rebuild [id=caa7228106b94ee8b1bf1432d67e47bb]) successfully deleted
I20250815 13:55:42.078436 7028 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250815 13:55:42.082943 6893 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250815 13:55:42.111613 7169 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250815 13:55:42.275810 7127 tablet_bootstrap.cc:492] T d1b55d7048ac490bb56a174acd73c669 P efd284df86ed4e399d6c6dcfa5ecafc7: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 13:55:42.276962 7127 tablet_bootstrap.cc:492] T d1b55d7048ac490bb56a174acd73c669 P efd284df86ed4e399d6c6dcfa5ecafc7: Bootstrap complete.
I20250815 13:55:42.278743 7127 ts_tablet_manager.cc:1397] T d1b55d7048ac490bb56a174acd73c669 P efd284df86ed4e399d6c6dcfa5ecafc7: Time spent bootstrapping tablet: real 3.036s user 2.888s sys 0.072s
I20250815 13:55:42.286942 7127 raft_consensus.cc:357] T d1b55d7048ac490bb56a174acd73c669 P efd284df86ed4e399d6c6dcfa5ecafc7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6194ad8a8134493d97a72ac9cc40a705" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 33295 } } peers { permanent_uuid: "efd284df86ed4e399d6c6dcfa5ecafc7" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 45697 } } peers { permanent_uuid: "bb218cc976d246bb887cc4fa3bbde42b" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 40827 } }
I20250815 13:55:42.290300 7127 raft_consensus.cc:738] T d1b55d7048ac490bb56a174acd73c669 P efd284df86ed4e399d6c6dcfa5ecafc7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: efd284df86ed4e399d6c6dcfa5ecafc7, State: Initialized, Role: FOLLOWER
I20250815 13:55:42.291435 7127 consensus_queue.cc:260] T d1b55d7048ac490bb56a174acd73c669 P efd284df86ed4e399d6c6dcfa5ecafc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6194ad8a8134493d97a72ac9cc40a705" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 33295 } } peers { permanent_uuid: "efd284df86ed4e399d6c6dcfa5ecafc7" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 45697 } } peers { permanent_uuid: "bb218cc976d246bb887cc4fa3bbde42b" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 40827 } }
I20250815 13:55:42.324198 7127 ts_tablet_manager.cc:1428] T d1b55d7048ac490bb56a174acd73c669 P efd284df86ed4e399d6c6dcfa5ecafc7: Time spent starting tablet: real 0.045s user 0.028s sys 0.016s
I20250815 13:55:42.332087 7169 tablet_service.cc:1515] Processing DeleteTablet for tablet d1b55d7048ac490bb56a174acd73c669 with delete_type TABLET_DATA_TOMBSTONED (Replica has no consensus available (current committed config index is 208)) from {username='slave'} at 127.0.0.1:41168
I20250815 13:55:42.354610 7301 tablet_replica.cc:331] stopping tablet replica
I20250815 13:55:42.357712 7301 raft_consensus.cc:2241] T d1b55d7048ac490bb56a174acd73c669 P efd284df86ed4e399d6c6dcfa5ecafc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250815 13:55:42.359618 7301 raft_consensus.cc:2270] T d1b55d7048ac490bb56a174acd73c669 P efd284df86ed4e399d6c6dcfa5ecafc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250815 13:55:42.412325 7301 ts_tablet_manager.cc:1905] T d1b55d7048ac490bb56a174acd73c669 P efd284df86ed4e399d6c6dcfa5ecafc7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250815 13:55:42.432268 7301 ts_tablet_manager.cc:1918] T d1b55d7048ac490bb56a174acd73c669 P efd284df86ed4e399d6c6dcfa5ecafc7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.205
I20250815 13:55:42.433220 7301 log.cc:1199] T d1b55d7048ac490bb56a174acd73c669 P efd284df86ed4e399d6c6dcfa5ecafc7: Deleting WAL directory at /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-2/wal/wals/d1b55d7048ac490bb56a174acd73c669
I20250815 13:55:42.437618 6755 catalog_manager.cc:4928] TS efd284df86ed4e399d6c6dcfa5ecafc7 (127.0.106.131:45697): tablet d1b55d7048ac490bb56a174acd73c669 (table pre_rebuild [id=caa7228106b94ee8b1bf1432d67e47bb]) successfully deleted
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
6741580a9c154f2b9587bd3c456696e0 | 127.0.106.190:45437 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.0.106.148:35301 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
6194ad8a8134493d97a72ac9cc40a705 | 127.0.106.130:33295 | HEALTHY | <none> | 0 | 0
bb218cc976d246bb887cc4fa3bbde42b | 127.0.106.129:40827 | HEALTHY | <none> | 1 | 0
efd284df86ed4e399d6c6dcfa5ecafc7 | 127.0.106.131:45697 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.0.106.129 | experimental | 127.0.106.129:40827
local_ip_for_outbound_sockets | 127.0.106.130 | experimental | 127.0.106.130:33295
local_ip_for_outbound_sockets | 127.0.106.131 | experimental | 127.0.106.131:45697
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-0/data/info.pb | hidden | 127.0.106.129:40827
server_dump_info_path | /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-1/data/info.pb | hidden | 127.0.106.130:33295
server_dump_info_path | /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-2/data/info.pb | hidden | 127.0.106.131:45697
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.0.106.148:35301 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------+----+---------+---------------+---------+------------+------------------+-------------
pre_rebuild | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 0
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 1
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250815 13:55:42.581465 426 log_verifier.cc:126] Checking tablet d1b55d7048ac490bb56a174acd73c669
I20250815 13:55:42.847960 426 log_verifier.cc:177] Verified matching terms for 208 ops in tablet d1b55d7048ac490bb56a174acd73c669
I20250815 13:55:42.851207 6770 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46516:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250815 13:55:42.851837 6770 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46516:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250815 13:55:42.864946 6770 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 6741580a9c154f2b9587bd3c456696e0: Sending DeleteTablet for 1 replicas of tablet d1b55d7048ac490bb56a174acd73c669
I20250815 13:55:42.867545 426 test_util.cc:276] Using random seed: -1091661773
I20250815 13:55:42.868302 6893 tablet_service.cc:1515] Processing DeleteTablet for tablet d1b55d7048ac490bb56a174acd73c669 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-08-15 13:55:42 UTC) from {username='slave'} at 127.0.0.1:59318
I20250815 13:55:42.871439 7308 tablet_replica.cc:331] stopping tablet replica
I20250815 13:55:42.872952 7308 raft_consensus.cc:2241] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b [term 2 LEADER]: Raft consensus shutting down.
I20250815 13:55:42.875434 7308 raft_consensus.cc:2270] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250815 13:55:42.905771 6770 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46558:
name: "post_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250815 13:55:42.909188 6770 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table post_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250815 13:55:42.928715 7308 ts_tablet_manager.cc:1905] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250815 13:55:42.936733 7028 tablet_service.cc:1468] Processing CreateTablet for tablet 6f66fa1cc1f24329a9029cf85d097825 (DEFAULT_TABLE table=post_rebuild [id=c0becf05eec848a387b46701e66c524f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 13:55:42.938349 7028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f66fa1cc1f24329a9029cf85d097825. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:55:42.941406 7169 tablet_service.cc:1468] Processing CreateTablet for tablet 6f66fa1cc1f24329a9029cf85d097825 (DEFAULT_TABLE table=post_rebuild [id=c0becf05eec848a387b46701e66c524f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 13:55:42.941708 6893 tablet_service.cc:1468] Processing CreateTablet for tablet 6f66fa1cc1f24329a9029cf85d097825 (DEFAULT_TABLE table=post_rebuild [id=c0becf05eec848a387b46701e66c524f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 13:55:42.942800 7169 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f66fa1cc1f24329a9029cf85d097825. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:55:42.943678 6893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f66fa1cc1f24329a9029cf85d097825. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 13:55:42.960381 7308 ts_tablet_manager.cc:1918] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.208
I20250815 13:55:42.969094 7308 log.cc:1199] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b: Deleting WAL directory at /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-0/wal/wals/d1b55d7048ac490bb56a174acd73c669
I20250815 13:55:42.970778 7308 ts_tablet_manager.cc:1939] T d1b55d7048ac490bb56a174acd73c669 P bb218cc976d246bb887cc4fa3bbde42b: Deleting consensus metadata
I20250815 13:55:42.971971 7315 tablet_bootstrap.cc:492] T 6f66fa1cc1f24329a9029cf85d097825 P bb218cc976d246bb887cc4fa3bbde42b: Bootstrap starting.
I20250815 13:55:42.975589 6755 catalog_manager.cc:4928] TS bb218cc976d246bb887cc4fa3bbde42b (127.0.106.129:40827): tablet d1b55d7048ac490bb56a174acd73c669 (table pre_rebuild [id=caa7228106b94ee8b1bf1432d67e47bb]) successfully deleted
I20250815 13:55:42.976066 7316 tablet_bootstrap.cc:492] T 6f66fa1cc1f24329a9029cf85d097825 P efd284df86ed4e399d6c6dcfa5ecafc7: Bootstrap starting.
I20250815 13:55:42.984350 7317 tablet_bootstrap.cc:492] T 6f66fa1cc1f24329a9029cf85d097825 P 6194ad8a8134493d97a72ac9cc40a705: Bootstrap starting.
I20250815 13:55:42.985004 7316 tablet_bootstrap.cc:654] T 6f66fa1cc1f24329a9029cf85d097825 P efd284df86ed4e399d6c6dcfa5ecafc7: Neither blocks nor log segments found. Creating new log.
I20250815 13:55:42.989813 7315 tablet_bootstrap.cc:654] T 6f66fa1cc1f24329a9029cf85d097825 P bb218cc976d246bb887cc4fa3bbde42b: Neither blocks nor log segments found. Creating new log.
I20250815 13:55:42.995863 7317 tablet_bootstrap.cc:654] T 6f66fa1cc1f24329a9029cf85d097825 P 6194ad8a8134493d97a72ac9cc40a705: Neither blocks nor log segments found. Creating new log.
I20250815 13:55:43.005576 7316 tablet_bootstrap.cc:492] T 6f66fa1cc1f24329a9029cf85d097825 P efd284df86ed4e399d6c6dcfa5ecafc7: No bootstrap required, opened a new log
I20250815 13:55:43.006047 7316 ts_tablet_manager.cc:1397] T 6f66fa1cc1f24329a9029cf85d097825 P efd284df86ed4e399d6c6dcfa5ecafc7: Time spent bootstrapping tablet: real 0.030s user 0.021s sys 0.009s
I20250815 13:55:43.006443 7315 tablet_bootstrap.cc:492] T 6f66fa1cc1f24329a9029cf85d097825 P bb218cc976d246bb887cc4fa3bbde42b: No bootstrap required, opened a new log
I20250815 13:55:43.007086 7315 ts_tablet_manager.cc:1397] T 6f66fa1cc1f24329a9029cf85d097825 P bb218cc976d246bb887cc4fa3bbde42b: Time spent bootstrapping tablet: real 0.038s user 0.012s sys 0.013s
I20250815 13:55:43.008265 7317 tablet_bootstrap.cc:492] T 6f66fa1cc1f24329a9029cf85d097825 P 6194ad8a8134493d97a72ac9cc40a705: No bootstrap required, opened a new log
I20250815 13:55:43.008764 7317 ts_tablet_manager.cc:1397] T 6f66fa1cc1f24329a9029cf85d097825 P 6194ad8a8134493d97a72ac9cc40a705: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.005s
I20250815 13:55:43.009121 7316 raft_consensus.cc:357] T 6f66fa1cc1f24329a9029cf85d097825 P efd284df86ed4e399d6c6dcfa5ecafc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6194ad8a8134493d97a72ac9cc40a705" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 33295 } } peers { permanent_uuid: "efd284df86ed4e399d6c6dcfa5ecafc7" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 45697 } } peers { permanent_uuid: "bb218cc976d246bb887cc4fa3bbde42b" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 40827 } }
I20250815 13:55:43.009922 7316 raft_consensus.cc:383] T 6f66fa1cc1f24329a9029cf85d097825 P efd284df86ed4e399d6c6dcfa5ecafc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 13:55:43.010242 7316 raft_consensus.cc:738] T 6f66fa1cc1f24329a9029cf85d097825 P efd284df86ed4e399d6c6dcfa5ecafc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efd284df86ed4e399d6c6dcfa5ecafc7, State: Initialized, Role: FOLLOWER
I20250815 13:55:43.010545 7315 raft_consensus.cc:357] T 6f66fa1cc1f24329a9029cf85d097825 P bb218cc976d246bb887cc4fa3bbde42b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6194ad8a8134493d97a72ac9cc40a705" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 33295 } } peers { permanent_uuid: "efd284df86ed4e399d6c6dcfa5ecafc7" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 45697 } } peers { permanent_uuid: "bb218cc976d246bb887cc4fa3bbde42b" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 40827 } }
I20250815 13:55:43.011560 7315 raft_consensus.cc:383] T 6f66fa1cc1f24329a9029cf85d097825 P bb218cc976d246bb887cc4fa3bbde42b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 13:55:43.011101 7316 consensus_queue.cc:260] T 6f66fa1cc1f24329a9029cf85d097825 P efd284df86ed4e399d6c6dcfa5ecafc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6194ad8a8134493d97a72ac9cc40a705" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 33295 } } peers { permanent_uuid: "efd284df86ed4e399d6c6dcfa5ecafc7" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 45697 } } peers { permanent_uuid: "bb218cc976d246bb887cc4fa3bbde42b" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 40827 } }
I20250815 13:55:43.011927 7315 raft_consensus.cc:738] T 6f66fa1cc1f24329a9029cf85d097825 P bb218cc976d246bb887cc4fa3bbde42b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb218cc976d246bb887cc4fa3bbde42b, State: Initialized, Role: FOLLOWER
I20250815 13:55:43.011754 7317 raft_consensus.cc:357] T 6f66fa1cc1f24329a9029cf85d097825 P 6194ad8a8134493d97a72ac9cc40a705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6194ad8a8134493d97a72ac9cc40a705" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 33295 } } peers { permanent_uuid: "efd284df86ed4e399d6c6dcfa5ecafc7" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 45697 } } peers { permanent_uuid: "bb218cc976d246bb887cc4fa3bbde42b" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 40827 } }
I20250815 13:55:43.012676 7317 raft_consensus.cc:383] T 6f66fa1cc1f24329a9029cf85d097825 P 6194ad8a8134493d97a72ac9cc40a705 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 13:55:43.013016 7317 raft_consensus.cc:738] T 6f66fa1cc1f24329a9029cf85d097825 P 6194ad8a8134493d97a72ac9cc40a705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6194ad8a8134493d97a72ac9cc40a705, State: Initialized, Role: FOLLOWER
I20250815 13:55:43.012732 7315 consensus_queue.cc:260] T 6f66fa1cc1f24329a9029cf85d097825 P bb218cc976d246bb887cc4fa3bbde42b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6194ad8a8134493d97a72ac9cc40a705" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 33295 } } peers { permanent_uuid: "efd284df86ed4e399d6c6dcfa5ecafc7" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 45697 } } peers { permanent_uuid: "bb218cc976d246bb887cc4fa3bbde42b" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 40827 } }
I20250815 13:55:43.014036 7317 consensus_queue.cc:260] T 6f66fa1cc1f24329a9029cf85d097825 P 6194ad8a8134493d97a72ac9cc40a705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6194ad8a8134493d97a72ac9cc40a705" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 33295 } } peers { permanent_uuid: "efd284df86ed4e399d6c6dcfa5ecafc7" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 45697 } } peers { permanent_uuid: "bb218cc976d246bb887cc4fa3bbde42b" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 40827 } }
I20250815 13:55:43.020326 7316 ts_tablet_manager.cc:1428] T 6f66fa1cc1f24329a9029cf85d097825 P efd284df86ed4e399d6c6dcfa5ecafc7: Time spent starting tablet: real 0.014s user 0.011s sys 0.000s
I20250815 13:55:43.020817 7317 ts_tablet_manager.cc:1428] T 6f66fa1cc1f24329a9029cf85d097825 P 6194ad8a8134493d97a72ac9cc40a705: Time spent starting tablet: real 0.012s user 0.009s sys 0.000s
I20250815 13:55:43.024339 7315 ts_tablet_manager.cc:1428] T 6f66fa1cc1f24329a9029cf85d097825 P bb218cc976d246bb887cc4fa3bbde42b: Time spent starting tablet: real 0.017s user 0.011s sys 0.000s
I20250815 13:55:43.110733 7323 raft_consensus.cc:491] T 6f66fa1cc1f24329a9029cf85d097825 P 6194ad8a8134493d97a72ac9cc40a705 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 13:55:43.111327 7323 raft_consensus.cc:513] T 6f66fa1cc1f24329a9029cf85d097825 P 6194ad8a8134493d97a72ac9cc40a705 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6194ad8a8134493d97a72ac9cc40a705" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 33295 } } peers { permanent_uuid: "efd284df86ed4e399d6c6dcfa5ecafc7" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 45697 } } peers { permanent_uuid: "bb218cc976d246bb887cc4fa3bbde42b" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 40827 } }
I20250815 13:55:43.115571 7323 leader_election.cc:290] T 6f66fa1cc1f24329a9029cf85d097825 P 6194ad8a8134493d97a72ac9cc40a705 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efd284df86ed4e399d6c6dcfa5ecafc7 (127.0.106.131:45697), bb218cc976d246bb887cc4fa3bbde42b (127.0.106.129:40827)
I20250815 13:55:43.133468 7189 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f66fa1cc1f24329a9029cf85d097825" candidate_uuid: "6194ad8a8134493d97a72ac9cc40a705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efd284df86ed4e399d6c6dcfa5ecafc7" is_pre_election: true
I20250815 13:55:43.134253 7189 raft_consensus.cc:2466] T 6f66fa1cc1f24329a9029cf85d097825 P efd284df86ed4e399d6c6dcfa5ecafc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6194ad8a8134493d97a72ac9cc40a705 in term 0.
I20250815 13:55:43.135926 6981 leader_election.cc:304] T 6f66fa1cc1f24329a9029cf85d097825 P 6194ad8a8134493d97a72ac9cc40a705 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6194ad8a8134493d97a72ac9cc40a705, efd284df86ed4e399d6c6dcfa5ecafc7; no voters:
I20250815 13:55:43.136863 7323 raft_consensus.cc:2802] T 6f66fa1cc1f24329a9029cf85d097825 P 6194ad8a8134493d97a72ac9cc40a705 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250815 13:55:43.137187 7323 raft_consensus.cc:491] T 6f66fa1cc1f24329a9029cf85d097825 P 6194ad8a8134493d97a72ac9cc40a705 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 13:55:43.137471 7323 raft_consensus.cc:3058] T 6f66fa1cc1f24329a9029cf85d097825 P 6194ad8a8134493d97a72ac9cc40a705 [term 0 FOLLOWER]: Advancing to term 1
I20250815 13:55:43.145931 6913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f66fa1cc1f24329a9029cf85d097825" candidate_uuid: "6194ad8a8134493d97a72ac9cc40a705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb218cc976d246bb887cc4fa3bbde42b" is_pre_election: true
I20250815 13:55:43.146965 6913 raft_consensus.cc:2466] T 6f66fa1cc1f24329a9029cf85d097825 P bb218cc976d246bb887cc4fa3bbde42b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6194ad8a8134493d97a72ac9cc40a705 in term 0.
I20250815 13:55:43.148814 7323 raft_consensus.cc:513] T 6f66fa1cc1f24329a9029cf85d097825 P 6194ad8a8134493d97a72ac9cc40a705 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6194ad8a8134493d97a72ac9cc40a705" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 33295 } } peers { permanent_uuid: "efd284df86ed4e399d6c6dcfa5ecafc7" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 45697 } } peers { permanent_uuid: "bb218cc976d246bb887cc4fa3bbde42b" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 40827 } }
I20250815 13:55:43.151832 7189 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f66fa1cc1f24329a9029cf85d097825" candidate_uuid: "6194ad8a8134493d97a72ac9cc40a705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efd284df86ed4e399d6c6dcfa5ecafc7"
I20250815 13:55:43.152360 7189 raft_consensus.cc:3058] T 6f66fa1cc1f24329a9029cf85d097825 P efd284df86ed4e399d6c6dcfa5ecafc7 [term 0 FOLLOWER]: Advancing to term 1
W20250815 13:55:43.156211 6959 tablet.cc:2378] T 6f66fa1cc1f24329a9029cf85d097825 P bb218cc976d246bb887cc4fa3bbde42b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 13:55:43.159024 7189 raft_consensus.cc:2466] T 6f66fa1cc1f24329a9029cf85d097825 P efd284df86ed4e399d6c6dcfa5ecafc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6194ad8a8134493d97a72ac9cc40a705 in term 1.
I20250815 13:55:43.160585 6981 leader_election.cc:304] T 6f66fa1cc1f24329a9029cf85d097825 P 6194ad8a8134493d97a72ac9cc40a705 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6194ad8a8134493d97a72ac9cc40a705, efd284df86ed4e399d6c6dcfa5ecafc7; no voters:
I20250815 13:55:43.162823 7323 leader_election.cc:290] T 6f66fa1cc1f24329a9029cf85d097825 P 6194ad8a8134493d97a72ac9cc40a705 [CANDIDATE]: Term 1 election: Requested vote from peers efd284df86ed4e399d6c6dcfa5ecafc7 (127.0.106.131:45697), bb218cc976d246bb887cc4fa3bbde42b (127.0.106.129:40827)
I20250815 13:55:43.164186 7323 raft_consensus.cc:2802] T 6f66fa1cc1f24329a9029cf85d097825 P 6194ad8a8134493d97a72ac9cc40a705 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 13:55:43.165177 7323 raft_consensus.cc:695] T 6f66fa1cc1f24329a9029cf85d097825 P 6194ad8a8134493d97a72ac9cc40a705 [term 1 LEADER]: Becoming Leader. State: Replica: 6194ad8a8134493d97a72ac9cc40a705, State: Running, Role: LEADER
I20250815 13:55:43.166574 7323 consensus_queue.cc:237] T 6f66fa1cc1f24329a9029cf85d097825 P 6194ad8a8134493d97a72ac9cc40a705 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6194ad8a8134493d97a72ac9cc40a705" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 33295 } } peers { permanent_uuid: "efd284df86ed4e399d6c6dcfa5ecafc7" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 45697 } } peers { permanent_uuid: "bb218cc976d246bb887cc4fa3bbde42b" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 40827 } }
I20250815 13:55:43.168216 6913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f66fa1cc1f24329a9029cf85d097825" candidate_uuid: "6194ad8a8134493d97a72ac9cc40a705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb218cc976d246bb887cc4fa3bbde42b"
I20250815 13:55:43.168668 6913 raft_consensus.cc:3058] T 6f66fa1cc1f24329a9029cf85d097825 P bb218cc976d246bb887cc4fa3bbde42b [term 0 FOLLOWER]: Advancing to term 1
I20250815 13:55:43.176555 6913 raft_consensus.cc:2466] T 6f66fa1cc1f24329a9029cf85d097825 P bb218cc976d246bb887cc4fa3bbde42b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6194ad8a8134493d97a72ac9cc40a705 in term 1.
I20250815 13:55:43.184569 6769 catalog_manager.cc:5582] T 6f66fa1cc1f24329a9029cf85d097825 P 6194ad8a8134493d97a72ac9cc40a705 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6194ad8a8134493d97a72ac9cc40a705 (127.0.106.130). New cstate: current_term: 1 leader_uuid: "6194ad8a8134493d97a72ac9cc40a705" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6194ad8a8134493d97a72ac9cc40a705" member_type: VOTER last_known_addr { host: "127.0.106.130" port: 33295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efd284df86ed4e399d6c6dcfa5ecafc7" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 45697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb218cc976d246bb887cc4fa3bbde42b" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 40827 } health_report { overall_health: UNKNOWN } } }
W20250815 13:55:43.220176 7094 tablet.cc:2378] T 6f66fa1cc1f24329a9029cf85d097825 P 6194ad8a8134493d97a72ac9cc40a705: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250815 13:55:43.220301 7235 tablet.cc:2378] T 6f66fa1cc1f24329a9029cf85d097825 P efd284df86ed4e399d6c6dcfa5ecafc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 13:55:43.383486 7189 raft_consensus.cc:1273] T 6f66fa1cc1f24329a9029cf85d097825 P efd284df86ed4e399d6c6dcfa5ecafc7 [term 1 FOLLOWER]: Refusing update from remote peer 6194ad8a8134493d97a72ac9cc40a705: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250815 13:55:43.383591 6913 raft_consensus.cc:1273] T 6f66fa1cc1f24329a9029cf85d097825 P bb218cc976d246bb887cc4fa3bbde42b [term 1 FOLLOWER]: Refusing update from remote peer 6194ad8a8134493d97a72ac9cc40a705: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250815 13:55:43.385442 7323 consensus_queue.cc:1035] T 6f66fa1cc1f24329a9029cf85d097825 P 6194ad8a8134493d97a72ac9cc40a705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd284df86ed4e399d6c6dcfa5ecafc7" member_type: VOTER last_known_addr { host: "127.0.106.131" port: 45697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250815 13:55:43.386397 7328 consensus_queue.cc:1035] T 6f66fa1cc1f24329a9029cf85d097825 P 6194ad8a8134493d97a72ac9cc40a705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb218cc976d246bb887cc4fa3bbde42b" member_type: VOTER last_known_addr { host: "127.0.106.129" port: 40827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250815 13:55:43.422679 7336 mvcc.cc:204] Tried to move back new op lower bound from 7189570123276771328 to 7189570122423758848. Current Snapshot: MvccSnapshot[applied={T|T < 7189570123276771328}]
I20250815 13:55:43.434638 7339 mvcc.cc:204] Tried to move back new op lower bound from 7189570123276771328 to 7189570122423758848. Current Snapshot: MvccSnapshot[applied={T|T < 7189570123276771328}]
W20250815 13:55:44.711481 7310 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.0.106.130:33295, user_credentials={real_user=slave}} blocked reactor thread for 43930.4us
I20250815 13:55:48.525539 7028 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250815 13:55:48.532502 6893 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250815 13:55:48.541472 7169 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
6741580a9c154f2b9587bd3c456696e0 | 127.0.106.190:45437 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.0.106.148:35301 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
6194ad8a8134493d97a72ac9cc40a705 | 127.0.106.130:33295 | HEALTHY | <none> | 1 | 0
bb218cc976d246bb887cc4fa3bbde42b | 127.0.106.129:40827 | HEALTHY | <none> | 0 | 0
efd284df86ed4e399d6c6dcfa5ecafc7 | 127.0.106.131:45697 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.0.106.129 | experimental | 127.0.106.129:40827
local_ip_for_outbound_sockets | 127.0.106.130 | experimental | 127.0.106.130:33295
local_ip_for_outbound_sockets | 127.0.106.131 | experimental | 127.0.106.131:45697
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-0/data/info.pb | hidden | 127.0.106.129:40827
server_dump_info_path | /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-1/data/info.pb | hidden | 127.0.106.130:33295
server_dump_info_path | /tmp/dist-test-taskRjMaS9/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755265962145449-426-0/minicluster-data/ts-2/data/info.pb | hidden | 127.0.106.131:45697
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.0.106.148:35301 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
--------------+----+---------+---------------+---------+------------+------------------+-------------
post_rebuild | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250815 13:55:48.758109 426 log_verifier.cc:126] Checking tablet 6f66fa1cc1f24329a9029cf85d097825
I20250815 13:55:49.604244 426 log_verifier.cc:177] Verified matching terms for 205 ops in tablet 6f66fa1cc1f24329a9029cf85d097825
I20250815 13:55:49.605433 426 log_verifier.cc:126] Checking tablet d1b55d7048ac490bb56a174acd73c669
I20250815 13:55:49.605791 426 log_verifier.cc:177] Verified matching terms for 0 ops in tablet d1b55d7048ac490bb56a174acd73c669
I20250815 13:55:49.636530 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 6807
I20250815 13:55:49.681293 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 6962
I20250815 13:55:49.719415 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 7097
I20250815 13:55:49.758342 426 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRjMaS9/build/tsan/bin/kudu with pid 6737
2025-08-15T13:55:49Z chronyd exiting
[ OK ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0 (35878 ms)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest (35878 ms total)
[----------] Global test environment tear-down
[==========] 9 tests from 5 test suites ran. (187600 ms total)
[ PASSED ] 8 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AdminCliTest.TestRebuildTables
1 FAILED TEST
I20250815 13:55:49.830457 426 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/meta_cache.cc:302: suppressed but not reported on 2 messages since previous log ~48 seconds ago