Note: This is test shard 6 of 8.
[==========] Running 9 tests from 5 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from AdminCliTest
[ RUN ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250623 01:57:33.621881 14510 test_util.cc:276] Using random seed: -1996097025
W20250623 01:57:34.742791 14510 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.078s user 0.426s sys 0.652s
W20250623 01:57:34.743140 14510 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.079s user 0.426s sys 0.652s
I20250623 01:57:34.755928 14510 ts_itest-base.cc:115] Starting cluster with:
I20250623 01:57:34.756130 14510 ts_itest-base.cc:116] --------------
I20250623 01:57:34.756284 14510 ts_itest-base.cc:117] 4 tablet servers
I20250623 01:57:34.756433 14510 ts_itest-base.cc:118] 3 replicas per TS
I20250623 01:57:34.756582 14510 ts_itest-base.cc:119] --------------
2025-06-23T01:57:34Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-23T01:57:34Z Disabled control of system clock
I20250623 01:57:34.788537 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.43.190:37567
--webserver_interface=127.14.43.190
--webserver_port=0
--builtin_ntp_servers=127.14.43.148:45451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.43.190:37567 with env {}
W20250623 01:57:35.071077 14524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:57:35.071647 14524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:57:35.072096 14524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:57:35.102700 14524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250623 01:57:35.103014 14524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:57:35.103253 14524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250623 01:57:35.103469 14524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250623 01:57:35.137801 14524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:45451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.43.190:37567
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.43.190:37567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.14.43.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:57:35.139106 14524 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:57:35.140707 14524 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:57:35.155066 14531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:57:35.156587 14530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:57:35.157397 14524 server_base.cc:1048] running on GCE node
W20250623 01:57:35.156859 14533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:57:36.258652 14524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:57:36.261210 14524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:57:36.262639 14524 hybrid_clock.cc:648] HybridClock initialized: now 1750643856262600 us; error 55 us; skew 500 ppm
I20250623 01:57:36.263535 14524 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:57:36.269649 14524 webserver.cc:469] Webserver started at http://127.14.43.190:46469/ using document root <none> and password file <none>
I20250623 01:57:36.270622 14524 fs_manager.cc:362] Metadata directory not provided
I20250623 01:57:36.270819 14524 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:57:36.271242 14524 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:57:36.275547 14524 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "617bf001a689445bb005a7acaadbad73"
format_stamp: "Formatted at 2025-06-23 01:57:36 on dist-test-slave-9rpg"
I20250623 01:57:36.276660 14524 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "617bf001a689445bb005a7acaadbad73"
format_stamp: "Formatted at 2025-06-23 01:57:36 on dist-test-slave-9rpg"
I20250623 01:57:36.283604 14524 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20250623 01:57:36.288985 14540 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:36.290107 14524 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250623 01:57:36.290413 14524 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
uuid: "617bf001a689445bb005a7acaadbad73"
format_stamp: "Formatted at 2025-06-23 01:57:36 on dist-test-slave-9rpg"
I20250623 01:57:36.290720 14524 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:57:36.344790 14524 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:57:36.346246 14524 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:57:36.346647 14524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:57:36.418839 14524 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.190:37567
I20250623 01:57:36.418921 14591 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.190:37567 every 8 connection(s)
I20250623 01:57:36.421470 14524 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250623 01:57:36.426779 14592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:36.429836 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 14524
I20250623 01:57:36.430261 14510 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250623 01:57:36.445003 14592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73: Bootstrap starting.
I20250623 01:57:36.450224 14592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:36.451906 14592 log.cc:826] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73: Log is configured to *not* fsync() on all Append() calls
I20250623 01:57:36.456437 14592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73: No bootstrap required, opened a new log
I20250623 01:57:36.474334 14592 raft_consensus.cc:357] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617bf001a689445bb005a7acaadbad73" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 37567 } }
I20250623 01:57:36.474963 14592 raft_consensus.cc:383] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:36.475200 14592 raft_consensus.cc:738] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 617bf001a689445bb005a7acaadbad73, State: Initialized, Role: FOLLOWER
I20250623 01:57:36.475821 14592 consensus_queue.cc:260] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617bf001a689445bb005a7acaadbad73" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 37567 } }
I20250623 01:57:36.476310 14592 raft_consensus.cc:397] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250623 01:57:36.476557 14592 raft_consensus.cc:491] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250623 01:57:36.476835 14592 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:36.481081 14592 raft_consensus.cc:513] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617bf001a689445bb005a7acaadbad73" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 37567 } }
I20250623 01:57:36.481788 14592 leader_election.cc:304] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 617bf001a689445bb005a7acaadbad73; no voters:
I20250623 01:57:36.483348 14592 leader_election.cc:290] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250623 01:57:36.484050 14597 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:57:36.485905 14597 raft_consensus.cc:695] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73 [term 1 LEADER]: Becoming Leader. State: Replica: 617bf001a689445bb005a7acaadbad73, State: Running, Role: LEADER
I20250623 01:57:36.486753 14597 consensus_queue.cc:237] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617bf001a689445bb005a7acaadbad73" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 37567 } }
I20250623 01:57:36.488488 14592 sys_catalog.cc:564] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73 [sys.catalog]: configured and running, proceeding with master startup.
I20250623 01:57:36.496155 14599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "617bf001a689445bb005a7acaadbad73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617bf001a689445bb005a7acaadbad73" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 37567 } } }
I20250623 01:57:36.496894 14599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73 [sys.catalog]: This master's current role is: LEADER
I20250623 01:57:36.497593 14598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 617bf001a689445bb005a7acaadbad73. Latest consensus state: current_term: 1 leader_uuid: "617bf001a689445bb005a7acaadbad73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617bf001a689445bb005a7acaadbad73" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 37567 } } }
I20250623 01:57:36.498137 14598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73 [sys.catalog]: This master's current role is: LEADER
I20250623 01:57:36.503415 14606 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250623 01:57:36.513744 14606 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250623 01:57:36.531294 14606 catalog_manager.cc:1349] Generated new cluster ID: 007b950910b243e385ab834806ef7f1a
I20250623 01:57:36.531599 14606 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250623 01:57:36.556305 14606 catalog_manager.cc:1372] Generated new certificate authority record
I20250623 01:57:36.557746 14606 catalog_manager.cc:1506] Loading token signing keys...
I20250623 01:57:36.569089 14606 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73: Generated new TSK 0
I20250623 01:57:36.570045 14606 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250623 01:57:36.589644 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.43.129:0
--local_ip_for_outbound_sockets=127.14.43.129
--webserver_interface=127.14.43.129
--webserver_port=0
--tserver_master_addrs=127.14.43.190:37567
--builtin_ntp_servers=127.14.43.148:45451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250623 01:57:36.874203 14616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:57:36.874694 14616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:57:36.875166 14616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:57:36.906288 14616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:57:36.907182 14616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.43.129
I20250623 01:57:36.942297 14616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:45451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.43.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.14.43.129
--webserver_port=0
--tserver_master_addrs=127.14.43.190:37567
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.43.129
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:57:36.943593 14616 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:57:36.945210 14616 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:57:36.962361 14623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:57:36.965631 14616 server_base.cc:1048] running on GCE node
W20250623 01:57:36.963819 14625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:57:36.963084 14622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:57:38.041841 14616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:57:38.044512 14616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:57:38.045946 14616 hybrid_clock.cc:648] HybridClock initialized: now 1750643858045894 us; error 47 us; skew 500 ppm
I20250623 01:57:38.047000 14616 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:57:38.054127 14616 webserver.cc:469] Webserver started at http://127.14.43.129:36427/ using document root <none> and password file <none>
I20250623 01:57:38.055301 14616 fs_manager.cc:362] Metadata directory not provided
I20250623 01:57:38.055557 14616 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:57:38.056097 14616 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:57:38.064193 14616 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "9ce9e2fa9e934e74b7e50719bb379eec"
format_stamp: "Formatted at 2025-06-23 01:57:38 on dist-test-slave-9rpg"
I20250623 01:57:38.065729 14616 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "9ce9e2fa9e934e74b7e50719bb379eec"
format_stamp: "Formatted at 2025-06-23 01:57:38 on dist-test-slave-9rpg"
I20250623 01:57:38.074808 14616 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20250623 01:57:38.081807 14632 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:38.082975 14616 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250623 01:57:38.083355 14616 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "9ce9e2fa9e934e74b7e50719bb379eec"
format_stamp: "Formatted at 2025-06-23 01:57:38 on dist-test-slave-9rpg"
I20250623 01:57:38.083767 14616 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:57:38.173251 14616 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:57:38.175104 14616 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:57:38.175617 14616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:57:38.178691 14616 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:57:38.184013 14616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250623 01:57:38.184283 14616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:38.184557 14616 ts_tablet_manager.cc:610] Registered 0 tablets
I20250623 01:57:38.184757 14616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:38.315814 14616 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.129:37411
I20250623 01:57:38.315924 14744 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.129:37411 every 8 connection(s)
I20250623 01:57:38.318282 14616 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250623 01:57:38.324791 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 14616
I20250623 01:57:38.325201 14510 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250623 01:57:38.331218 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.43.130:0
--local_ip_for_outbound_sockets=127.14.43.130
--webserver_interface=127.14.43.130
--webserver_port=0
--tserver_master_addrs=127.14.43.190:37567
--builtin_ntp_servers=127.14.43.148:45451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250623 01:57:38.339393 14745 heartbeater.cc:344] Connected to a master server at 127.14.43.190:37567
I20250623 01:57:38.339783 14745 heartbeater.cc:461] Registering TS with master...
I20250623 01:57:38.340778 14745 heartbeater.cc:507] Master 127.14.43.190:37567 requested a full tablet report, sending...
I20250623 01:57:38.343199 14557 ts_manager.cc:194] Registered new tserver with Master: 9ce9e2fa9e934e74b7e50719bb379eec (127.14.43.129:37411)
I20250623 01:57:38.345082 14557 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.43.129:48225
W20250623 01:57:38.616377 14749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:57:38.616876 14749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:57:38.617368 14749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:57:38.648087 14749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:57:38.648919 14749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.43.130
I20250623 01:57:38.682518 14749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:45451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.43.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.14.43.130
--webserver_port=0
--tserver_master_addrs=127.14.43.190:37567
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.43.130
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:57:38.683775 14749 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:57:38.685364 14749 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:57:38.699224 14756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:57:39.348165 14745 heartbeater.cc:499] Master 127.14.43.190:37567 was elected leader, sending a full tablet report...
W20250623 01:57:38.700227 14758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:57:38.699841 14755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:57:39.780771 14757 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1078 milliseconds
I20250623 01:57:39.780877 14749 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250623 01:57:39.781988 14749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:57:39.784451 14749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:57:39.785845 14749 hybrid_clock.cc:648] HybridClock initialized: now 1750643859785809 us; error 55 us; skew 500 ppm
I20250623 01:57:39.786625 14749 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:57:39.792830 14749 webserver.cc:469] Webserver started at http://127.14.43.130:44571/ using document root <none> and password file <none>
I20250623 01:57:39.793663 14749 fs_manager.cc:362] Metadata directory not provided
I20250623 01:57:39.793839 14749 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:57:39.794296 14749 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:57:39.798462 14749 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "9420c477c61c4d7daf69357dcc965bc2"
format_stamp: "Formatted at 2025-06-23 01:57:39 on dist-test-slave-9rpg"
I20250623 01:57:39.799458 14749 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "9420c477c61c4d7daf69357dcc965bc2"
format_stamp: "Formatted at 2025-06-23 01:57:39 on dist-test-slave-9rpg"
I20250623 01:57:39.806054 14749 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250623 01:57:39.811271 14765 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:39.812207 14749 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250623 01:57:39.812491 14749 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "9420c477c61c4d7daf69357dcc965bc2"
format_stamp: "Formatted at 2025-06-23 01:57:39 on dist-test-slave-9rpg"
I20250623 01:57:39.812803 14749 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:57:39.868041 14749 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:57:39.869422 14749 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:57:39.869825 14749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:57:39.872267 14749 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:57:39.876161 14749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250623 01:57:39.876365 14749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:39.876595 14749 ts_tablet_manager.cc:610] Registered 0 tablets
I20250623 01:57:39.876746 14749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:39.998109 14749 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.130:32775
I20250623 01:57:39.998214 14877 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.130:32775 every 8 connection(s)
I20250623 01:57:40.000545 14749 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250623 01:57:40.004941 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 14749
I20250623 01:57:40.005442 14510 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250623 01:57:40.011345 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.43.131:0
--local_ip_for_outbound_sockets=127.14.43.131
--webserver_interface=127.14.43.131
--webserver_port=0
--tserver_master_addrs=127.14.43.190:37567
--builtin_ntp_servers=127.14.43.148:45451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250623 01:57:40.019768 14878 heartbeater.cc:344] Connected to a master server at 127.14.43.190:37567
I20250623 01:57:40.020285 14878 heartbeater.cc:461] Registering TS with master...
I20250623 01:57:40.021492 14878 heartbeater.cc:507] Master 127.14.43.190:37567 requested a full tablet report, sending...
I20250623 01:57:40.023762 14557 ts_manager.cc:194] Registered new tserver with Master: 9420c477c61c4d7daf69357dcc965bc2 (127.14.43.130:32775)
I20250623 01:57:40.025004 14557 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.43.130:36737
W20250623 01:57:40.295565 14882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:57:40.296059 14882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:57:40.296540 14882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:57:40.327137 14882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:57:40.327934 14882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.43.131
I20250623 01:57:40.362069 14882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:45451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.43.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.14.43.131
--webserver_port=0
--tserver_master_addrs=127.14.43.190:37567
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.43.131
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:57:40.363319 14882 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:57:40.364873 14882 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:57:40.379644 14888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:57:41.028671 14878 heartbeater.cc:499] Master 127.14.43.190:37567 was elected leader, sending a full tablet report...
W20250623 01:57:40.379635 14889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:57:41.481724 14890 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1102 milliseconds
W20250623 01:57:41.482403 14891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:57:41.482441 14882 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250623 01:57:41.486207 14882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:57:41.488627 14882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:57:41.490065 14882 hybrid_clock.cc:648] HybridClock initialized: now 1750643861489991 us; error 54 us; skew 500 ppm
I20250623 01:57:41.490859 14882 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:57:41.500334 14882 webserver.cc:469] Webserver started at http://127.14.43.131:39673/ using document root <none> and password file <none>
I20250623 01:57:41.501207 14882 fs_manager.cc:362] Metadata directory not provided
I20250623 01:57:41.501415 14882 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:57:41.501837 14882 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:57:41.506119 14882 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "b5c946e0639844848a1ffd5e069bd094"
format_stamp: "Formatted at 2025-06-23 01:57:41 on dist-test-slave-9rpg"
I20250623 01:57:41.507153 14882 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "b5c946e0639844848a1ffd5e069bd094"
format_stamp: "Formatted at 2025-06-23 01:57:41 on dist-test-slave-9rpg"
I20250623 01:57:41.513736 14882 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250623 01:57:41.518981 14898 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:41.519917 14882 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250623 01:57:41.520391 14882 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "b5c946e0639844848a1ffd5e069bd094"
format_stamp: "Formatted at 2025-06-23 01:57:41 on dist-test-slave-9rpg"
I20250623 01:57:41.520725 14882 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:57:41.582369 14882 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:57:41.583770 14882 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:57:41.584190 14882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:57:41.586848 14882 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:57:41.590824 14882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250623 01:57:41.591027 14882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:41.591261 14882 ts_tablet_manager.cc:610] Registered 0 tablets
I20250623 01:57:41.591405 14882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:41.715641 14882 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.131:33033
I20250623 01:57:41.715791 15010 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.131:33033 every 8 connection(s)
I20250623 01:57:41.718214 14882 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250623 01:57:41.727535 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 14882
I20250623 01:57:41.727897 14510 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250623 01:57:41.733167 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.43.132:0
--local_ip_for_outbound_sockets=127.14.43.132
--webserver_interface=127.14.43.132
--webserver_port=0
--tserver_master_addrs=127.14.43.190:37567
--builtin_ntp_servers=127.14.43.148:45451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250623 01:57:41.737504 15011 heartbeater.cc:344] Connected to a master server at 127.14.43.190:37567
I20250623 01:57:41.737908 15011 heartbeater.cc:461] Registering TS with master...
I20250623 01:57:41.738900 15011 heartbeater.cc:507] Master 127.14.43.190:37567 requested a full tablet report, sending...
I20250623 01:57:41.741132 14557 ts_manager.cc:194] Registered new tserver with Master: b5c946e0639844848a1ffd5e069bd094 (127.14.43.131:33033)
I20250623 01:57:41.742738 14557 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.43.131:45955
W20250623 01:57:42.014731 15015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:57:42.015200 15015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:57:42.015640 15015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:57:42.047379 15015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:57:42.048183 15015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.43.132
I20250623 01:57:42.082394 15015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:45451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.43.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.14.43.132
--webserver_port=0
--tserver_master_addrs=127.14.43.190:37567
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.43.132
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:57:42.083597 15015 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:57:42.085146 15015 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:57:42.099476 15021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:57:42.745990 15011 heartbeater.cc:499] Master 127.14.43.190:37567 was elected leader, sending a full tablet report...
W20250623 01:57:42.099808 15022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:57:42.102317 15024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:57:42.101876 15015 server_base.cc:1048] running on GCE node
I20250623 01:57:43.183302 15015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:57:43.186337 15015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:57:43.187789 15015 hybrid_clock.cc:648] HybridClock initialized: now 1750643863187721 us; error 82 us; skew 500 ppm
I20250623 01:57:43.188620 15015 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:57:43.195003 15015 webserver.cc:469] Webserver started at http://127.14.43.132:34417/ using document root <none> and password file <none>
I20250623 01:57:43.195911 15015 fs_manager.cc:362] Metadata directory not provided
I20250623 01:57:43.196115 15015 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:57:43.196549 15015 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:57:43.202082 15015 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "92705fe433a14034b602e066d7327bf7"
format_stamp: "Formatted at 2025-06-23 01:57:43 on dist-test-slave-9rpg"
I20250623 01:57:43.203250 15015 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "92705fe433a14034b602e066d7327bf7"
format_stamp: "Formatted at 2025-06-23 01:57:43 on dist-test-slave-9rpg"
I20250623 01:57:43.210124 15015 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20250623 01:57:43.215456 15031 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:43.216472 15015 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.004s
I20250623 01:57:43.216768 15015 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "92705fe433a14034b602e066d7327bf7"
format_stamp: "Formatted at 2025-06-23 01:57:43 on dist-test-slave-9rpg"
I20250623 01:57:43.217079 15015 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:57:43.264397 15015 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:57:43.265817 15015 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:57:43.266258 15015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:57:43.268658 15015 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:57:43.272663 15015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250623 01:57:43.272861 15015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:43.273100 15015 ts_tablet_manager.cc:610] Registered 0 tablets
I20250623 01:57:43.273249 15015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:43.400979 15015 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.132:36877
I20250623 01:57:43.401093 15143 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.132:36877 every 8 connection(s)
I20250623 01:57:43.403497 15015 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250623 01:57:43.407482 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 15015
I20250623 01:57:43.407891 14510 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250623 01:57:43.424064 15144 heartbeater.cc:344] Connected to a master server at 127.14.43.190:37567
I20250623 01:57:43.424533 15144 heartbeater.cc:461] Registering TS with master...
I20250623 01:57:43.425733 15144 heartbeater.cc:507] Master 127.14.43.190:37567 requested a full tablet report, sending...
I20250623 01:57:43.428597 14556 ts_manager.cc:194] Registered new tserver with Master: 92705fe433a14034b602e066d7327bf7 (127.14.43.132:36877)
I20250623 01:57:43.430308 14556 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.43.132:36943
I20250623 01:57:43.444289 14510 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250623 01:57:43.478958 14556 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40880:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250623 01:57:43.540550 14946 tablet_service.cc:1468] Processing CreateTablet for tablet af9071880fc946329db80c284049bca8 (DEFAULT_TABLE table=TestTable [id=004dae82583b482383f1d80a26efd436]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 01:57:43.543427 14946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af9071880fc946329db80c284049bca8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:43.543795 15079 tablet_service.cc:1468] Processing CreateTablet for tablet af9071880fc946329db80c284049bca8 (DEFAULT_TABLE table=TestTable [id=004dae82583b482383f1d80a26efd436]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 01:57:43.544584 14680 tablet_service.cc:1468] Processing CreateTablet for tablet af9071880fc946329db80c284049bca8 (DEFAULT_TABLE table=TestTable [id=004dae82583b482383f1d80a26efd436]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 01:57:43.545596 15079 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af9071880fc946329db80c284049bca8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:43.546051 14680 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af9071880fc946329db80c284049bca8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:43.567057 15163 tablet_bootstrap.cc:492] T af9071880fc946329db80c284049bca8 P 9ce9e2fa9e934e74b7e50719bb379eec: Bootstrap starting.
I20250623 01:57:43.570369 15164 tablet_bootstrap.cc:492] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094: Bootstrap starting.
I20250623 01:57:43.573136 15165 tablet_bootstrap.cc:492] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7: Bootstrap starting.
I20250623 01:57:43.574813 15163 tablet_bootstrap.cc:654] T af9071880fc946329db80c284049bca8 P 9ce9e2fa9e934e74b7e50719bb379eec: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:43.577171 15163 log.cc:826] T af9071880fc946329db80c284049bca8 P 9ce9e2fa9e934e74b7e50719bb379eec: Log is configured to *not* fsync() on all Append() calls
I20250623 01:57:43.577852 15164 tablet_bootstrap.cc:654] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:43.579872 15164 log.cc:826] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094: Log is configured to *not* fsync() on all Append() calls
I20250623 01:57:43.580344 15165 tablet_bootstrap.cc:654] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:43.582557 15165 log.cc:826] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7: Log is configured to *not* fsync() on all Append() calls
I20250623 01:57:43.582764 15163 tablet_bootstrap.cc:492] T af9071880fc946329db80c284049bca8 P 9ce9e2fa9e934e74b7e50719bb379eec: No bootstrap required, opened a new log
I20250623 01:57:43.583251 15163 ts_tablet_manager.cc:1397] T af9071880fc946329db80c284049bca8 P 9ce9e2fa9e934e74b7e50719bb379eec: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.003s
I20250623 01:57:43.584580 15164 tablet_bootstrap.cc:492] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094: No bootstrap required, opened a new log
I20250623 01:57:43.585155 15164 ts_tablet_manager.cc:1397] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.005s
I20250623 01:57:43.590768 15165 tablet_bootstrap.cc:492] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7: No bootstrap required, opened a new log
I20250623 01:57:43.591284 15165 ts_tablet_manager.cc:1397] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7: Time spent bootstrapping tablet: real 0.019s user 0.014s sys 0.000s
I20250623 01:57:43.604038 15164 raft_consensus.cc:357] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 } } peers { permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 } } peers { permanent_uuid: "9ce9e2fa9e934e74b7e50719bb379eec" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37411 } }
I20250623 01:57:43.604707 15164 raft_consensus.cc:383] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:43.604962 15164 raft_consensus.cc:738] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5c946e0639844848a1ffd5e069bd094, State: Initialized, Role: FOLLOWER
I20250623 01:57:43.605676 15164 consensus_queue.cc:260] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 } } peers { permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 } } peers { permanent_uuid: "9ce9e2fa9e934e74b7e50719bb379eec" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37411 } }
I20250623 01:57:43.608295 15163 raft_consensus.cc:357] T af9071880fc946329db80c284049bca8 P 9ce9e2fa9e934e74b7e50719bb379eec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 } } peers { permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 } } peers { permanent_uuid: "9ce9e2fa9e934e74b7e50719bb379eec" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37411 } }
I20250623 01:57:43.609197 15163 raft_consensus.cc:383] T af9071880fc946329db80c284049bca8 P 9ce9e2fa9e934e74b7e50719bb379eec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:43.609474 15164 ts_tablet_manager.cc:1428] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094: Time spent starting tablet: real 0.024s user 0.019s sys 0.003s
I20250623 01:57:43.609505 15163 raft_consensus.cc:738] T af9071880fc946329db80c284049bca8 P 9ce9e2fa9e934e74b7e50719bb379eec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ce9e2fa9e934e74b7e50719bb379eec, State: Initialized, Role: FOLLOWER
I20250623 01:57:43.610402 15163 consensus_queue.cc:260] T af9071880fc946329db80c284049bca8 P 9ce9e2fa9e934e74b7e50719bb379eec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 } } peers { permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 } } peers { permanent_uuid: "9ce9e2fa9e934e74b7e50719bb379eec" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37411 } }
I20250623 01:57:43.614876 15163 ts_tablet_manager.cc:1428] T af9071880fc946329db80c284049bca8 P 9ce9e2fa9e934e74b7e50719bb379eec: Time spent starting tablet: real 0.031s user 0.031s sys 0.001s
I20250623 01:57:43.616837 15165 raft_consensus.cc:357] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 } } peers { permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 } } peers { permanent_uuid: "9ce9e2fa9e934e74b7e50719bb379eec" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37411 } }
I20250623 01:57:43.617483 15165 raft_consensus.cc:383] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:43.617739 15165 raft_consensus.cc:738] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92705fe433a14034b602e066d7327bf7, State: Initialized, Role: FOLLOWER
I20250623 01:57:43.618546 15165 consensus_queue.cc:260] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 } } peers { permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 } } peers { permanent_uuid: "9ce9e2fa9e934e74b7e50719bb379eec" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37411 } }
I20250623 01:57:43.621724 15144 heartbeater.cc:499] Master 127.14.43.190:37567 was elected leader, sending a full tablet report...
I20250623 01:57:43.622901 15165 ts_tablet_manager.cc:1428] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7: Time spent starting tablet: real 0.031s user 0.022s sys 0.009s
W20250623 01:57:43.657711 15145 tablet.cc:2378] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250623 01:57:43.724153 15012 tablet.cc:2378] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250623 01:57:43.806504 15170 raft_consensus.cc:491] T af9071880fc946329db80c284049bca8 P 9ce9e2fa9e934e74b7e50719bb379eec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:57:43.806972 15170 raft_consensus.cc:513] T af9071880fc946329db80c284049bca8 P 9ce9e2fa9e934e74b7e50719bb379eec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 } } peers { permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 } } peers { permanent_uuid: "9ce9e2fa9e934e74b7e50719bb379eec" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37411 } }
I20250623 01:57:43.809388 15170 leader_election.cc:290] T af9071880fc946329db80c284049bca8 P 9ce9e2fa9e934e74b7e50719bb379eec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92705fe433a14034b602e066d7327bf7 (127.14.43.132:36877), b5c946e0639844848a1ffd5e069bd094 (127.14.43.131:33033)
I20250623 01:57:43.822189 15099 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af9071880fc946329db80c284049bca8" candidate_uuid: "9ce9e2fa9e934e74b7e50719bb379eec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92705fe433a14034b602e066d7327bf7" is_pre_election: true
I20250623 01:57:43.822876 15099 raft_consensus.cc:2466] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ce9e2fa9e934e74b7e50719bb379eec in term 0.
I20250623 01:57:43.822813 14966 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af9071880fc946329db80c284049bca8" candidate_uuid: "9ce9e2fa9e934e74b7e50719bb379eec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5c946e0639844848a1ffd5e069bd094" is_pre_election: true
I20250623 01:57:43.823473 14966 raft_consensus.cc:2466] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ce9e2fa9e934e74b7e50719bb379eec in term 0.
I20250623 01:57:43.824033 14633 leader_election.cc:304] T af9071880fc946329db80c284049bca8 P 9ce9e2fa9e934e74b7e50719bb379eec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92705fe433a14034b602e066d7327bf7, 9ce9e2fa9e934e74b7e50719bb379eec; no voters:
I20250623 01:57:43.824867 15170 raft_consensus.cc:2802] T af9071880fc946329db80c284049bca8 P 9ce9e2fa9e934e74b7e50719bb379eec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:57:43.825153 15170 raft_consensus.cc:491] T af9071880fc946329db80c284049bca8 P 9ce9e2fa9e934e74b7e50719bb379eec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:57:43.825407 15170 raft_consensus.cc:3058] T af9071880fc946329db80c284049bca8 P 9ce9e2fa9e934e74b7e50719bb379eec [term 0 FOLLOWER]: Advancing to term 1
W20250623 01:57:43.827661 14746 tablet.cc:2378] T af9071880fc946329db80c284049bca8 P 9ce9e2fa9e934e74b7e50719bb379eec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250623 01:57:43.829782 15170 raft_consensus.cc:513] T af9071880fc946329db80c284049bca8 P 9ce9e2fa9e934e74b7e50719bb379eec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 } } peers { permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 } } peers { permanent_uuid: "9ce9e2fa9e934e74b7e50719bb379eec" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37411 } }
I20250623 01:57:43.831187 15170 leader_election.cc:290] T af9071880fc946329db80c284049bca8 P 9ce9e2fa9e934e74b7e50719bb379eec [CANDIDATE]: Term 1 election: Requested vote from peers 92705fe433a14034b602e066d7327bf7 (127.14.43.132:36877), b5c946e0639844848a1ffd5e069bd094 (127.14.43.131:33033)
I20250623 01:57:43.831944 15099 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af9071880fc946329db80c284049bca8" candidate_uuid: "9ce9e2fa9e934e74b7e50719bb379eec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92705fe433a14034b602e066d7327bf7"
I20250623 01:57:43.831998 14966 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af9071880fc946329db80c284049bca8" candidate_uuid: "9ce9e2fa9e934e74b7e50719bb379eec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5c946e0639844848a1ffd5e069bd094"
I20250623 01:57:43.832378 15099 raft_consensus.cc:3058] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:43.832392 14966 raft_consensus.cc:3058] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:43.836593 15099 raft_consensus.cc:2466] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ce9e2fa9e934e74b7e50719bb379eec in term 1.
I20250623 01:57:43.836589 14966 raft_consensus.cc:2466] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ce9e2fa9e934e74b7e50719bb379eec in term 1.
I20250623 01:57:43.837502 14636 leader_election.cc:304] T af9071880fc946329db80c284049bca8 P 9ce9e2fa9e934e74b7e50719bb379eec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ce9e2fa9e934e74b7e50719bb379eec, b5c946e0639844848a1ffd5e069bd094; no voters:
I20250623 01:57:43.838223 15170 raft_consensus.cc:2802] T af9071880fc946329db80c284049bca8 P 9ce9e2fa9e934e74b7e50719bb379eec [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:57:43.839625 15170 raft_consensus.cc:695] T af9071880fc946329db80c284049bca8 P 9ce9e2fa9e934e74b7e50719bb379eec [term 1 LEADER]: Becoming Leader. State: Replica: 9ce9e2fa9e934e74b7e50719bb379eec, State: Running, Role: LEADER
I20250623 01:57:43.840505 15170 consensus_queue.cc:237] T af9071880fc946329db80c284049bca8 P 9ce9e2fa9e934e74b7e50719bb379eec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 } } peers { permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 } } peers { permanent_uuid: "9ce9e2fa9e934e74b7e50719bb379eec" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37411 } }
I20250623 01:57:43.851780 14557 catalog_manager.cc:5582] T af9071880fc946329db80c284049bca8 P 9ce9e2fa9e934e74b7e50719bb379eec reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ce9e2fa9e934e74b7e50719bb379eec (127.14.43.129). New cstate: current_term: 1 leader_uuid: "9ce9e2fa9e934e74b7e50719bb379eec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ce9e2fa9e934e74b7e50719bb379eec" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37411 } health_report { overall_health: HEALTHY } } }
I20250623 01:57:43.923882 14510 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250623 01:57:43.927673 14510 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 9ce9e2fa9e934e74b7e50719bb379eec to finish bootstrapping
I20250623 01:57:43.941464 14510 ts_itest-base.cc:246] Waiting for 1 tablets on tserver b5c946e0639844848a1ffd5e069bd094 to finish bootstrapping
I20250623 01:57:43.951822 14510 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 92705fe433a14034b602e066d7327bf7 to finish bootstrapping
I20250623 01:57:43.961908 14510 kudu-admin-test.cc:709] Waiting for Master to see the current replicas...
I20250623 01:57:43.965150 14510 kudu-admin-test.cc:716] Tablet locations:
tablet_locations {
tablet_id: "af9071880fc946329db80c284049bca8"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 2
role: LEADER
}
}
ts_infos {
permanent_uuid: "92705fe433a14034b602e066d7327bf7"
rpc_addresses {
host: "127.14.43.132"
port: 36877
}
}
ts_infos {
permanent_uuid: "b5c946e0639844848a1ffd5e069bd094"
rpc_addresses {
host: "127.14.43.131"
port: 33033
}
}
ts_infos {
permanent_uuid: "9ce9e2fa9e934e74b7e50719bb379eec"
rpc_addresses {
host: "127.14.43.129"
port: 37411
}
}
I20250623 01:57:44.286756 15170 consensus_queue.cc:1035] T af9071880fc946329db80c284049bca8 P 9ce9e2fa9e934e74b7e50719bb379eec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:57:44.290448 15170 consensus_queue.cc:1035] T af9071880fc946329db80c284049bca8 P 9ce9e2fa9e934e74b7e50719bb379eec [LEADER]: Connected to new peer: Peer: permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:57:44.305410 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 14616
W20250623 01:57:44.327800 14542 connection.cc:537] server connection from 127.14.43.129:48225 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250623 01:57:44.327953 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 14524
I20250623 01:57:44.352561 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.43.190:37567
--webserver_interface=127.14.43.190
--webserver_port=46469
--builtin_ntp_servers=127.14.43.148:45451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.43.190:37567 with env {}
W20250623 01:57:44.656858 15185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:57:44.657400 15185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:57:44.657799 15185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:57:44.689311 15185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250623 01:57:44.689599 15185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:57:44.689797 15185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250623 01:57:44.689994 15185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250623 01:57:44.725190 15185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:45451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.43.190:37567
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.43.190:37567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.14.43.190
--webserver_port=46469
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:57:44.726533 15185 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:57:44.728420 15185 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:57:44.743831 15192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:57:45.046330 14878 heartbeater.cc:646] Failed to heartbeat to 127.14.43.190:37567 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.43.190:37567: connect: Connection refused (error 111)
W20250623 01:57:45.329015 15011 heartbeater.cc:646] Failed to heartbeat to 127.14.43.190:37567 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.43.190:37567: connect: Connection refused (error 111)
W20250623 01:57:45.331872 15144 heartbeater.cc:646] Failed to heartbeat to 127.14.43.190:37567 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.43.190:37567: connect: Connection refused (error 111)
I20250623 01:57:45.796931 15200 raft_consensus.cc:491] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9ce9e2fa9e934e74b7e50719bb379eec)
I20250623 01:57:45.797451 15200 raft_consensus.cc:513] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 } } peers { permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 } } peers { permanent_uuid: "9ce9e2fa9e934e74b7e50719bb379eec" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37411 } }
I20250623 01:57:45.814695 15201 raft_consensus.cc:491] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9ce9e2fa9e934e74b7e50719bb379eec)
I20250623 01:57:45.807312 15200 leader_election.cc:290] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b5c946e0639844848a1ffd5e069bd094 (127.14.43.131:33033), 9ce9e2fa9e934e74b7e50719bb379eec (127.14.43.129:37411)
I20250623 01:57:45.817517 15201 raft_consensus.cc:513] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 } } peers { permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 } } peers { permanent_uuid: "9ce9e2fa9e934e74b7e50719bb379eec" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37411 } }
W20250623 01:57:45.821712 15032 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.43.129:37411: connect: Connection refused (error 111)
W20250623 01:57:45.831835 15032 leader_election.cc:336] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9ce9e2fa9e934e74b7e50719bb379eec (127.14.43.129:37411): Network error: Client connection negotiation failed: client connection to 127.14.43.129:37411: connect: Connection refused (error 111)
I20250623 01:57:45.833045 15201 leader_election.cc:290] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92705fe433a14034b602e066d7327bf7 (127.14.43.132:36877), 9ce9e2fa9e934e74b7e50719bb379eec (127.14.43.129:37411)
I20250623 01:57:45.842756 14966 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af9071880fc946329db80c284049bca8" candidate_uuid: "92705fe433a14034b602e066d7327bf7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b5c946e0639844848a1ffd5e069bd094" is_pre_election: true
I20250623 01:57:45.843420 14966 raft_consensus.cc:2466] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92705fe433a14034b602e066d7327bf7 in term 1.
I20250623 01:57:45.845173 15035 leader_election.cc:304] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92705fe433a14034b602e066d7327bf7, b5c946e0639844848a1ffd5e069bd094; no voters: 9ce9e2fa9e934e74b7e50719bb379eec
I20250623 01:57:45.846210 15200 raft_consensus.cc:2802] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 01:57:45.846571 15200 raft_consensus.cc:491] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9ce9e2fa9e934e74b7e50719bb379eec)
I20250623 01:57:45.846891 15200 raft_consensus.cc:3058] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [term 1 FOLLOWER]: Advancing to term 2
W20250623 01:57:45.849923 14899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.43.129:37411: connect: Connection refused (error 111)
I20250623 01:57:45.854422 15200 raft_consensus.cc:513] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 } } peers { permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 } } peers { permanent_uuid: "9ce9e2fa9e934e74b7e50719bb379eec" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37411 } }
W20250623 01:57:45.861960 15032 leader_election.cc:336] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9ce9e2fa9e934e74b7e50719bb379eec (127.14.43.129:37411): Network error: Client connection negotiation failed: client connection to 127.14.43.129:37411: connect: Connection refused (error 111)
I20250623 01:57:45.863968 14966 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af9071880fc946329db80c284049bca8" candidate_uuid: "92705fe433a14034b602e066d7327bf7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b5c946e0639844848a1ffd5e069bd094"
I20250623 01:57:45.864557 14966 raft_consensus.cc:3058] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:45.871595 14966 raft_consensus.cc:2466] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92705fe433a14034b602e066d7327bf7 in term 2.
I20250623 01:57:45.872879 15035 leader_election.cc:304] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92705fe433a14034b602e066d7327bf7, b5c946e0639844848a1ffd5e069bd094; no voters: 9ce9e2fa9e934e74b7e50719bb379eec
I20250623 01:57:45.874768 15200 leader_election.cc:290] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [CANDIDATE]: Term 2 election: Requested vote from peers b5c946e0639844848a1ffd5e069bd094 (127.14.43.131:33033), 9ce9e2fa9e934e74b7e50719bb379eec (127.14.43.129:37411)
I20250623 01:57:45.875427 15208 raft_consensus.cc:2802] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:57:45.877308 15208 raft_consensus.cc:695] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [term 2 LEADER]: Becoming Leader. State: Replica: 92705fe433a14034b602e066d7327bf7, State: Running, Role: LEADER
I20250623 01:57:45.877543 15099 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af9071880fc946329db80c284049bca8" candidate_uuid: "b5c946e0639844848a1ffd5e069bd094" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "92705fe433a14034b602e066d7327bf7" is_pre_election: true
I20250623 01:57:45.878695 15208 consensus_queue.cc:237] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 } } peers { permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 } } peers { permanent_uuid: "9ce9e2fa9e934e74b7e50719bb379eec" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37411 } }
W20250623 01:57:45.880975 14899 leader_election.cc:336] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9ce9e2fa9e934e74b7e50719bb379eec (127.14.43.129:37411): Network error: Client connection negotiation failed: client connection to 127.14.43.129:37411: connect: Connection refused (error 111)
I20250623 01:57:45.881489 14899 leader_election.cc:304] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5c946e0639844848a1ffd5e069bd094; no voters: 92705fe433a14034b602e066d7327bf7, 9ce9e2fa9e934e74b7e50719bb379eec
I20250623 01:57:45.882287 15201 raft_consensus.cc:2747] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250623 01:57:44.746390 15191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:57:44.743860 15194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:57:45.910610 15193 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1165 milliseconds
I20250623 01:57:45.910724 15185 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250623 01:57:45.912045 15185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:57:45.914728 15185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:57:45.916059 15185 hybrid_clock.cc:648] HybridClock initialized: now 1750643865916018 us; error 51 us; skew 500 ppm
I20250623 01:57:45.916800 15185 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:57:45.922576 15185 webserver.cc:469] Webserver started at http://127.14.43.190:46469/ using document root <none> and password file <none>
I20250623 01:57:45.923456 15185 fs_manager.cc:362] Metadata directory not provided
I20250623 01:57:45.923651 15185 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:57:45.931013 15185 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20250623 01:57:45.935223 15214 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:45.936182 15185 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20250623 01:57:45.936475 15185 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
uuid: "617bf001a689445bb005a7acaadbad73"
format_stamp: "Formatted at 2025-06-23 01:57:36 on dist-test-slave-9rpg"
I20250623 01:57:45.938433 15185 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:57:46.002575 15185 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:57:46.004462 15185 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:57:46.004992 15185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:57:46.077647 15185 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.190:37567
I20250623 01:57:46.077731 15266 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.190:37567 every 8 connection(s)
I20250623 01:57:46.081218 15185 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250623 01:57:46.085564 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 15185
I20250623 01:57:46.086227 14510 kudu-admin-test.cc:735] Forcing unsafe config change on tserver b5c946e0639844848a1ffd5e069bd094
I20250623 01:57:46.097678 15267 sys_catalog.cc:263] Verifying existing consensus state
I20250623 01:57:46.104844 15267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73: Bootstrap starting.
I20250623 01:57:46.144923 15267 log.cc:826] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73: Log is configured to *not* fsync() on all Append() calls
I20250623 01:57:46.166409 15267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 01:57:46.167213 15267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73: Bootstrap complete.
I20250623 01:57:46.187959 15267 raft_consensus.cc:357] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617bf001a689445bb005a7acaadbad73" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 37567 } }
I20250623 01:57:46.190063 15267 raft_consensus.cc:738] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 617bf001a689445bb005a7acaadbad73, State: Initialized, Role: FOLLOWER
I20250623 01:57:46.190847 15267 consensus_queue.cc:260] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617bf001a689445bb005a7acaadbad73" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 37567 } }
I20250623 01:57:46.191320 15267 raft_consensus.cc:397] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250623 01:57:46.191592 15267 raft_consensus.cc:491] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250623 01:57:46.191872 15267 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:57:46.197458 15267 raft_consensus.cc:513] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617bf001a689445bb005a7acaadbad73" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 37567 } }
I20250623 01:57:46.198155 15267 leader_election.cc:304] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 617bf001a689445bb005a7acaadbad73; no voters:
I20250623 01:57:46.200112 15267 leader_election.cc:290] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250623 01:57:46.200558 15271 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:57:46.203576 15271 raft_consensus.cc:695] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73 [term 2 LEADER]: Becoming Leader. State: Replica: 617bf001a689445bb005a7acaadbad73, State: Running, Role: LEADER
I20250623 01:57:46.204388 15271 consensus_queue.cc:237] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617bf001a689445bb005a7acaadbad73" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 37567 } }
I20250623 01:57:46.205120 15267 sys_catalog.cc:564] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73 [sys.catalog]: configured and running, proceeding with master startup.
I20250623 01:57:46.213932 15273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 617bf001a689445bb005a7acaadbad73. Latest consensus state: current_term: 2 leader_uuid: "617bf001a689445bb005a7acaadbad73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617bf001a689445bb005a7acaadbad73" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 37567 } } }
I20250623 01:57:46.215332 15272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "617bf001a689445bb005a7acaadbad73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617bf001a689445bb005a7acaadbad73" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 37567 } } }
I20250623 01:57:46.215950 15272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73 [sys.catalog]: This master's current role is: LEADER
I20250623 01:57:46.218621 15273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73 [sys.catalog]: This master's current role is: LEADER
I20250623 01:57:46.220428 15278 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250623 01:57:46.232354 15278 catalog_manager.cc:671] Loaded metadata for table TestTable [id=004dae82583b482383f1d80a26efd436]
I20250623 01:57:46.239390 15278 tablet_loader.cc:96] loaded metadata for tablet af9071880fc946329db80c284049bca8 (table TestTable [id=004dae82583b482383f1d80a26efd436])
I20250623 01:57:46.240877 15278 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250623 01:57:46.245594 15278 catalog_manager.cc:1261] Loaded cluster ID: 007b950910b243e385ab834806ef7f1a
I20250623 01:57:46.245910 15278 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250623 01:57:46.253598 15278 catalog_manager.cc:1506] Loading token signing keys...
I20250623 01:57:46.258391 15278 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 617bf001a689445bb005a7acaadbad73: Loaded TSK: 0
I20250623 01:57:46.260123 15278 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250623 01:57:46.349032 15011 heartbeater.cc:344] Connected to a master server at 127.14.43.190:37567
I20250623 01:57:46.355253 15231 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" instance_seqno: 1750643861686926) as {username='slave'} at 127.14.43.131:51147; Asking this server to re-register.
I20250623 01:57:46.356705 15011 heartbeater.cc:461] Registering TS with master...
I20250623 01:57:46.357311 15011 heartbeater.cc:507] Master 127.14.43.190:37567 requested a full tablet report, sending...
I20250623 01:57:46.360532 15231 ts_manager.cc:194] Registered new tserver with Master: b5c946e0639844848a1ffd5e069bd094 (127.14.43.131:33033)
I20250623 01:57:46.448959 14966 raft_consensus.cc:1273] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [term 2 FOLLOWER]: Refusing update from remote peer 92705fe433a14034b602e066d7327bf7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:57:46.450480 15294 consensus_queue.cc:1035] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250623 01:57:46.451545 15269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:57:46.452245 15269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250623 01:57:46.462368 15231 catalog_manager.cc:5582] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 reported cstate change: term changed from 1 to 2, leader changed from 9ce9e2fa9e934e74b7e50719bb379eec (127.14.43.129) to 92705fe433a14034b602e066d7327bf7 (127.14.43.132). New cstate: current_term: 2 leader_uuid: "92705fe433a14034b602e066d7327bf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 } } peers { permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 } } peers { permanent_uuid: "9ce9e2fa9e934e74b7e50719bb379eec" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37411 } } }
W20250623 01:57:46.508632 15269 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250623 01:57:46.516103 15144 heartbeater.cc:344] Connected to a master server at 127.14.43.190:37567
W20250623 01:57:46.516553 15032 consensus_peers.cc:487] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 -> Peer 9ce9e2fa9e934e74b7e50719bb379eec (127.14.43.129:37411): Couldn't send request to peer 9ce9e2fa9e934e74b7e50719bb379eec. Status: Network error: Client connection negotiation failed: client connection to 127.14.43.129:37411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250623 01:57:46.519796 15230 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "92705fe433a14034b602e066d7327bf7" instance_seqno: 1750643863370944) as {username='slave'} at 127.14.43.132:38005; Asking this server to re-register.
I20250623 01:57:46.521423 15144 heartbeater.cc:461] Registering TS with master...
I20250623 01:57:46.522150 15144 heartbeater.cc:507] Master 127.14.43.190:37567 requested a full tablet report, sending...
I20250623 01:57:46.525847 15230 ts_manager.cc:194] Registered new tserver with Master: 92705fe433a14034b602e066d7327bf7 (127.14.43.132:36877)
I20250623 01:57:47.078460 14878 heartbeater.cc:344] Connected to a master server at 127.14.43.190:37567
I20250623 01:57:47.080538 15230 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "9420c477c61c4d7daf69357dcc965bc2" instance_seqno: 1750643859969822) as {username='slave'} at 127.14.43.130:58881; Asking this server to re-register.
I20250623 01:57:47.081854 14878 heartbeater.cc:461] Registering TS with master...
I20250623 01:57:47.082466 14878 heartbeater.cc:507] Master 127.14.43.190:37567 requested a full tablet report, sending...
I20250623 01:57:47.084416 15230 ts_manager.cc:194] Registered new tserver with Master: 9420c477c61c4d7daf69357dcc965bc2 (127.14.43.130:32775)
W20250623 01:57:47.787627 15269 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.229s user 0.436s sys 0.761s
W20250623 01:57:47.788012 15269 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.229s user 0.436s sys 0.761s
I20250623 01:57:47.846630 14966 tablet_service.cc:1905] Received UnsafeChangeConfig RPC: dest_uuid: "b5c946e0639844848a1ffd5e069bd094"
tablet_id: "af9071880fc946329db80c284049bca8"
caller_id: "kudu-tools"
new_config {
peers {
permanent_uuid: "92705fe433a14034b602e066d7327bf7"
}
peers {
permanent_uuid: "b5c946e0639844848a1ffd5e069bd094"
}
}
from {username='slave'} at 127.0.0.1:33822
W20250623 01:57:47.848078 14966 raft_consensus.cc:2216] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [term 2 FOLLOWER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 } } peers { permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 } } peers { permanent_uuid: "9ce9e2fa9e934e74b7e50719bb379eec" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37411 } }NEW CONFIG: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 } } peers { permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 } } unsafe_config_change: true
I20250623 01:57:47.849193 14966 raft_consensus.cc:3058] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:57:47.898818 14966 raft_consensus.cc:1238] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [term 3 FOLLOWER]: Rejecting Update request from peer 92705fe433a14034b602e066d7327bf7 for earlier term 2. Current term is 3. Ops: []
I20250623 01:57:47.900055 15298 consensus_queue.cc:1046] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:57:47.901312 15324 raft_consensus.cc:3053] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [term 2 LEADER]: Stepping down as leader of term 2
I20250623 01:57:47.901537 15324 raft_consensus.cc:738] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 92705fe433a14034b602e066d7327bf7, State: Running, Role: LEADER
I20250623 01:57:47.902101 15324 consensus_queue.cc:260] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 } } peers { permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 } } peers { permanent_uuid: "9ce9e2fa9e934e74b7e50719bb379eec" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37411 } }
I20250623 01:57:47.902973 15324 raft_consensus.cc:3058] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:57:49.181564 15328 raft_consensus.cc:491] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:57:49.181934 15328 raft_consensus.cc:513] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 } } peers { permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 } } peers { permanent_uuid: "9ce9e2fa9e934e74b7e50719bb379eec" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37411 } }
I20250623 01:57:49.183377 15328 leader_election.cc:290] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b5c946e0639844848a1ffd5e069bd094 (127.14.43.131:33033), 9ce9e2fa9e934e74b7e50719bb379eec (127.14.43.129:37411)
I20250623 01:57:49.184167 14966 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af9071880fc946329db80c284049bca8" candidate_uuid: "92705fe433a14034b602e066d7327bf7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b5c946e0639844848a1ffd5e069bd094" is_pre_election: true
W20250623 01:57:49.187480 15032 leader_election.cc:336] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9ce9e2fa9e934e74b7e50719bb379eec (127.14.43.129:37411): Network error: Client connection negotiation failed: client connection to 127.14.43.129:37411: connect: Connection refused (error 111)
I20250623 01:57:49.187788 15032 leader_election.cc:304] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92705fe433a14034b602e066d7327bf7; no voters: 9ce9e2fa9e934e74b7e50719bb379eec, b5c946e0639844848a1ffd5e069bd094
I20250623 01:57:49.188328 15328 raft_consensus.cc:2747] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 01:57:49.357779 15331 raft_consensus.cc:491] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools)
I20250623 01:57:49.358198 15331 raft_consensus.cc:513] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 } } peers { permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 } } unsafe_config_change: true
I20250623 01:57:49.359237 15331 leader_election.cc:290] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 92705fe433a14034b602e066d7327bf7 (127.14.43.132:36877)
I20250623 01:57:49.360234 15099 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af9071880fc946329db80c284049bca8" candidate_uuid: "b5c946e0639844848a1ffd5e069bd094" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "92705fe433a14034b602e066d7327bf7" is_pre_election: true
I20250623 01:57:49.360654 15099 raft_consensus.cc:2466] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5c946e0639844848a1ffd5e069bd094 in term 3.
I20250623 01:57:49.361491 14899 leader_election.cc:304] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 92705fe433a14034b602e066d7327bf7, b5c946e0639844848a1ffd5e069bd094; no voters:
I20250623 01:57:49.362072 15331 raft_consensus.cc:2802] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 01:57:49.362339 15331 raft_consensus.cc:491] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [term 3 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools)
I20250623 01:57:49.362550 15331 raft_consensus.cc:3058] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:57:49.366511 15331 raft_consensus.cc:513] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 } } peers { permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 } } unsafe_config_change: true
I20250623 01:57:49.367440 15331 leader_election.cc:290] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [CANDIDATE]: Term 4 election: Requested vote from peers 92705fe433a14034b602e066d7327bf7 (127.14.43.132:36877)
I20250623 01:57:49.368319 15099 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af9071880fc946329db80c284049bca8" candidate_uuid: "b5c946e0639844848a1ffd5e069bd094" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "92705fe433a14034b602e066d7327bf7"
I20250623 01:57:49.368680 15099 raft_consensus.cc:3058] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:57:49.372685 15099 raft_consensus.cc:2466] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5c946e0639844848a1ffd5e069bd094 in term 4.
I20250623 01:57:49.373476 14899 leader_election.cc:304] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 92705fe433a14034b602e066d7327bf7, b5c946e0639844848a1ffd5e069bd094; no voters:
I20250623 01:57:49.374032 15331 raft_consensus.cc:2802] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:57:49.374779 15331 raft_consensus.cc:695] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [term 4 LEADER]: Becoming Leader. State: Replica: b5c946e0639844848a1ffd5e069bd094, State: Running, Role: LEADER
I20250623 01:57:49.375412 15331 consensus_queue.cc:237] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 } } peers { permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 } } unsafe_config_change: true
I20250623 01:57:49.381011 15230 catalog_manager.cc:5582] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 reported cstate change: term changed from 2 to 4, leader changed from 92705fe433a14034b602e066d7327bf7 (127.14.43.132) to b5c946e0639844848a1ffd5e069bd094 (127.14.43.131), now has a pending config: VOTER 92705fe433a14034b602e066d7327bf7 (127.14.43.132), VOTER b5c946e0639844848a1ffd5e069bd094 (127.14.43.131). New cstate: current_term: 4 leader_uuid: "b5c946e0639844848a1ffd5e069bd094" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ce9e2fa9e934e74b7e50719bb379eec" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37411 } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 } } peers { permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 } } unsafe_config_change: true }
I20250623 01:57:49.971504 15099 raft_consensus.cc:1273] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [term 4 FOLLOWER]: Refusing update from remote peer b5c946e0639844848a1ffd5e069bd094: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250623 01:57:49.972657 15338 consensus_queue.cc:1035] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250623 01:57:49.979754 15341 raft_consensus.cc:2953] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [term 4 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER 9ce9e2fa9e934e74b7e50719bb379eec (127.14.43.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 } } peers { permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 } } unsafe_config_change: true }
I20250623 01:57:49.981125 15099 raft_consensus.cc:2953] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [term 4 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER 9ce9e2fa9e934e74b7e50719bb379eec (127.14.43.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 } } peers { permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 } } unsafe_config_change: true }
I20250623 01:57:49.989373 15230 catalog_manager.cc:5582] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 reported cstate change: config changed from index -1 to 3, VOTER 9ce9e2fa9e934e74b7e50719bb379eec (127.14.43.129) evicted, no longer has a pending config: VOTER 92705fe433a14034b602e066d7327bf7 (127.14.43.132), VOTER b5c946e0639844848a1ffd5e069bd094 (127.14.43.131). New cstate: current_term: 4 leader_uuid: "b5c946e0639844848a1ffd5e069bd094" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
W20250623 01:57:49.997928 15230 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet af9071880fc946329db80c284049bca8 on TS 9ce9e2fa9e934e74b7e50719bb379eec: Not found: failed to reset TS proxy: Could not find TS for UUID 9ce9e2fa9e934e74b7e50719bb379eec
I20250623 01:57:50.018990 14966 consensus_queue.cc:237] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 } } peers { permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 } } peers { permanent_uuid: "9420c477c61c4d7daf69357dcc965bc2" member_type: NON_VOTER last_known_addr { host: "127.14.43.130" port: 32775 } attrs { promote: true } } unsafe_config_change: true
I20250623 01:57:50.023134 15099 raft_consensus.cc:1273] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [term 4 FOLLOWER]: Refusing update from remote peer b5c946e0639844848a1ffd5e069bd094: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250623 01:57:50.024214 15341 consensus_queue.cc:1035] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250623 01:57:50.029274 15338 raft_consensus.cc:2953] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER 9420c477c61c4d7daf69357dcc965bc2 (127.14.43.130) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 } } peers { permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 } } peers { permanent_uuid: "9420c477c61c4d7daf69357dcc965bc2" member_type: NON_VOTER last_known_addr { host: "127.14.43.130" port: 32775 } attrs { promote: true } } unsafe_config_change: true }
I20250623 01:57:50.030462 15099 raft_consensus.cc:2953] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER 9420c477c61c4d7daf69357dcc965bc2 (127.14.43.130) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 } } peers { permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 } } peers { permanent_uuid: "9420c477c61c4d7daf69357dcc965bc2" member_type: NON_VOTER last_known_addr { host: "127.14.43.130" port: 32775 } attrs { promote: true } } unsafe_config_change: true }
W20250623 01:57:50.035737 14900 consensus_peers.cc:487] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 -> Peer 9420c477c61c4d7daf69357dcc965bc2 (127.14.43.130:32775): Couldn't send request to peer 9420c477c61c4d7daf69357dcc965bc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af9071880fc946329db80c284049bca8. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:57:50.036953 15218 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet af9071880fc946329db80c284049bca8 with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250623 01:57:50.039613 15231 catalog_manager.cc:5582] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 reported cstate change: config changed from index 3 to 5, NON_VOTER 9420c477c61c4d7daf69357dcc965bc2 (127.14.43.130) added. New cstate: current_term: 4 leader_uuid: "b5c946e0639844848a1ffd5e069bd094" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9420c477c61c4d7daf69357dcc965bc2" member_type: NON_VOTER last_known_addr { host: "127.14.43.130" port: 32775 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true }
W20250623 01:57:50.050392 15216 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet af9071880fc946329db80c284049bca8 on TS 9ce9e2fa9e934e74b7e50719bb379eec failed: Not found: failed to reset TS proxy: Could not find TS for UUID 9ce9e2fa9e934e74b7e50719bb379eec
I20250623 01:57:50.472721 15352 ts_tablet_manager.cc:927] T af9071880fc946329db80c284049bca8 P 9420c477c61c4d7daf69357dcc965bc2: Initiating tablet copy from peer b5c946e0639844848a1ffd5e069bd094 (127.14.43.131:33033)
I20250623 01:57:50.475091 15352 tablet_copy_client.cc:323] T af9071880fc946329db80c284049bca8 P 9420c477c61c4d7daf69357dcc965bc2: tablet copy: Beginning tablet copy session from remote peer at address 127.14.43.131:33033
I20250623 01:57:50.485740 14986 tablet_copy_service.cc:140] P b5c946e0639844848a1ffd5e069bd094: Received BeginTabletCopySession request for tablet af9071880fc946329db80c284049bca8 from peer 9420c477c61c4d7daf69357dcc965bc2 ({username='slave'} at 127.14.43.130:45445)
I20250623 01:57:50.486348 14986 tablet_copy_service.cc:161] P b5c946e0639844848a1ffd5e069bd094: Beginning new tablet copy session on tablet af9071880fc946329db80c284049bca8 from peer 9420c477c61c4d7daf69357dcc965bc2 at {username='slave'} at 127.14.43.130:45445: session id = 9420c477c61c4d7daf69357dcc965bc2-af9071880fc946329db80c284049bca8
I20250623 01:57:50.491761 14986 tablet_copy_source_session.cc:215] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094: Tablet Copy: opened 0 blocks and 1 log segments
I20250623 01:57:50.497010 15352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af9071880fc946329db80c284049bca8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:50.515046 15352 tablet_copy_client.cc:806] T af9071880fc946329db80c284049bca8 P 9420c477c61c4d7daf69357dcc965bc2: tablet copy: Starting download of 0 data blocks...
I20250623 01:57:50.515610 15352 tablet_copy_client.cc:670] T af9071880fc946329db80c284049bca8 P 9420c477c61c4d7daf69357dcc965bc2: tablet copy: Starting download of 1 WAL segments...
I20250623 01:57:50.519291 15352 tablet_copy_client.cc:538] T af9071880fc946329db80c284049bca8 P 9420c477c61c4d7daf69357dcc965bc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250623 01:57:50.525693 15352 tablet_bootstrap.cc:492] T af9071880fc946329db80c284049bca8 P 9420c477c61c4d7daf69357dcc965bc2: Bootstrap starting.
I20250623 01:57:50.537322 15352 log.cc:826] T af9071880fc946329db80c284049bca8 P 9420c477c61c4d7daf69357dcc965bc2: Log is configured to *not* fsync() on all Append() calls
I20250623 01:57:50.548322 15352 tablet_bootstrap.cc:492] T af9071880fc946329db80c284049bca8 P 9420c477c61c4d7daf69357dcc965bc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 01:57:50.549012 15352 tablet_bootstrap.cc:492] T af9071880fc946329db80c284049bca8 P 9420c477c61c4d7daf69357dcc965bc2: Bootstrap complete.
I20250623 01:57:50.549558 15352 ts_tablet_manager.cc:1397] T af9071880fc946329db80c284049bca8 P 9420c477c61c4d7daf69357dcc965bc2: Time spent bootstrapping tablet: real 0.024s user 0.023s sys 0.000s
I20250623 01:57:50.565922 15352 raft_consensus.cc:357] T af9071880fc946329db80c284049bca8 P 9420c477c61c4d7daf69357dcc965bc2 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 } } peers { permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 } } peers { permanent_uuid: "9420c477c61c4d7daf69357dcc965bc2" member_type: NON_VOTER last_known_addr { host: "127.14.43.130" port: 32775 } attrs { promote: true } } unsafe_config_change: true
I20250623 01:57:50.566803 15352 raft_consensus.cc:738] T af9071880fc946329db80c284049bca8 P 9420c477c61c4d7daf69357dcc965bc2 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 9420c477c61c4d7daf69357dcc965bc2, State: Initialized, Role: LEARNER
I20250623 01:57:50.567394 15352 consensus_queue.cc:260] T af9071880fc946329db80c284049bca8 P 9420c477c61c4d7daf69357dcc965bc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 } } peers { permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 } } peers { permanent_uuid: "9420c477c61c4d7daf69357dcc965bc2" member_type: NON_VOTER last_known_addr { host: "127.14.43.130" port: 32775 } attrs { promote: true } } unsafe_config_change: true
I20250623 01:57:50.570523 15352 ts_tablet_manager.cc:1428] T af9071880fc946329db80c284049bca8 P 9420c477c61c4d7daf69357dcc965bc2: Time spent starting tablet: real 0.021s user 0.024s sys 0.000s
I20250623 01:57:50.572045 14986 tablet_copy_service.cc:342] P b5c946e0639844848a1ffd5e069bd094: Request end of tablet copy session 9420c477c61c4d7daf69357dcc965bc2-af9071880fc946329db80c284049bca8 received from {username='slave'} at 127.14.43.130:45445
I20250623 01:57:50.572504 14986 tablet_copy_service.cc:434] P b5c946e0639844848a1ffd5e069bd094: ending tablet copy session 9420c477c61c4d7daf69357dcc965bc2-af9071880fc946329db80c284049bca8 on tablet af9071880fc946329db80c284049bca8 with peer 9420c477c61c4d7daf69357dcc965bc2
I20250623 01:57:50.913463 14833 raft_consensus.cc:1215] T af9071880fc946329db80c284049bca8 P 9420c477c61c4d7daf69357dcc965bc2 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.4->[4.5-4.5] Dedup: 4.5->[]
W20250623 01:57:51.216504 15216 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet af9071880fc946329db80c284049bca8 on TS 9ce9e2fa9e934e74b7e50719bb379eec failed: Not found: failed to reset TS proxy: Could not find TS for UUID 9ce9e2fa9e934e74b7e50719bb379eec
I20250623 01:57:51.420125 15358 raft_consensus.cc:1062] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094: attempting to promote NON_VOTER 9420c477c61c4d7daf69357dcc965bc2 to VOTER
I20250623 01:57:51.421746 15358 consensus_queue.cc:237] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 } } peers { permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 } } peers { permanent_uuid: "9420c477c61c4d7daf69357dcc965bc2" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 32775 } attrs { promote: false } } unsafe_config_change: true
I20250623 01:57:51.426810 14833 raft_consensus.cc:1273] T af9071880fc946329db80c284049bca8 P 9420c477c61c4d7daf69357dcc965bc2 [term 4 LEARNER]: Refusing update from remote peer b5c946e0639844848a1ffd5e069bd094: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250623 01:57:51.426885 15099 raft_consensus.cc:1273] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [term 4 FOLLOWER]: Refusing update from remote peer b5c946e0639844848a1ffd5e069bd094: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250623 01:57:51.427970 15358 consensus_queue.cc:1035] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:57:51.428642 15361 consensus_queue.cc:1035] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9420c477c61c4d7daf69357dcc965bc2" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250623 01:57:51.434548 15358 raft_consensus.cc:2953] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 9420c477c61c4d7daf69357dcc965bc2 (127.14.43.130) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 } } peers { permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 } } peers { permanent_uuid: "9420c477c61c4d7daf69357dcc965bc2" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 32775 } attrs { promote: false } } unsafe_config_change: true }
I20250623 01:57:51.436054 15099 raft_consensus.cc:2953] T af9071880fc946329db80c284049bca8 P 92705fe433a14034b602e066d7327bf7 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 9420c477c61c4d7daf69357dcc965bc2 (127.14.43.130) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 } } peers { permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 } } peers { permanent_uuid: "9420c477c61c4d7daf69357dcc965bc2" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 32775 } attrs { promote: false } } unsafe_config_change: true }
I20250623 01:57:51.436693 14833 raft_consensus.cc:2953] T af9071880fc946329db80c284049bca8 P 9420c477c61c4d7daf69357dcc965bc2 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 9420c477c61c4d7daf69357dcc965bc2 (127.14.43.130) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 } } peers { permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 } } peers { permanent_uuid: "9420c477c61c4d7daf69357dcc965bc2" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 32775 } attrs { promote: false } } unsafe_config_change: true }
I20250623 01:57:51.446503 15231 catalog_manager.cc:5582] T af9071880fc946329db80c284049bca8 P b5c946e0639844848a1ffd5e069bd094 reported cstate change: config changed from index 5 to 6, 9420c477c61c4d7daf69357dcc965bc2 (127.14.43.130) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "b5c946e0639844848a1ffd5e069bd094" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92705fe433a14034b602e066d7327bf7" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 36877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5c946e0639844848a1ffd5e069bd094" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 33033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9420c477c61c4d7daf69357dcc965bc2" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 32775 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
I20250623 01:57:51.512876 14510 kudu-admin-test.cc:751] Waiting for Master to see new config...
I20250623 01:57:51.526299 14510 kudu-admin-test.cc:756] Tablet locations:
tablet_locations {
tablet_id: "af9071880fc946329db80c284049bca8"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: LEADER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "92705fe433a14034b602e066d7327bf7"
rpc_addresses {
host: "127.14.43.132"
port: 36877
}
}
ts_infos {
permanent_uuid: "b5c946e0639844848a1ffd5e069bd094"
rpc_addresses {
host: "127.14.43.131"
port: 33033
}
}
ts_infos {
permanent_uuid: "9420c477c61c4d7daf69357dcc965bc2"
rpc_addresses {
host: "127.14.43.130"
port: 32775
}
}
I20250623 01:57:51.528553 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 14749
I20250623 01:57:51.547292 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 14882
I20250623 01:57:51.569067 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 15015
I20250623 01:57:51.594626 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 15185
2025-06-23T01:57:51Z chronyd exiting
[ OK ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes (18026 ms)
[ RUN ] AdminCliTest.TestGracefulSpecificLeaderStepDown
I20250623 01:57:51.646212 14510 test_util.cc:276] Using random seed: -1978072602
I20250623 01:57:51.651872 14510 ts_itest-base.cc:115] Starting cluster with:
I20250623 01:57:51.652050 14510 ts_itest-base.cc:116] --------------
I20250623 01:57:51.652204 14510 ts_itest-base.cc:117] 3 tablet servers
I20250623 01:57:51.652335 14510 ts_itest-base.cc:118] 3 replicas per TS
I20250623 01:57:51.652467 14510 ts_itest-base.cc:119] --------------
2025-06-23T01:57:51Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-23T01:57:51Z Disabled control of system clock
I20250623 01:57:51.684219 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.43.190:41909
--webserver_interface=127.14.43.190
--webserver_port=0
--builtin_ntp_servers=127.14.43.148:33379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.43.190:41909
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250623 01:57:51.965508 15379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:57:51.966085 15379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:57:51.966493 15379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:57:51.997107 15379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250623 01:57:51.997390 15379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:57:51.997586 15379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250623 01:57:51.997776 15379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250623 01:57:52.032438 15379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:33379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.14.43.190:41909
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.43.190:41909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.14.43.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:57:52.033634 15379 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:57:52.035213 15379 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:57:52.049746 15386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:57:52.050434 15388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:57:52.054119 15379 server_base.cc:1048] running on GCE node
W20250623 01:57:52.050513 15385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:57:53.133858 15379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:57:53.136371 15379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:57:53.137727 15379 hybrid_clock.cc:648] HybridClock initialized: now 1750643873137709 us; error 54 us; skew 500 ppm
I20250623 01:57:53.138528 15379 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:57:53.149645 15379 webserver.cc:469] Webserver started at http://127.14.43.190:35359/ using document root <none> and password file <none>
I20250623 01:57:53.150590 15379 fs_manager.cc:362] Metadata directory not provided
I20250623 01:57:53.150795 15379 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:57:53.151239 15379 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:57:53.155593 15379 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "3fa7639e8d9e45f5966d4a0ea287a351"
format_stamp: "Formatted at 2025-06-23 01:57:53 on dist-test-slave-9rpg"
I20250623 01:57:53.156651 15379 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "3fa7639e8d9e45f5966d4a0ea287a351"
format_stamp: "Formatted at 2025-06-23 01:57:53 on dist-test-slave-9rpg"
I20250623 01:57:53.163359 15379 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20250623 01:57:53.168704 15395 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:53.169701 15379 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250623 01:57:53.170051 15379 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
uuid: "3fa7639e8d9e45f5966d4a0ea287a351"
format_stamp: "Formatted at 2025-06-23 01:57:53 on dist-test-slave-9rpg"
I20250623 01:57:53.170380 15379 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:57:53.221436 15379 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:57:53.222846 15379 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:57:53.223274 15379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:57:53.288303 15379 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.190:41909
I20250623 01:57:53.288390 15446 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.190:41909 every 8 connection(s)
I20250623 01:57:53.290964 15379 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250623 01:57:53.295797 15447 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:53.296489 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 15379
I20250623 01:57:53.296864 14510 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250623 01:57:53.320246 15447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fa7639e8d9e45f5966d4a0ea287a351: Bootstrap starting.
I20250623 01:57:53.325577 15447 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fa7639e8d9e45f5966d4a0ea287a351: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:53.327404 15447 log.cc:826] T 00000000000000000000000000000000 P 3fa7639e8d9e45f5966d4a0ea287a351: Log is configured to *not* fsync() on all Append() calls
I20250623 01:57:53.331409 15447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fa7639e8d9e45f5966d4a0ea287a351: No bootstrap required, opened a new log
I20250623 01:57:53.347481 15447 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3fa7639e8d9e45f5966d4a0ea287a351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa7639e8d9e45f5966d4a0ea287a351" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 41909 } }
I20250623 01:57:53.348094 15447 raft_consensus.cc:383] T 00000000000000000000000000000000 P 3fa7639e8d9e45f5966d4a0ea287a351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:57:53.348289 15447 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3fa7639e8d9e45f5966d4a0ea287a351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fa7639e8d9e45f5966d4a0ea287a351, State: Initialized, Role: FOLLOWER
I20250623 01:57:53.348860 15447 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fa7639e8d9e45f5966d4a0ea287a351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa7639e8d9e45f5966d4a0ea287a351" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 41909 } }
I20250623 01:57:53.349324 15447 raft_consensus.cc:397] T 00000000000000000000000000000000 P 3fa7639e8d9e45f5966d4a0ea287a351 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250623 01:57:53.349581 15447 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3fa7639e8d9e45f5966d4a0ea287a351 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250623 01:57:53.349877 15447 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 3fa7639e8d9e45f5966d4a0ea287a351 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:53.353737 15447 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3fa7639e8d9e45f5966d4a0ea287a351 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa7639e8d9e45f5966d4a0ea287a351" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 41909 } }
I20250623 01:57:53.354405 15447 leader_election.cc:304] T 00000000000000000000000000000000 P 3fa7639e8d9e45f5966d4a0ea287a351 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fa7639e8d9e45f5966d4a0ea287a351; no voters:
I20250623 01:57:53.355932 15447 leader_election.cc:290] T 00000000000000000000000000000000 P 3fa7639e8d9e45f5966d4a0ea287a351 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250623 01:57:53.356690 15452 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 3fa7639e8d9e45f5966d4a0ea287a351 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:57:53.359613 15452 raft_consensus.cc:695] T 00000000000000000000000000000000 P 3fa7639e8d9e45f5966d4a0ea287a351 [term 1 LEADER]: Becoming Leader. State: Replica: 3fa7639e8d9e45f5966d4a0ea287a351, State: Running, Role: LEADER
I20250623 01:57:53.360433 15452 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fa7639e8d9e45f5966d4a0ea287a351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa7639e8d9e45f5966d4a0ea287a351" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 41909 } }
I20250623 01:57:53.360790 15447 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fa7639e8d9e45f5966d4a0ea287a351 [sys.catalog]: configured and running, proceeding with master startup.
I20250623 01:57:53.373720 15454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fa7639e8d9e45f5966d4a0ea287a351 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fa7639e8d9e45f5966d4a0ea287a351. Latest consensus state: current_term: 1 leader_uuid: "3fa7639e8d9e45f5966d4a0ea287a351" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa7639e8d9e45f5966d4a0ea287a351" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 41909 } } }
I20250623 01:57:53.374668 15454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fa7639e8d9e45f5966d4a0ea287a351 [sys.catalog]: This master's current role is: LEADER
I20250623 01:57:53.370126 15453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fa7639e8d9e45f5966d4a0ea287a351 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fa7639e8d9e45f5966d4a0ea287a351" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa7639e8d9e45f5966d4a0ea287a351" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 41909 } } }
I20250623 01:57:53.376511 15453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fa7639e8d9e45f5966d4a0ea287a351 [sys.catalog]: This master's current role is: LEADER
I20250623 01:57:53.378800 15461 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250623 01:57:53.388715 15461 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250623 01:57:53.405256 15461 catalog_manager.cc:1349] Generated new cluster ID: 06ff1ea4e2af4e71986deddd39b57786
I20250623 01:57:53.405507 15461 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250623 01:57:53.420614 15461 catalog_manager.cc:1372] Generated new certificate authority record
I20250623 01:57:53.421960 15461 catalog_manager.cc:1506] Loading token signing keys...
I20250623 01:57:53.433593 15461 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 3fa7639e8d9e45f5966d4a0ea287a351: Generated new TSK 0
I20250623 01:57:53.434430 15461 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250623 01:57:53.451452 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.43.129:0
--local_ip_for_outbound_sockets=127.14.43.129
--webserver_interface=127.14.43.129
--webserver_port=0
--tserver_master_addrs=127.14.43.190:41909
--builtin_ntp_servers=127.14.43.148:33379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250623 01:57:53.741122 15471 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250623 01:57:53.741737 15471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:57:53.741997 15471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:57:53.742507 15471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:57:53.773208 15471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:57:53.774068 15471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.43.129
I20250623 01:57:53.808467 15471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:33379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.43.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.14.43.129
--webserver_port=0
--tserver_master_addrs=127.14.43.190:41909
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.43.129
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:57:53.809723 15471 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:57:53.811364 15471 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:57:53.870186 15479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:57:53.870414 15471 server_base.cc:1048] running on GCE node
W20250623 01:57:53.870781 15481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:57:53.870554 15478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:57:54.972992 15471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:57:54.975548 15471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:57:54.976996 15471 hybrid_clock.cc:648] HybridClock initialized: now 1750643874976942 us; error 71 us; skew 500 ppm
I20250623 01:57:54.977787 15471 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:57:54.984616 15471 webserver.cc:469] Webserver started at http://127.14.43.129:38227/ using document root <none> and password file <none>
I20250623 01:57:54.985517 15471 fs_manager.cc:362] Metadata directory not provided
I20250623 01:57:54.985719 15471 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:57:54.986204 15471 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:57:54.990646 15471 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "8034f64844464438bc6d8ae70a202e4e"
format_stamp: "Formatted at 2025-06-23 01:57:54 on dist-test-slave-9rpg"
I20250623 01:57:54.991689 15471 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "8034f64844464438bc6d8ae70a202e4e"
format_stamp: "Formatted at 2025-06-23 01:57:54 on dist-test-slave-9rpg"
I20250623 01:57:54.999178 15471 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20250623 01:57:55.004971 15488 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:55.005990 15471 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20250623 01:57:55.006340 15471 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "8034f64844464438bc6d8ae70a202e4e"
format_stamp: "Formatted at 2025-06-23 01:57:54 on dist-test-slave-9rpg"
I20250623 01:57:55.006650 15471 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:57:55.060289 15471 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:57:55.061697 15471 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:57:55.062135 15471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:57:55.064599 15471 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:57:55.068495 15471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250623 01:57:55.068683 15471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:55.068872 15471 ts_tablet_manager.cc:610] Registered 0 tablets
I20250623 01:57:55.069017 15471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:55.194487 15471 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.129:37367
I20250623 01:57:55.194588 15600 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.129:37367 every 8 connection(s)
I20250623 01:57:55.197005 15471 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250623 01:57:55.197655 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 15471
I20250623 01:57:55.198168 14510 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250623 01:57:55.204489 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.43.130:0
--local_ip_for_outbound_sockets=127.14.43.130
--webserver_interface=127.14.43.130
--webserver_port=0
--tserver_master_addrs=127.14.43.190:41909
--builtin_ntp_servers=127.14.43.148:33379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250623 01:57:55.220734 15601 heartbeater.cc:344] Connected to a master server at 127.14.43.190:41909
I20250623 01:57:55.221195 15601 heartbeater.cc:461] Registering TS with master...
I20250623 01:57:55.222245 15601 heartbeater.cc:507] Master 127.14.43.190:41909 requested a full tablet report, sending...
I20250623 01:57:55.224860 15412 ts_manager.cc:194] Registered new tserver with Master: 8034f64844464438bc6d8ae70a202e4e (127.14.43.129:37367)
I20250623 01:57:55.227460 15412 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.43.129:46631
W20250623 01:57:55.491494 15605 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250623 01:57:55.492077 15605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:57:55.492288 15605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:57:55.492714 15605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:57:55.523808 15605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:57:55.524605 15605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.43.130
I20250623 01:57:55.559341 15605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:33379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.43.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.14.43.130
--webserver_port=0
--tserver_master_addrs=127.14.43.190:41909
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.43.130
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:57:55.560545 15605 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:57:55.562175 15605 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:57:55.577003 15612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:57:56.230842 15601 heartbeater.cc:499] Master 127.14.43.190:41909 was elected leader, sending a full tablet report...
W20250623 01:57:55.577427 15611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:57:55.577510 15614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:57:56.652907 15613 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250623 01:57:56.653012 15605 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250623 01:57:56.656672 15605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:57:56.659142 15605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:57:56.660557 15605 hybrid_clock.cc:648] HybridClock initialized: now 1750643876660521 us; error 56 us; skew 500 ppm
I20250623 01:57:56.661350 15605 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:57:56.667604 15605 webserver.cc:469] Webserver started at http://127.14.43.130:39859/ using document root <none> and password file <none>
I20250623 01:57:56.668469 15605 fs_manager.cc:362] Metadata directory not provided
I20250623 01:57:56.668671 15605 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:57:56.669137 15605 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:57:56.673388 15605 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "e74212b2c3ed4a81bc3e086fac7873de"
format_stamp: "Formatted at 2025-06-23 01:57:56 on dist-test-slave-9rpg"
I20250623 01:57:56.674484 15605 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "e74212b2c3ed4a81bc3e086fac7873de"
format_stamp: "Formatted at 2025-06-23 01:57:56 on dist-test-slave-9rpg"
I20250623 01:57:56.681257 15605 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250623 01:57:56.686686 15621 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:56.687659 15605 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250623 01:57:56.687995 15605 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "e74212b2c3ed4a81bc3e086fac7873de"
format_stamp: "Formatted at 2025-06-23 01:57:56 on dist-test-slave-9rpg"
I20250623 01:57:56.688305 15605 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:57:56.737053 15605 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:57:56.738469 15605 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:57:56.738899 15605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:57:56.741281 15605 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:57:56.745151 15605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250623 01:57:56.745347 15605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:56.745569 15605 ts_tablet_manager.cc:610] Registered 0 tablets
I20250623 01:57:56.745716 15605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:56.869469 15605 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.130:44439
I20250623 01:57:56.869576 15733 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.130:44439 every 8 connection(s)
I20250623 01:57:56.871847 15605 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250623 01:57:56.878870 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 15605
I20250623 01:57:56.879222 14510 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250623 01:57:56.884495 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.43.131:0
--local_ip_for_outbound_sockets=127.14.43.131
--webserver_interface=127.14.43.131
--webserver_port=0
--tserver_master_addrs=127.14.43.190:41909
--builtin_ntp_servers=127.14.43.148:33379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250623 01:57:56.891561 15734 heartbeater.cc:344] Connected to a master server at 127.14.43.190:41909
I20250623 01:57:56.891978 15734 heartbeater.cc:461] Registering TS with master...
I20250623 01:57:56.892906 15734 heartbeater.cc:507] Master 127.14.43.190:41909 requested a full tablet report, sending...
I20250623 01:57:56.894896 15412 ts_manager.cc:194] Registered new tserver with Master: e74212b2c3ed4a81bc3e086fac7873de (127.14.43.130:44439)
I20250623 01:57:56.896075 15412 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.43.130:37849
W20250623 01:57:57.167610 15738 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250623 01:57:57.168350 15738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:57:57.168704 15738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:57:57.169442 15738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:57:57.201902 15738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:57:57.202714 15738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.43.131
I20250623 01:57:57.236370 15738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:33379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.43.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.14.43.131
--webserver_port=0
--tserver_master_addrs=127.14.43.190:41909
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.43.131
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:57:57.237919 15738 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:57:57.239576 15738 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:57:57.254343 15745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:57:57.899063 15734 heartbeater.cc:499] Master 127.14.43.190:41909 was elected leader, sending a full tablet report...
W20250623 01:57:57.257416 15747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:57:57.254667 15744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:57:58.326498 15746 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250623 01:57:58.326550 15738 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250623 01:57:58.330453 15738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:57:58.332494 15738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:57:58.333844 15738 hybrid_clock.cc:648] HybridClock initialized: now 1750643878333808 us; error 53 us; skew 500 ppm
I20250623 01:57:58.334622 15738 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:57:58.344046 15738 webserver.cc:469] Webserver started at http://127.14.43.131:38133/ using document root <none> and password file <none>
I20250623 01:57:58.344939 15738 fs_manager.cc:362] Metadata directory not provided
I20250623 01:57:58.345151 15738 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:57:58.345582 15738 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:57:58.349948 15738 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "689ac203c05647e7898a7e84fbf2135b"
format_stamp: "Formatted at 2025-06-23 01:57:58 on dist-test-slave-9rpg"
I20250623 01:57:58.350998 15738 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "689ac203c05647e7898a7e84fbf2135b"
format_stamp: "Formatted at 2025-06-23 01:57:58 on dist-test-slave-9rpg"
I20250623 01:57:58.358132 15738 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250623 01:57:58.363596 15754 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:58.364612 15738 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20250623 01:57:58.364917 15738 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "689ac203c05647e7898a7e84fbf2135b"
format_stamp: "Formatted at 2025-06-23 01:57:58 on dist-test-slave-9rpg"
I20250623 01:57:58.365235 15738 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:57:58.410724 15738 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:57:58.412087 15738 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:57:58.412489 15738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:57:58.414846 15738 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:57:58.418715 15738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250623 01:57:58.418913 15738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:58.419140 15738 ts_tablet_manager.cc:610] Registered 0 tablets
I20250623 01:57:58.419296 15738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250623 01:57:58.545197 15738 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.131:43121
I20250623 01:57:58.545265 15866 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.131:43121 every 8 connection(s)
I20250623 01:57:58.547783 15738 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250623 01:57:58.557353 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 15738
I20250623 01:57:58.557703 14510 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250623 01:57:58.567548 15867 heartbeater.cc:344] Connected to a master server at 127.14.43.190:41909
I20250623 01:57:58.567976 15867 heartbeater.cc:461] Registering TS with master...
I20250623 01:57:58.568938 15867 heartbeater.cc:507] Master 127.14.43.190:41909 requested a full tablet report, sending...
I20250623 01:57:58.570863 15412 ts_manager.cc:194] Registered new tserver with Master: 689ac203c05647e7898a7e84fbf2135b (127.14.43.131:43121)
I20250623 01:57:58.572029 15412 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.43.131:38409
I20250623 01:57:58.576546 14510 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250623 01:57:58.609892 15412 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56748:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250623 01:57:58.628458 15412 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 01:57:58.684922 15536 tablet_service.cc:1468] Processing CreateTablet for tablet 6e0fa892ff914a39aaab3aea4036b4fd (DEFAULT_TABLE table=TestTable [id=389e52d441f9472faa2d407bcf63b5a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 01:57:58.685760 15802 tablet_service.cc:1468] Processing CreateTablet for tablet 6e0fa892ff914a39aaab3aea4036b4fd (DEFAULT_TABLE table=TestTable [id=389e52d441f9472faa2d407bcf63b5a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 01:57:58.686328 15669 tablet_service.cc:1468] Processing CreateTablet for tablet 6e0fa892ff914a39aaab3aea4036b4fd (DEFAULT_TABLE table=TestTable [id=389e52d441f9472faa2d407bcf63b5a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 01:57:58.686797 15536 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e0fa892ff914a39aaab3aea4036b4fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:58.687505 15802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e0fa892ff914a39aaab3aea4036b4fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:58.687605 15669 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e0fa892ff914a39aaab3aea4036b4fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:57:58.707931 15886 tablet_bootstrap.cc:492] T 6e0fa892ff914a39aaab3aea4036b4fd P e74212b2c3ed4a81bc3e086fac7873de: Bootstrap starting.
I20250623 01:57:58.713346 15887 tablet_bootstrap.cc:492] T 6e0fa892ff914a39aaab3aea4036b4fd P 689ac203c05647e7898a7e84fbf2135b: Bootstrap starting.
I20250623 01:57:58.713897 15886 tablet_bootstrap.cc:654] T 6e0fa892ff914a39aaab3aea4036b4fd P e74212b2c3ed4a81bc3e086fac7873de: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:58.716666 15888 tablet_bootstrap.cc:492] T 6e0fa892ff914a39aaab3aea4036b4fd P 8034f64844464438bc6d8ae70a202e4e: Bootstrap starting.
I20250623 01:57:58.718426 15886 log.cc:826] T 6e0fa892ff914a39aaab3aea4036b4fd P e74212b2c3ed4a81bc3e086fac7873de: Log is configured to *not* fsync() on all Append() calls
I20250623 01:57:58.720697 15887 tablet_bootstrap.cc:654] T 6e0fa892ff914a39aaab3aea4036b4fd P 689ac203c05647e7898a7e84fbf2135b: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:58.722301 15888 tablet_bootstrap.cc:654] T 6e0fa892ff914a39aaab3aea4036b4fd P 8034f64844464438bc6d8ae70a202e4e: Neither blocks nor log segments found. Creating new log.
I20250623 01:57:58.722733 15887 log.cc:826] T 6e0fa892ff914a39aaab3aea4036b4fd P 689ac203c05647e7898a7e84fbf2135b: Log is configured to *not* fsync() on all Append() calls
I20250623 01:57:58.723873 15886 tablet_bootstrap.cc:492] T 6e0fa892ff914a39aaab3aea4036b4fd P e74212b2c3ed4a81bc3e086fac7873de: No bootstrap required, opened a new log
I20250623 01:57:58.723937 15888 log.cc:826] T 6e0fa892ff914a39aaab3aea4036b4fd P 8034f64844464438bc6d8ae70a202e4e: Log is configured to *not* fsync() on all Append() calls
I20250623 01:57:58.724426 15886 ts_tablet_manager.cc:1397] T 6e0fa892ff914a39aaab3aea4036b4fd P e74212b2c3ed4a81bc3e086fac7873de: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.008s
I20250623 01:57:58.728935 15887 tablet_bootstrap.cc:492] T 6e0fa892ff914a39aaab3aea4036b4fd P 689ac203c05647e7898a7e84fbf2135b: No bootstrap required, opened a new log
I20250623 01:57:58.729435 15887 ts_tablet_manager.cc:1397] T 6e0fa892ff914a39aaab3aea4036b4fd P 689ac203c05647e7898a7e84fbf2135b: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.005s
I20250623 01:57:58.731118 15888 tablet_bootstrap.cc:492] T 6e0fa892ff914a39aaab3aea4036b4fd P 8034f64844464438bc6d8ae70a202e4e: No bootstrap required, opened a new log
I20250623 01:57:58.731500 15888 ts_tablet_manager.cc:1397] T 6e0fa892ff914a39aaab3aea4036b4fd P 8034f64844464438bc6d8ae70a202e4e: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.004s
I20250623 01:57:58.749537 15888 raft_consensus.cc:357] T 6e0fa892ff914a39aaab3aea4036b4fd P 8034f64844464438bc6d8ae70a202e4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8034f64844464438bc6d8ae70a202e4e" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37367 } } peers { permanent_uuid: "689ac203c05647e7898a7e84fbf2135b" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 43121 } } peers { permanent_uuid: "e74212b2c3ed4a81bc3e086fac7873de" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44439 } }
I20250623 01:57:58.750283 15888 raft_consensus.cc:738] T 6e0fa892ff914a39aaab3aea4036b4fd P 8034f64844464438bc6d8ae70a202e4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8034f64844464438bc6d8ae70a202e4e, State: Initialized, Role: FOLLOWER
I20250623 01:57:58.750877 15888 consensus_queue.cc:260] T 6e0fa892ff914a39aaab3aea4036b4fd P 8034f64844464438bc6d8ae70a202e4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8034f64844464438bc6d8ae70a202e4e" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37367 } } peers { permanent_uuid: "689ac203c05647e7898a7e84fbf2135b" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 43121 } } peers { permanent_uuid: "e74212b2c3ed4a81bc3e086fac7873de" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44439 } }
I20250623 01:57:58.751358 15886 raft_consensus.cc:357] T 6e0fa892ff914a39aaab3aea4036b4fd P e74212b2c3ed4a81bc3e086fac7873de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8034f64844464438bc6d8ae70a202e4e" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37367 } } peers { permanent_uuid: "689ac203c05647e7898a7e84fbf2135b" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 43121 } } peers { permanent_uuid: "e74212b2c3ed4a81bc3e086fac7873de" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44439 } }
I20250623 01:57:58.752324 15886 raft_consensus.cc:738] T 6e0fa892ff914a39aaab3aea4036b4fd P e74212b2c3ed4a81bc3e086fac7873de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e74212b2c3ed4a81bc3e086fac7873de, State: Initialized, Role: FOLLOWER
I20250623 01:57:58.753067 15886 consensus_queue.cc:260] T 6e0fa892ff914a39aaab3aea4036b4fd P e74212b2c3ed4a81bc3e086fac7873de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8034f64844464438bc6d8ae70a202e4e" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37367 } } peers { permanent_uuid: "689ac203c05647e7898a7e84fbf2135b" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 43121 } } peers { permanent_uuid: "e74212b2c3ed4a81bc3e086fac7873de" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44439 } }
I20250623 01:57:58.754639 15888 ts_tablet_manager.cc:1428] T 6e0fa892ff914a39aaab3aea4036b4fd P 8034f64844464438bc6d8ae70a202e4e: Time spent starting tablet: real 0.023s user 0.021s sys 0.003s
I20250623 01:57:58.754930 15887 raft_consensus.cc:357] T 6e0fa892ff914a39aaab3aea4036b4fd P 689ac203c05647e7898a7e84fbf2135b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8034f64844464438bc6d8ae70a202e4e" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37367 } } peers { permanent_uuid: "689ac203c05647e7898a7e84fbf2135b" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 43121 } } peers { permanent_uuid: "e74212b2c3ed4a81bc3e086fac7873de" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44439 } }
I20250623 01:57:58.755832 15887 raft_consensus.cc:738] T 6e0fa892ff914a39aaab3aea4036b4fd P 689ac203c05647e7898a7e84fbf2135b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 689ac203c05647e7898a7e84fbf2135b, State: Initialized, Role: FOLLOWER
I20250623 01:57:58.756595 15887 consensus_queue.cc:260] T 6e0fa892ff914a39aaab3aea4036b4fd P 689ac203c05647e7898a7e84fbf2135b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8034f64844464438bc6d8ae70a202e4e" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37367 } } peers { permanent_uuid: "689ac203c05647e7898a7e84fbf2135b" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 43121 } } peers { permanent_uuid: "e74212b2c3ed4a81bc3e086fac7873de" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44439 } }
I20250623 01:57:58.758879 15886 ts_tablet_manager.cc:1428] T 6e0fa892ff914a39aaab3aea4036b4fd P e74212b2c3ed4a81bc3e086fac7873de: Time spent starting tablet: real 0.034s user 0.030s sys 0.000s
I20250623 01:57:58.760406 15867 heartbeater.cc:499] Master 127.14.43.190:41909 was elected leader, sending a full tablet report...
I20250623 01:57:58.762352 15887 ts_tablet_manager.cc:1428] T 6e0fa892ff914a39aaab3aea4036b4fd P 689ac203c05647e7898a7e84fbf2135b: Time spent starting tablet: real 0.033s user 0.029s sys 0.003s
I20250623 01:57:58.788033 14510 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250623 01:57:58.791064 14510 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 8034f64844464438bc6d8ae70a202e4e to finish bootstrapping
W20250623 01:57:58.801972 15868 tablet.cc:2378] T 6e0fa892ff914a39aaab3aea4036b4fd P 689ac203c05647e7898a7e84fbf2135b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250623 01:57:58.803736 14510 ts_itest-base.cc:246] Waiting for 1 tablets on tserver e74212b2c3ed4a81bc3e086fac7873de to finish bootstrapping
I20250623 01:57:58.813386 14510 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 689ac203c05647e7898a7e84fbf2135b to finish bootstrapping
I20250623 01:57:58.853286 15556 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "6e0fa892ff914a39aaab3aea4036b4fd"
dest_uuid: "8034f64844464438bc6d8ae70a202e4e"
from {username='slave'} at 127.0.0.1:39352
I20250623 01:57:58.853801 15556 raft_consensus.cc:491] T 6e0fa892ff914a39aaab3aea4036b4fd P 8034f64844464438bc6d8ae70a202e4e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:57:58.854099 15556 raft_consensus.cc:3058] T 6e0fa892ff914a39aaab3aea4036b4fd P 8034f64844464438bc6d8ae70a202e4e [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:58.858830 15556 raft_consensus.cc:513] T 6e0fa892ff914a39aaab3aea4036b4fd P 8034f64844464438bc6d8ae70a202e4e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8034f64844464438bc6d8ae70a202e4e" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37367 } } peers { permanent_uuid: "689ac203c05647e7898a7e84fbf2135b" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 43121 } } peers { permanent_uuid: "e74212b2c3ed4a81bc3e086fac7873de" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44439 } }
I20250623 01:57:58.861001 15556 leader_election.cc:290] T 6e0fa892ff914a39aaab3aea4036b4fd P 8034f64844464438bc6d8ae70a202e4e [CANDIDATE]: Term 1 election: Requested vote from peers 689ac203c05647e7898a7e84fbf2135b (127.14.43.131:43121), e74212b2c3ed4a81bc3e086fac7873de (127.14.43.130:44439)
I20250623 01:57:58.871062 14510 cluster_itest_util.cc:257] Not converged past 1 yet: 0.0 0.0 0.0
I20250623 01:57:58.873728 15822 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e0fa892ff914a39aaab3aea4036b4fd" candidate_uuid: "8034f64844464438bc6d8ae70a202e4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "689ac203c05647e7898a7e84fbf2135b"
I20250623 01:57:58.874435 15822 raft_consensus.cc:3058] T 6e0fa892ff914a39aaab3aea4036b4fd P 689ac203c05647e7898a7e84fbf2135b [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:57:58.874567 15689 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e0fa892ff914a39aaab3aea4036b4fd" candidate_uuid: "8034f64844464438bc6d8ae70a202e4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e74212b2c3ed4a81bc3e086fac7873de"
I20250623 01:57:58.875265 15689 raft_consensus.cc:3058] T 6e0fa892ff914a39aaab3aea4036b4fd P e74212b2c3ed4a81bc3e086fac7873de [term 0 FOLLOWER]: Advancing to term 1
W20250623 01:57:58.877735 15735 tablet.cc:2378] T 6e0fa892ff914a39aaab3aea4036b4fd P e74212b2c3ed4a81bc3e086fac7873de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250623 01:57:58.881035 15822 raft_consensus.cc:2466] T 6e0fa892ff914a39aaab3aea4036b4fd P 689ac203c05647e7898a7e84fbf2135b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8034f64844464438bc6d8ae70a202e4e in term 1.
I20250623 01:57:58.881704 15689 raft_consensus.cc:2466] T 6e0fa892ff914a39aaab3aea4036b4fd P e74212b2c3ed4a81bc3e086fac7873de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8034f64844464438bc6d8ae70a202e4e in term 1.
I20250623 01:57:58.882335 15491 leader_election.cc:304] T 6e0fa892ff914a39aaab3aea4036b4fd P 8034f64844464438bc6d8ae70a202e4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 689ac203c05647e7898a7e84fbf2135b, 8034f64844464438bc6d8ae70a202e4e; no voters:
I20250623 01:57:58.883445 15892 raft_consensus.cc:2802] T 6e0fa892ff914a39aaab3aea4036b4fd P 8034f64844464438bc6d8ae70a202e4e [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:57:58.885286 15892 raft_consensus.cc:695] T 6e0fa892ff914a39aaab3aea4036b4fd P 8034f64844464438bc6d8ae70a202e4e [term 1 LEADER]: Becoming Leader. State: Replica: 8034f64844464438bc6d8ae70a202e4e, State: Running, Role: LEADER
I20250623 01:57:58.886169 15892 consensus_queue.cc:237] T 6e0fa892ff914a39aaab3aea4036b4fd P 8034f64844464438bc6d8ae70a202e4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8034f64844464438bc6d8ae70a202e4e" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37367 } } peers { permanent_uuid: "689ac203c05647e7898a7e84fbf2135b" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 43121 } } peers { permanent_uuid: "e74212b2c3ed4a81bc3e086fac7873de" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44439 } }
I20250623 01:57:58.898531 15412 catalog_manager.cc:5582] T 6e0fa892ff914a39aaab3aea4036b4fd P 8034f64844464438bc6d8ae70a202e4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8034f64844464438bc6d8ae70a202e4e (127.14.43.129). New cstate: current_term: 1 leader_uuid: "8034f64844464438bc6d8ae70a202e4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8034f64844464438bc6d8ae70a202e4e" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "689ac203c05647e7898a7e84fbf2135b" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 43121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e74212b2c3ed4a81bc3e086fac7873de" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44439 } health_report { overall_health: UNKNOWN } } }
W20250623 01:57:58.959173 15602 tablet.cc:2378] T 6e0fa892ff914a39aaab3aea4036b4fd P 8034f64844464438bc6d8ae70a202e4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250623 01:57:58.976444 14510 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250623 01:57:59.181684 14510 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250623 01:57:59.408926 15901 consensus_queue.cc:1035] T 6e0fa892ff914a39aaab3aea4036b4fd P 8034f64844464438bc6d8ae70a202e4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "689ac203c05647e7898a7e84fbf2135b" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 43121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:57:59.424901 15904 consensus_queue.cc:1035] T 6e0fa892ff914a39aaab3aea4036b4fd P 8034f64844464438bc6d8ae70a202e4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e74212b2c3ed4a81bc3e086fac7873de" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:01.026091 15556 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "6e0fa892ff914a39aaab3aea4036b4fd"
dest_uuid: "8034f64844464438bc6d8ae70a202e4e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39360
I20250623 01:58:01.026579 15556 raft_consensus.cc:604] T 6e0fa892ff914a39aaab3aea4036b4fd P 8034f64844464438bc6d8ae70a202e4e [term 1 LEADER]: Received request to transfer leadership
I20250623 01:58:01.270088 15901 raft_consensus.cc:991] T 6e0fa892ff914a39aaab3aea4036b4fd P 8034f64844464438bc6d8ae70a202e4e: : Instructing follower e74212b2c3ed4a81bc3e086fac7873de to start an election
I20250623 01:58:01.270504 15930 raft_consensus.cc:1079] T 6e0fa892ff914a39aaab3aea4036b4fd P 8034f64844464438bc6d8ae70a202e4e [term 1 LEADER]: Signalling peer e74212b2c3ed4a81bc3e086fac7873de to start an election
I20250623 01:58:01.271763 15689 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "6e0fa892ff914a39aaab3aea4036b4fd"
dest_uuid: "e74212b2c3ed4a81bc3e086fac7873de"
from {username='slave'} at 127.14.43.129:44055
I20250623 01:58:01.272254 15689 raft_consensus.cc:491] T 6e0fa892ff914a39aaab3aea4036b4fd P e74212b2c3ed4a81bc3e086fac7873de [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250623 01:58:01.272502 15689 raft_consensus.cc:3058] T 6e0fa892ff914a39aaab3aea4036b4fd P e74212b2c3ed4a81bc3e086fac7873de [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:01.276358 15689 raft_consensus.cc:513] T 6e0fa892ff914a39aaab3aea4036b4fd P e74212b2c3ed4a81bc3e086fac7873de [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8034f64844464438bc6d8ae70a202e4e" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37367 } } peers { permanent_uuid: "689ac203c05647e7898a7e84fbf2135b" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 43121 } } peers { permanent_uuid: "e74212b2c3ed4a81bc3e086fac7873de" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44439 } }
I20250623 01:58:01.278316 15689 leader_election.cc:290] T 6e0fa892ff914a39aaab3aea4036b4fd P e74212b2c3ed4a81bc3e086fac7873de [CANDIDATE]: Term 2 election: Requested vote from peers 8034f64844464438bc6d8ae70a202e4e (127.14.43.129:37367), 689ac203c05647e7898a7e84fbf2135b (127.14.43.131:43121)
I20250623 01:58:01.289007 15556 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e0fa892ff914a39aaab3aea4036b4fd" candidate_uuid: "e74212b2c3ed4a81bc3e086fac7873de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8034f64844464438bc6d8ae70a202e4e"
I20250623 01:58:01.289573 15556 raft_consensus.cc:3053] T 6e0fa892ff914a39aaab3aea4036b4fd P 8034f64844464438bc6d8ae70a202e4e [term 1 LEADER]: Stepping down as leader of term 1
I20250623 01:58:01.289865 15556 raft_consensus.cc:738] T 6e0fa892ff914a39aaab3aea4036b4fd P 8034f64844464438bc6d8ae70a202e4e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8034f64844464438bc6d8ae70a202e4e, State: Running, Role: LEADER
I20250623 01:58:01.290505 15556 consensus_queue.cc:260] T 6e0fa892ff914a39aaab3aea4036b4fd P 8034f64844464438bc6d8ae70a202e4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8034f64844464438bc6d8ae70a202e4e" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37367 } } peers { permanent_uuid: "689ac203c05647e7898a7e84fbf2135b" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 43121 } } peers { permanent_uuid: "e74212b2c3ed4a81bc3e086fac7873de" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44439 } }
I20250623 01:58:01.290732 15822 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e0fa892ff914a39aaab3aea4036b4fd" candidate_uuid: "e74212b2c3ed4a81bc3e086fac7873de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "689ac203c05647e7898a7e84fbf2135b"
I20250623 01:58:01.291239 15822 raft_consensus.cc:3058] T 6e0fa892ff914a39aaab3aea4036b4fd P 689ac203c05647e7898a7e84fbf2135b [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:01.291391 15556 raft_consensus.cc:3058] T 6e0fa892ff914a39aaab3aea4036b4fd P 8034f64844464438bc6d8ae70a202e4e [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:01.295300 15556 raft_consensus.cc:2466] T 6e0fa892ff914a39aaab3aea4036b4fd P 8034f64844464438bc6d8ae70a202e4e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e74212b2c3ed4a81bc3e086fac7873de in term 2.
I20250623 01:58:01.296237 15623 leader_election.cc:304] T 6e0fa892ff914a39aaab3aea4036b4fd P e74212b2c3ed4a81bc3e086fac7873de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8034f64844464438bc6d8ae70a202e4e, e74212b2c3ed4a81bc3e086fac7873de; no voters:
I20250623 01:58:01.297266 15822 raft_consensus.cc:2466] T 6e0fa892ff914a39aaab3aea4036b4fd P 689ac203c05647e7898a7e84fbf2135b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e74212b2c3ed4a81bc3e086fac7873de in term 2.
I20250623 01:58:01.298357 15934 raft_consensus.cc:2802] T 6e0fa892ff914a39aaab3aea4036b4fd P e74212b2c3ed4a81bc3e086fac7873de [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:58:01.299757 15934 raft_consensus.cc:695] T 6e0fa892ff914a39aaab3aea4036b4fd P e74212b2c3ed4a81bc3e086fac7873de [term 2 LEADER]: Becoming Leader. State: Replica: e74212b2c3ed4a81bc3e086fac7873de, State: Running, Role: LEADER
I20250623 01:58:01.300462 15934 consensus_queue.cc:237] T 6e0fa892ff914a39aaab3aea4036b4fd P e74212b2c3ed4a81bc3e086fac7873de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8034f64844464438bc6d8ae70a202e4e" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37367 } } peers { permanent_uuid: "689ac203c05647e7898a7e84fbf2135b" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 43121 } } peers { permanent_uuid: "e74212b2c3ed4a81bc3e086fac7873de" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44439 } }
I20250623 01:58:01.308248 15411 catalog_manager.cc:5582] T 6e0fa892ff914a39aaab3aea4036b4fd P e74212b2c3ed4a81bc3e086fac7873de reported cstate change: term changed from 1 to 2, leader changed from 8034f64844464438bc6d8ae70a202e4e (127.14.43.129) to e74212b2c3ed4a81bc3e086fac7873de (127.14.43.130). New cstate: current_term: 2 leader_uuid: "e74212b2c3ed4a81bc3e086fac7873de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8034f64844464438bc6d8ae70a202e4e" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "689ac203c05647e7898a7e84fbf2135b" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 43121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e74212b2c3ed4a81bc3e086fac7873de" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44439 } health_report { overall_health: HEALTHY } } }
I20250623 01:58:01.688278 15822 raft_consensus.cc:1273] T 6e0fa892ff914a39aaab3aea4036b4fd P 689ac203c05647e7898a7e84fbf2135b [term 2 FOLLOWER]: Refusing update from remote peer e74212b2c3ed4a81bc3e086fac7873de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:58:01.689435 15934 consensus_queue.cc:1035] T 6e0fa892ff914a39aaab3aea4036b4fd P e74212b2c3ed4a81bc3e086fac7873de [LEADER]: Connected to new peer: Peer: permanent_uuid: "689ac203c05647e7898a7e84fbf2135b" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 43121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 01:58:01.700210 15556 raft_consensus.cc:1273] T 6e0fa892ff914a39aaab3aea4036b4fd P 8034f64844464438bc6d8ae70a202e4e [term 2 FOLLOWER]: Refusing update from remote peer e74212b2c3ed4a81bc3e086fac7873de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 01:58:01.701508 15935 consensus_queue.cc:1035] T 6e0fa892ff914a39aaab3aea4036b4fd P e74212b2c3ed4a81bc3e086fac7873de [LEADER]: Connected to new peer: Peer: permanent_uuid: "8034f64844464438bc6d8ae70a202e4e" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250623 01:58:03.589071 15556 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "6e0fa892ff914a39aaab3aea4036b4fd"
dest_uuid: "8034f64844464438bc6d8ae70a202e4e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39376
I20250623 01:58:03.589644 15556 raft_consensus.cc:604] T 6e0fa892ff914a39aaab3aea4036b4fd P 8034f64844464438bc6d8ae70a202e4e [term 2 FOLLOWER]: Received request to transfer leadership
I20250623 01:58:03.589977 15556 raft_consensus.cc:612] T 6e0fa892ff914a39aaab3aea4036b4fd P 8034f64844464438bc6d8ae70a202e4e [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250623 01:58:04.620630 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 15471
I20250623 01:58:04.639739 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 15605
I20250623 01:58:04.661111 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 15738
I20250623 01:58:04.680730 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 15379
2025-06-23T01:58:04Z chronyd exiting
[ OK ] AdminCliTest.TestGracefulSpecificLeaderStepDown (13082 ms)
[ RUN ] AdminCliTest.TestDescribeTableColumnFlags
I20250623 01:58:04.728773 14510 test_util.cc:276] Using random seed: -1964990036
I20250623 01:58:04.732703 14510 ts_itest-base.cc:115] Starting cluster with:
I20250623 01:58:04.732865 14510 ts_itest-base.cc:116] --------------
I20250623 01:58:04.732980 14510 ts_itest-base.cc:117] 3 tablet servers
I20250623 01:58:04.733083 14510 ts_itest-base.cc:118] 3 replicas per TS
I20250623 01:58:04.733192 14510 ts_itest-base.cc:119] --------------
2025-06-23T01:58:04Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-23T01:58:04Z Disabled control of system clock
I20250623 01:58:04.765343 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.43.190:35095
--webserver_interface=127.14.43.190
--webserver_port=0
--builtin_ntp_servers=127.14.43.148:33955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.43.190:35095 with env {}
W20250623 01:58:05.046262 15979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:58:05.046799 15979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:58:05.047210 15979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:58:05.087361 15979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250623 01:58:05.087666 15979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:58:05.087929 15979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250623 01:58:05.088231 15979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250623 01:58:05.123338 15979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:33955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.43.190:35095
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.43.190:35095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.14.43.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:58:05.124591 15979 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:58:05.126221 15979 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:58:05.140357 15985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:58:05.141830 15979 server_base.cc:1048] running on GCE node
W20250623 01:58:05.140718 15986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:58:05.141117 15988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:58:06.232501 15979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:58:06.235055 15979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:58:06.236419 15979 hybrid_clock.cc:648] HybridClock initialized: now 1750643886236392 us; error 43 us; skew 500 ppm
I20250623 01:58:06.237207 15979 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:58:06.243144 15979 webserver.cc:469] Webserver started at http://127.14.43.190:43765/ using document root <none> and password file <none>
I20250623 01:58:06.244035 15979 fs_manager.cc:362] Metadata directory not provided
I20250623 01:58:06.244240 15979 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:58:06.244673 15979 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:58:06.248961 15979 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "0d7b977a6168463da3204adfd8dad7b0"
format_stamp: "Formatted at 2025-06-23 01:58:06 on dist-test-slave-9rpg"
I20250623 01:58:06.250077 15979 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "0d7b977a6168463da3204adfd8dad7b0"
format_stamp: "Formatted at 2025-06-23 01:58:06 on dist-test-slave-9rpg"
I20250623 01:58:06.256800 15979 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250623 01:58:06.262205 15995 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:06.263175 15979 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250623 01:58:06.263473 15979 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
uuid: "0d7b977a6168463da3204adfd8dad7b0"
format_stamp: "Formatted at 2025-06-23 01:58:06 on dist-test-slave-9rpg"
I20250623 01:58:06.263782 15979 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:58:06.305581 15979 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:58:06.307055 15979 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:58:06.307472 15979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:58:06.373859 15979 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.190:35095
I20250623 01:58:06.373927 16046 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.190:35095 every 8 connection(s)
I20250623 01:58:06.376458 15979 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250623 01:58:06.381287 16047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:06.382436 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 15979
I20250623 01:58:06.382910 14510 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250623 01:58:06.405684 16047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d7b977a6168463da3204adfd8dad7b0: Bootstrap starting.
I20250623 01:58:06.411355 16047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d7b977a6168463da3204adfd8dad7b0: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:06.412912 16047 log.cc:826] T 00000000000000000000000000000000 P 0d7b977a6168463da3204adfd8dad7b0: Log is configured to *not* fsync() on all Append() calls
I20250623 01:58:06.416929 16047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d7b977a6168463da3204adfd8dad7b0: No bootstrap required, opened a new log
I20250623 01:58:06.433727 16047 raft_consensus.cc:357] T 00000000000000000000000000000000 P 0d7b977a6168463da3204adfd8dad7b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d7b977a6168463da3204adfd8dad7b0" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 35095 } }
I20250623 01:58:06.434509 16047 raft_consensus.cc:383] T 00000000000000000000000000000000 P 0d7b977a6168463da3204adfd8dad7b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:06.434732 16047 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0d7b977a6168463da3204adfd8dad7b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7b977a6168463da3204adfd8dad7b0, State: Initialized, Role: FOLLOWER
I20250623 01:58:06.435487 16047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d7b977a6168463da3204adfd8dad7b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d7b977a6168463da3204adfd8dad7b0" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 35095 } }
I20250623 01:58:06.435968 16047 raft_consensus.cc:397] T 00000000000000000000000000000000 P 0d7b977a6168463da3204adfd8dad7b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250623 01:58:06.436234 16047 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0d7b977a6168463da3204adfd8dad7b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250623 01:58:06.436512 16047 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 0d7b977a6168463da3204adfd8dad7b0 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:06.440348 16047 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0d7b977a6168463da3204adfd8dad7b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d7b977a6168463da3204adfd8dad7b0" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 35095 } }
I20250623 01:58:06.441016 16047 leader_election.cc:304] T 00000000000000000000000000000000 P 0d7b977a6168463da3204adfd8dad7b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d7b977a6168463da3204adfd8dad7b0; no voters:
I20250623 01:58:06.442795 16047 leader_election.cc:290] T 00000000000000000000000000000000 P 0d7b977a6168463da3204adfd8dad7b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250623 01:58:06.443447 16052 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 0d7b977a6168463da3204adfd8dad7b0 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:06.445394 16052 raft_consensus.cc:695] T 00000000000000000000000000000000 P 0d7b977a6168463da3204adfd8dad7b0 [term 1 LEADER]: Becoming Leader. State: Replica: 0d7b977a6168463da3204adfd8dad7b0, State: Running, Role: LEADER
I20250623 01:58:06.446156 16052 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d7b977a6168463da3204adfd8dad7b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d7b977a6168463da3204adfd8dad7b0" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 35095 } }
I20250623 01:58:06.447096 16047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d7b977a6168463da3204adfd8dad7b0 [sys.catalog]: configured and running, proceeding with master startup.
I20250623 01:58:06.455389 16053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d7b977a6168463da3204adfd8dad7b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d7b977a6168463da3204adfd8dad7b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d7b977a6168463da3204adfd8dad7b0" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 35095 } } }
I20250623 01:58:06.455482 16054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d7b977a6168463da3204adfd8dad7b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d7b977a6168463da3204adfd8dad7b0. Latest consensus state: current_term: 1 leader_uuid: "0d7b977a6168463da3204adfd8dad7b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d7b977a6168463da3204adfd8dad7b0" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 35095 } } }
I20250623 01:58:06.456249 16053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d7b977a6168463da3204adfd8dad7b0 [sys.catalog]: This master's current role is: LEADER
I20250623 01:58:06.456260 16054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d7b977a6168463da3204adfd8dad7b0 [sys.catalog]: This master's current role is: LEADER
I20250623 01:58:06.464686 16061 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250623 01:58:06.475081 16061 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250623 01:58:06.490832 16061 catalog_manager.cc:1349] Generated new cluster ID: 4ad1e4227a544b9abd1eea1f6490d708
I20250623 01:58:06.491120 16061 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250623 01:58:06.531689 16061 catalog_manager.cc:1372] Generated new certificate authority record
I20250623 01:58:06.533674 16061 catalog_manager.cc:1506] Loading token signing keys...
I20250623 01:58:06.551350 16061 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 0d7b977a6168463da3204adfd8dad7b0: Generated new TSK 0
I20250623 01:58:06.552450 16061 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250623 01:58:06.572381 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.43.129:0
--local_ip_for_outbound_sockets=127.14.43.129
--webserver_interface=127.14.43.129
--webserver_port=0
--tserver_master_addrs=127.14.43.190:35095
--builtin_ntp_servers=127.14.43.148:33955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250623 01:58:06.861212 16071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:58:06.861706 16071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:58:06.862233 16071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:58:06.892742 16071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:58:06.893571 16071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.43.129
I20250623 01:58:06.928730 16071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:33955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.43.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.14.43.129
--webserver_port=0
--tserver_master_addrs=127.14.43.190:35095
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.43.129
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:58:06.929998 16071 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:58:06.931682 16071 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:58:06.949519 16080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:58:06.950973 16077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:58:08.029006 16071 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250623 01:58:06.949605 16078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:58:08.028961 16079 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250623 01:58:08.033486 16071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:58:08.036031 16071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:58:08.037472 16071 hybrid_clock.cc:648] HybridClock initialized: now 1750643888037414 us; error 73 us; skew 500 ppm
I20250623 01:58:08.038316 16071 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:58:08.045102 16071 webserver.cc:469] Webserver started at http://127.14.43.129:35717/ using document root <none> and password file <none>
I20250623 01:58:08.046058 16071 fs_manager.cc:362] Metadata directory not provided
I20250623 01:58:08.046277 16071 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:58:08.046733 16071 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:58:08.051103 16071 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b45591d1dbb44c5d9215a04523ed6125"
format_stamp: "Formatted at 2025-06-23 01:58:08 on dist-test-slave-9rpg"
I20250623 01:58:08.052170 16071 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b45591d1dbb44c5d9215a04523ed6125"
format_stamp: "Formatted at 2025-06-23 01:58:08 on dist-test-slave-9rpg"
I20250623 01:58:08.059378 16071 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20250623 01:58:08.065161 16087 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:08.066285 16071 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250623 01:58:08.066591 16071 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b45591d1dbb44c5d9215a04523ed6125"
format_stamp: "Formatted at 2025-06-23 01:58:08 on dist-test-slave-9rpg"
I20250623 01:58:08.066896 16071 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:58:08.135965 16071 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:58:08.137449 16071 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:58:08.137869 16071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:58:08.140712 16071 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:58:08.144855 16071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250623 01:58:08.145048 16071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:08.145313 16071 ts_tablet_manager.cc:610] Registered 0 tablets
I20250623 01:58:08.145465 16071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:08.281967 16071 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.129:37671
I20250623 01:58:08.282135 16199 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.129:37671 every 8 connection(s)
I20250623 01:58:08.284489 16071 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250623 01:58:08.287555 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 16071
I20250623 01:58:08.288004 14510 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250623 01:58:08.294530 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.43.130:0
--local_ip_for_outbound_sockets=127.14.43.130
--webserver_interface=127.14.43.130
--webserver_port=0
--tserver_master_addrs=127.14.43.190:35095
--builtin_ntp_servers=127.14.43.148:33955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250623 01:58:08.305374 16200 heartbeater.cc:344] Connected to a master server at 127.14.43.190:35095
I20250623 01:58:08.305779 16200 heartbeater.cc:461] Registering TS with master...
I20250623 01:58:08.306758 16200 heartbeater.cc:507] Master 127.14.43.190:35095 requested a full tablet report, sending...
I20250623 01:58:08.309383 16012 ts_manager.cc:194] Registered new tserver with Master: b45591d1dbb44c5d9215a04523ed6125 (127.14.43.129:37671)
I20250623 01:58:08.312103 16012 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.43.129:42043
W20250623 01:58:08.587219 16204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:58:08.587682 16204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:58:08.588120 16204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:58:08.618742 16204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:58:08.619617 16204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.43.130
I20250623 01:58:08.653973 16204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:33955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.43.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.14.43.130
--webserver_port=0
--tserver_master_addrs=127.14.43.190:35095
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.43.130
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:58:08.655282 16204 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:58:08.656881 16204 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:58:08.671550 16210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:58:09.315675 16200 heartbeater.cc:499] Master 127.14.43.190:35095 was elected leader, sending a full tablet report...
W20250623 01:58:08.672077 16211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:58:08.673130 16213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:58:09.748009 16212 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250623 01:58:09.748062 16204 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250623 01:58:09.751760 16204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:58:09.753857 16204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:58:09.755264 16204 hybrid_clock.cc:648] HybridClock initialized: now 1750643889755234 us; error 42 us; skew 500 ppm
I20250623 01:58:09.756209 16204 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:58:09.765416 16204 webserver.cc:469] Webserver started at http://127.14.43.130:39261/ using document root <none> and password file <none>
I20250623 01:58:09.766373 16204 fs_manager.cc:362] Metadata directory not provided
I20250623 01:58:09.766579 16204 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:58:09.766996 16204 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:58:09.771311 16204 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "458e6bc4a3114c98b3583d8cad507d39"
format_stamp: "Formatted at 2025-06-23 01:58:09 on dist-test-slave-9rpg"
I20250623 01:58:09.772384 16204 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "458e6bc4a3114c98b3583d8cad507d39"
format_stamp: "Formatted at 2025-06-23 01:58:09 on dist-test-slave-9rpg"
I20250623 01:58:09.779433 16204 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250623 01:58:09.784969 16220 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:09.786005 16204 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250623 01:58:09.786343 16204 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "458e6bc4a3114c98b3583d8cad507d39"
format_stamp: "Formatted at 2025-06-23 01:58:09 on dist-test-slave-9rpg"
I20250623 01:58:09.786664 16204 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:58:09.854640 16204 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:58:09.856032 16204 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:58:09.856436 16204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:58:09.858840 16204 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:58:09.862882 16204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250623 01:58:09.863086 16204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:09.863297 16204 ts_tablet_manager.cc:610] Registered 0 tablets
I20250623 01:58:09.863442 16204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:09.987918 16204 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.130:37965
I20250623 01:58:09.988021 16332 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.130:37965 every 8 connection(s)
I20250623 01:58:09.990379 16204 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250623 01:58:09.991005 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 16204
I20250623 01:58:09.991392 14510 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250623 01:58:09.998364 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.43.131:0
--local_ip_for_outbound_sockets=127.14.43.131
--webserver_interface=127.14.43.131
--webserver_port=0
--tserver_master_addrs=127.14.43.190:35095
--builtin_ntp_servers=127.14.43.148:33955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250623 01:58:10.011132 16333 heartbeater.cc:344] Connected to a master server at 127.14.43.190:35095
I20250623 01:58:10.011561 16333 heartbeater.cc:461] Registering TS with master...
I20250623 01:58:10.012544 16333 heartbeater.cc:507] Master 127.14.43.190:35095 requested a full tablet report, sending...
I20250623 01:58:10.014703 16012 ts_manager.cc:194] Registered new tserver with Master: 458e6bc4a3114c98b3583d8cad507d39 (127.14.43.130:37965)
I20250623 01:58:10.015898 16012 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.43.130:42611
W20250623 01:58:10.283721 16337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:58:10.284206 16337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:58:10.284674 16337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:58:10.315973 16337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:58:10.316877 16337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.43.131
I20250623 01:58:10.352161 16337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:33955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.43.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.14.43.131
--webserver_port=0
--tserver_master_addrs=127.14.43.190:35095
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.43.131
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:58:10.353461 16337 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:58:10.355058 16337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:58:10.371021 16344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:58:11.018941 16333 heartbeater.cc:499] Master 127.14.43.190:35095 was elected leader, sending a full tablet report...
W20250623 01:58:10.372223 16343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:58:10.371148 16346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:58:11.451911 16345 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250623 01:58:11.452165 16337 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250623 01:58:11.455857 16337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:58:11.457942 16337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:58:11.459359 16337 hybrid_clock.cc:648] HybridClock initialized: now 1750643891459303 us; error 70 us; skew 500 ppm
I20250623 01:58:11.460130 16337 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:58:11.466342 16337 webserver.cc:469] Webserver started at http://127.14.43.131:32863/ using document root <none> and password file <none>
I20250623 01:58:11.467219 16337 fs_manager.cc:362] Metadata directory not provided
I20250623 01:58:11.467432 16337 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:58:11.467860 16337 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:58:11.472191 16337 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "acdd72e995154240b079652dd709dcb0"
format_stamp: "Formatted at 2025-06-23 01:58:11 on dist-test-slave-9rpg"
I20250623 01:58:11.473240 16337 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "acdd72e995154240b079652dd709dcb0"
format_stamp: "Formatted at 2025-06-23 01:58:11 on dist-test-slave-9rpg"
I20250623 01:58:11.480451 16337 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20250623 01:58:11.485714 16353 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:11.486722 16337 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250623 01:58:11.487010 16337 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "acdd72e995154240b079652dd709dcb0"
format_stamp: "Formatted at 2025-06-23 01:58:11 on dist-test-slave-9rpg"
I20250623 01:58:11.487316 16337 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:58:11.544389 16337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:58:11.545771 16337 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:58:11.546206 16337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:58:11.548643 16337 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:58:11.552469 16337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250623 01:58:11.552661 16337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:11.552855 16337 ts_tablet_manager.cc:610] Registered 0 tablets
I20250623 01:58:11.552980 16337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:11.676514 16337 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.131:43645
I20250623 01:58:11.676620 16465 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.131:43645 every 8 connection(s)
I20250623 01:58:11.678969 16337 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250623 01:58:11.682914 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 16337
I20250623 01:58:11.683342 14510 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250623 01:58:11.698930 16466 heartbeater.cc:344] Connected to a master server at 127.14.43.190:35095
I20250623 01:58:11.699345 16466 heartbeater.cc:461] Registering TS with master...
I20250623 01:58:11.700335 16466 heartbeater.cc:507] Master 127.14.43.190:35095 requested a full tablet report, sending...
I20250623 01:58:11.702488 16012 ts_manager.cc:194] Registered new tserver with Master: acdd72e995154240b079652dd709dcb0 (127.14.43.131:43645)
I20250623 01:58:11.703889 14510 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250623 01:58:11.704068 16012 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.43.131:37945
I20250623 01:58:11.733392 16012 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53538:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250623 01:58:11.751430 16012 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 01:58:11.801896 16268 tablet_service.cc:1468] Processing CreateTablet for tablet bba9194608a945d99fd47609b3762184 (DEFAULT_TABLE table=TestTable [id=b65026b4eaae4b89a5e3e0902af222a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 01:58:11.802412 16135 tablet_service.cc:1468] Processing CreateTablet for tablet bba9194608a945d99fd47609b3762184 (DEFAULT_TABLE table=TestTable [id=b65026b4eaae4b89a5e3e0902af222a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 01:58:11.803037 16401 tablet_service.cc:1468] Processing CreateTablet for tablet bba9194608a945d99fd47609b3762184 (DEFAULT_TABLE table=TestTable [id=b65026b4eaae4b89a5e3e0902af222a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 01:58:11.803790 16268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bba9194608a945d99fd47609b3762184. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.804112 16135 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bba9194608a945d99fd47609b3762184. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.804454 16401 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bba9194608a945d99fd47609b3762184. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:11.825620 16485 tablet_bootstrap.cc:492] T bba9194608a945d99fd47609b3762184 P acdd72e995154240b079652dd709dcb0: Bootstrap starting.
I20250623 01:58:11.830523 16486 tablet_bootstrap.cc:492] T bba9194608a945d99fd47609b3762184 P b45591d1dbb44c5d9215a04523ed6125: Bootstrap starting.
I20250623 01:58:11.833150 16485 tablet_bootstrap.cc:654] T bba9194608a945d99fd47609b3762184 P acdd72e995154240b079652dd709dcb0: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:11.833426 16487 tablet_bootstrap.cc:492] T bba9194608a945d99fd47609b3762184 P 458e6bc4a3114c98b3583d8cad507d39: Bootstrap starting.
I20250623 01:58:11.835559 16485 log.cc:826] T bba9194608a945d99fd47609b3762184 P acdd72e995154240b079652dd709dcb0: Log is configured to *not* fsync() on all Append() calls
I20250623 01:58:11.837845 16486 tablet_bootstrap.cc:654] T bba9194608a945d99fd47609b3762184 P b45591d1dbb44c5d9215a04523ed6125: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:11.839030 16487 tablet_bootstrap.cc:654] T bba9194608a945d99fd47609b3762184 P 458e6bc4a3114c98b3583d8cad507d39: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:11.839886 16486 log.cc:826] T bba9194608a945d99fd47609b3762184 P b45591d1dbb44c5d9215a04523ed6125: Log is configured to *not* fsync() on all Append() calls
I20250623 01:58:11.840673 16487 log.cc:826] T bba9194608a945d99fd47609b3762184 P 458e6bc4a3114c98b3583d8cad507d39: Log is configured to *not* fsync() on all Append() calls
I20250623 01:58:11.841068 16485 tablet_bootstrap.cc:492] T bba9194608a945d99fd47609b3762184 P acdd72e995154240b079652dd709dcb0: No bootstrap required, opened a new log
I20250623 01:58:11.841586 16485 ts_tablet_manager.cc:1397] T bba9194608a945d99fd47609b3762184 P acdd72e995154240b079652dd709dcb0: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.009s
I20250623 01:58:11.845089 16486 tablet_bootstrap.cc:492] T bba9194608a945d99fd47609b3762184 P b45591d1dbb44c5d9215a04523ed6125: No bootstrap required, opened a new log
I20250623 01:58:11.845644 16486 ts_tablet_manager.cc:1397] T bba9194608a945d99fd47609b3762184 P b45591d1dbb44c5d9215a04523ed6125: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.004s
I20250623 01:58:11.845654 16487 tablet_bootstrap.cc:492] T bba9194608a945d99fd47609b3762184 P 458e6bc4a3114c98b3583d8cad507d39: No bootstrap required, opened a new log
I20250623 01:58:11.846127 16487 ts_tablet_manager.cc:1397] T bba9194608a945d99fd47609b3762184 P 458e6bc4a3114c98b3583d8cad507d39: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.007s
I20250623 01:58:11.863404 16486 raft_consensus.cc:357] T bba9194608a945d99fd47609b3762184 P b45591d1dbb44c5d9215a04523ed6125 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458e6bc4a3114c98b3583d8cad507d39" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 37965 } } peers { permanent_uuid: "acdd72e995154240b079652dd709dcb0" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 43645 } } peers { permanent_uuid: "b45591d1dbb44c5d9215a04523ed6125" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37671 } }
I20250623 01:58:11.864086 16486 raft_consensus.cc:383] T bba9194608a945d99fd47609b3762184 P b45591d1dbb44c5d9215a04523ed6125 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:11.864347 16486 raft_consensus.cc:738] T bba9194608a945d99fd47609b3762184 P b45591d1dbb44c5d9215a04523ed6125 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b45591d1dbb44c5d9215a04523ed6125, State: Initialized, Role: FOLLOWER
I20250623 01:58:11.865067 16486 consensus_queue.cc:260] T bba9194608a945d99fd47609b3762184 P b45591d1dbb44c5d9215a04523ed6125 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458e6bc4a3114c98b3583d8cad507d39" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 37965 } } peers { permanent_uuid: "acdd72e995154240b079652dd709dcb0" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 43645 } } peers { permanent_uuid: "b45591d1dbb44c5d9215a04523ed6125" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37671 } }
I20250623 01:58:11.867223 16485 raft_consensus.cc:357] T bba9194608a945d99fd47609b3762184 P acdd72e995154240b079652dd709dcb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458e6bc4a3114c98b3583d8cad507d39" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 37965 } } peers { permanent_uuid: "acdd72e995154240b079652dd709dcb0" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 43645 } } peers { permanent_uuid: "b45591d1dbb44c5d9215a04523ed6125" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37671 } }
I20250623 01:58:11.868161 16485 raft_consensus.cc:383] T bba9194608a945d99fd47609b3762184 P acdd72e995154240b079652dd709dcb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:11.868498 16485 raft_consensus.cc:738] T bba9194608a945d99fd47609b3762184 P acdd72e995154240b079652dd709dcb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acdd72e995154240b079652dd709dcb0, State: Initialized, Role: FOLLOWER
I20250623 01:58:11.868852 16486 ts_tablet_manager.cc:1428] T bba9194608a945d99fd47609b3762184 P b45591d1dbb44c5d9215a04523ed6125: Time spent starting tablet: real 0.023s user 0.006s sys 0.014s
I20250623 01:58:11.869490 16485 consensus_queue.cc:260] T bba9194608a945d99fd47609b3762184 P acdd72e995154240b079652dd709dcb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458e6bc4a3114c98b3583d8cad507d39" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 37965 } } peers { permanent_uuid: "acdd72e995154240b079652dd709dcb0" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 43645 } } peers { permanent_uuid: "b45591d1dbb44c5d9215a04523ed6125" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37671 } }
I20250623 01:58:11.872498 16466 heartbeater.cc:499] Master 127.14.43.190:35095 was elected leader, sending a full tablet report...
I20250623 01:58:11.872056 16487 raft_consensus.cc:357] T bba9194608a945d99fd47609b3762184 P 458e6bc4a3114c98b3583d8cad507d39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458e6bc4a3114c98b3583d8cad507d39" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 37965 } } peers { permanent_uuid: "acdd72e995154240b079652dd709dcb0" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 43645 } } peers { permanent_uuid: "b45591d1dbb44c5d9215a04523ed6125" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37671 } }
I20250623 01:58:11.872932 16487 raft_consensus.cc:383] T bba9194608a945d99fd47609b3762184 P 458e6bc4a3114c98b3583d8cad507d39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:11.873266 16487 raft_consensus.cc:738] T bba9194608a945d99fd47609b3762184 P 458e6bc4a3114c98b3583d8cad507d39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 458e6bc4a3114c98b3583d8cad507d39, State: Initialized, Role: FOLLOWER
I20250623 01:58:11.874521 16485 ts_tablet_manager.cc:1428] T bba9194608a945d99fd47609b3762184 P acdd72e995154240b079652dd709dcb0: Time spent starting tablet: real 0.033s user 0.031s sys 0.000s
I20250623 01:58:11.874172 16487 consensus_queue.cc:260] T bba9194608a945d99fd47609b3762184 P 458e6bc4a3114c98b3583d8cad507d39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458e6bc4a3114c98b3583d8cad507d39" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 37965 } } peers { permanent_uuid: "acdd72e995154240b079652dd709dcb0" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 43645 } } peers { permanent_uuid: "b45591d1dbb44c5d9215a04523ed6125" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37671 } }
I20250623 01:58:11.878908 16487 ts_tablet_manager.cc:1428] T bba9194608a945d99fd47609b3762184 P 458e6bc4a3114c98b3583d8cad507d39: Time spent starting tablet: real 0.033s user 0.031s sys 0.001s
W20250623 01:58:11.936072 16467 tablet.cc:2378] T bba9194608a945d99fd47609b3762184 P acdd72e995154240b079652dd709dcb0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250623 01:58:11.997046 16334 tablet.cc:2378] T bba9194608a945d99fd47609b3762184 P 458e6bc4a3114c98b3583d8cad507d39: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250623 01:58:12.042304 16201 tablet.cc:2378] T bba9194608a945d99fd47609b3762184 P b45591d1dbb44c5d9215a04523ed6125: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250623 01:58:12.054250 16493 raft_consensus.cc:491] T bba9194608a945d99fd47609b3762184 P 458e6bc4a3114c98b3583d8cad507d39 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:12.054724 16493 raft_consensus.cc:513] T bba9194608a945d99fd47609b3762184 P 458e6bc4a3114c98b3583d8cad507d39 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458e6bc4a3114c98b3583d8cad507d39" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 37965 } } peers { permanent_uuid: "acdd72e995154240b079652dd709dcb0" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 43645 } } peers { permanent_uuid: "b45591d1dbb44c5d9215a04523ed6125" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37671 } }
I20250623 01:58:12.057008 16493 leader_election.cc:290] T bba9194608a945d99fd47609b3762184 P 458e6bc4a3114c98b3583d8cad507d39 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acdd72e995154240b079652dd709dcb0 (127.14.43.131:43645), b45591d1dbb44c5d9215a04523ed6125 (127.14.43.129:37671)
I20250623 01:58:12.066999 16421 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bba9194608a945d99fd47609b3762184" candidate_uuid: "458e6bc4a3114c98b3583d8cad507d39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acdd72e995154240b079652dd709dcb0" is_pre_election: true
I20250623 01:58:12.067724 16421 raft_consensus.cc:2466] T bba9194608a945d99fd47609b3762184 P acdd72e995154240b079652dd709dcb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 458e6bc4a3114c98b3583d8cad507d39 in term 0.
I20250623 01:58:12.068814 16155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bba9194608a945d99fd47609b3762184" candidate_uuid: "458e6bc4a3114c98b3583d8cad507d39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b45591d1dbb44c5d9215a04523ed6125" is_pre_election: true
I20250623 01:58:12.068985 16221 leader_election.cc:304] T bba9194608a945d99fd47609b3762184 P 458e6bc4a3114c98b3583d8cad507d39 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 458e6bc4a3114c98b3583d8cad507d39, acdd72e995154240b079652dd709dcb0; no voters:
I20250623 01:58:12.069651 16155 raft_consensus.cc:2466] T bba9194608a945d99fd47609b3762184 P b45591d1dbb44c5d9215a04523ed6125 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 458e6bc4a3114c98b3583d8cad507d39 in term 0.
I20250623 01:58:12.069795 16493 raft_consensus.cc:2802] T bba9194608a945d99fd47609b3762184 P 458e6bc4a3114c98b3583d8cad507d39 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:12.070087 16493 raft_consensus.cc:491] T bba9194608a945d99fd47609b3762184 P 458e6bc4a3114c98b3583d8cad507d39 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:12.070355 16493 raft_consensus.cc:3058] T bba9194608a945d99fd47609b3762184 P 458e6bc4a3114c98b3583d8cad507d39 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:12.074572 16493 raft_consensus.cc:513] T bba9194608a945d99fd47609b3762184 P 458e6bc4a3114c98b3583d8cad507d39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458e6bc4a3114c98b3583d8cad507d39" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 37965 } } peers { permanent_uuid: "acdd72e995154240b079652dd709dcb0" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 43645 } } peers { permanent_uuid: "b45591d1dbb44c5d9215a04523ed6125" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37671 } }
I20250623 01:58:12.075917 16493 leader_election.cc:290] T bba9194608a945d99fd47609b3762184 P 458e6bc4a3114c98b3583d8cad507d39 [CANDIDATE]: Term 1 election: Requested vote from peers acdd72e995154240b079652dd709dcb0 (127.14.43.131:43645), b45591d1dbb44c5d9215a04523ed6125 (127.14.43.129:37671)
I20250623 01:58:12.076545 16421 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bba9194608a945d99fd47609b3762184" candidate_uuid: "458e6bc4a3114c98b3583d8cad507d39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acdd72e995154240b079652dd709dcb0"
I20250623 01:58:12.076803 16155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bba9194608a945d99fd47609b3762184" candidate_uuid: "458e6bc4a3114c98b3583d8cad507d39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b45591d1dbb44c5d9215a04523ed6125"
I20250623 01:58:12.076936 16421 raft_consensus.cc:3058] T bba9194608a945d99fd47609b3762184 P acdd72e995154240b079652dd709dcb0 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:12.077237 16155 raft_consensus.cc:3058] T bba9194608a945d99fd47609b3762184 P b45591d1dbb44c5d9215a04523ed6125 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:12.081403 16155 raft_consensus.cc:2466] T bba9194608a945d99fd47609b3762184 P b45591d1dbb44c5d9215a04523ed6125 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 458e6bc4a3114c98b3583d8cad507d39 in term 1.
I20250623 01:58:12.081781 16421 raft_consensus.cc:2466] T bba9194608a945d99fd47609b3762184 P acdd72e995154240b079652dd709dcb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 458e6bc4a3114c98b3583d8cad507d39 in term 1.
I20250623 01:58:12.082218 16221 leader_election.cc:304] T bba9194608a945d99fd47609b3762184 P 458e6bc4a3114c98b3583d8cad507d39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 458e6bc4a3114c98b3583d8cad507d39, b45591d1dbb44c5d9215a04523ed6125; no voters:
I20250623 01:58:12.082753 16493 raft_consensus.cc:2802] T bba9194608a945d99fd47609b3762184 P 458e6bc4a3114c98b3583d8cad507d39 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:12.084159 16493 raft_consensus.cc:695] T bba9194608a945d99fd47609b3762184 P 458e6bc4a3114c98b3583d8cad507d39 [term 1 LEADER]: Becoming Leader. State: Replica: 458e6bc4a3114c98b3583d8cad507d39, State: Running, Role: LEADER
I20250623 01:58:12.084975 16493 consensus_queue.cc:237] T bba9194608a945d99fd47609b3762184 P 458e6bc4a3114c98b3583d8cad507d39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458e6bc4a3114c98b3583d8cad507d39" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 37965 } } peers { permanent_uuid: "acdd72e995154240b079652dd709dcb0" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 43645 } } peers { permanent_uuid: "b45591d1dbb44c5d9215a04523ed6125" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37671 } }
I20250623 01:58:12.095144 16012 catalog_manager.cc:5582] T bba9194608a945d99fd47609b3762184 P 458e6bc4a3114c98b3583d8cad507d39 reported cstate change: term changed from 0 to 1, leader changed from <none> to 458e6bc4a3114c98b3583d8cad507d39 (127.14.43.130). New cstate: current_term: 1 leader_uuid: "458e6bc4a3114c98b3583d8cad507d39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458e6bc4a3114c98b3583d8cad507d39" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 37965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acdd72e995154240b079652dd709dcb0" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 43645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b45591d1dbb44c5d9215a04523ed6125" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37671 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:12.175567 14510 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250623 01:58:12.178599 14510 ts_itest-base.cc:246] Waiting for 1 tablets on tserver b45591d1dbb44c5d9215a04523ed6125 to finish bootstrapping
I20250623 01:58:12.190411 14510 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 458e6bc4a3114c98b3583d8cad507d39 to finish bootstrapping
I20250623 01:58:12.200976 14510 ts_itest-base.cc:246] Waiting for 1 tablets on tserver acdd72e995154240b079652dd709dcb0 to finish bootstrapping
I20250623 01:58:12.212440 16012 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53538:
name: "TestAnotherTable"
schema {
columns {
name: "foo"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "bar"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
comment: "comment for bar"
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "foo"
}
}
}
W20250623 01:58:12.213869 16012 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAnotherTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 01:58:12.228899 16135 tablet_service.cc:1468] Processing CreateTablet for tablet d695222058a54b2eb1a3d1bbd0f7d4b9 (DEFAULT_TABLE table=TestAnotherTable [id=18caa89c41ec478195694522e86bd4c6]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250623 01:58:12.229952 16135 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d695222058a54b2eb1a3d1bbd0f7d4b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.230265 16401 tablet_service.cc:1468] Processing CreateTablet for tablet d695222058a54b2eb1a3d1bbd0f7d4b9 (DEFAULT_TABLE table=TestAnotherTable [id=18caa89c41ec478195694522e86bd4c6]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250623 01:58:12.231068 16401 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d695222058a54b2eb1a3d1bbd0f7d4b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.234939 16268 tablet_service.cc:1468] Processing CreateTablet for tablet d695222058a54b2eb1a3d1bbd0f7d4b9 (DEFAULT_TABLE table=TestAnotherTable [id=18caa89c41ec478195694522e86bd4c6]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250623 01:58:12.235936 16268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d695222058a54b2eb1a3d1bbd0f7d4b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:12.239840 16485 tablet_bootstrap.cc:492] T d695222058a54b2eb1a3d1bbd0f7d4b9 P acdd72e995154240b079652dd709dcb0: Bootstrap starting.
I20250623 01:58:12.240600 16486 tablet_bootstrap.cc:492] T d695222058a54b2eb1a3d1bbd0f7d4b9 P b45591d1dbb44c5d9215a04523ed6125: Bootstrap starting.
I20250623 01:58:12.244776 16485 tablet_bootstrap.cc:654] T d695222058a54b2eb1a3d1bbd0f7d4b9 P acdd72e995154240b079652dd709dcb0: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:12.246165 16486 tablet_bootstrap.cc:654] T d695222058a54b2eb1a3d1bbd0f7d4b9 P b45591d1dbb44c5d9215a04523ed6125: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:12.249877 16485 tablet_bootstrap.cc:492] T d695222058a54b2eb1a3d1bbd0f7d4b9 P acdd72e995154240b079652dd709dcb0: No bootstrap required, opened a new log
I20250623 01:58:12.250352 16485 ts_tablet_manager.cc:1397] T d695222058a54b2eb1a3d1bbd0f7d4b9 P acdd72e995154240b079652dd709dcb0: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.003s
I20250623 01:58:12.251194 16487 tablet_bootstrap.cc:492] T d695222058a54b2eb1a3d1bbd0f7d4b9 P 458e6bc4a3114c98b3583d8cad507d39: Bootstrap starting.
I20250623 01:58:12.252843 16485 raft_consensus.cc:357] T d695222058a54b2eb1a3d1bbd0f7d4b9 P acdd72e995154240b079652dd709dcb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45591d1dbb44c5d9215a04523ed6125" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37671 } } peers { permanent_uuid: "458e6bc4a3114c98b3583d8cad507d39" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 37965 } } peers { permanent_uuid: "acdd72e995154240b079652dd709dcb0" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 43645 } }
I20250623 01:58:12.253443 16485 raft_consensus.cc:383] T d695222058a54b2eb1a3d1bbd0f7d4b9 P acdd72e995154240b079652dd709dcb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:12.253644 16486 tablet_bootstrap.cc:492] T d695222058a54b2eb1a3d1bbd0f7d4b9 P b45591d1dbb44c5d9215a04523ed6125: No bootstrap required, opened a new log
I20250623 01:58:12.253656 16485 raft_consensus.cc:738] T d695222058a54b2eb1a3d1bbd0f7d4b9 P acdd72e995154240b079652dd709dcb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acdd72e995154240b079652dd709dcb0, State: Initialized, Role: FOLLOWER
I20250623 01:58:12.254081 16486 ts_tablet_manager.cc:1397] T d695222058a54b2eb1a3d1bbd0f7d4b9 P b45591d1dbb44c5d9215a04523ed6125: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.005s
I20250623 01:58:12.254207 16485 consensus_queue.cc:260] T d695222058a54b2eb1a3d1bbd0f7d4b9 P acdd72e995154240b079652dd709dcb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45591d1dbb44c5d9215a04523ed6125" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37671 } } peers { permanent_uuid: "458e6bc4a3114c98b3583d8cad507d39" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 37965 } } peers { permanent_uuid: "acdd72e995154240b079652dd709dcb0" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 43645 } }
I20250623 01:58:12.256086 16485 ts_tablet_manager.cc:1428] T d695222058a54b2eb1a3d1bbd0f7d4b9 P acdd72e995154240b079652dd709dcb0: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250623 01:58:12.256986 16487 tablet_bootstrap.cc:654] T d695222058a54b2eb1a3d1bbd0f7d4b9 P 458e6bc4a3114c98b3583d8cad507d39: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:12.257925 16486 raft_consensus.cc:357] T d695222058a54b2eb1a3d1bbd0f7d4b9 P b45591d1dbb44c5d9215a04523ed6125 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45591d1dbb44c5d9215a04523ed6125" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37671 } } peers { permanent_uuid: "458e6bc4a3114c98b3583d8cad507d39" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 37965 } } peers { permanent_uuid: "acdd72e995154240b079652dd709dcb0" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 43645 } }
I20250623 01:58:12.258445 16486 raft_consensus.cc:383] T d695222058a54b2eb1a3d1bbd0f7d4b9 P b45591d1dbb44c5d9215a04523ed6125 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:12.258662 16486 raft_consensus.cc:738] T d695222058a54b2eb1a3d1bbd0f7d4b9 P b45591d1dbb44c5d9215a04523ed6125 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b45591d1dbb44c5d9215a04523ed6125, State: Initialized, Role: FOLLOWER
I20250623 01:58:12.259187 16486 consensus_queue.cc:260] T d695222058a54b2eb1a3d1bbd0f7d4b9 P b45591d1dbb44c5d9215a04523ed6125 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45591d1dbb44c5d9215a04523ed6125" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37671 } } peers { permanent_uuid: "458e6bc4a3114c98b3583d8cad507d39" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 37965 } } peers { permanent_uuid: "acdd72e995154240b079652dd709dcb0" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 43645 } }
I20250623 01:58:12.262826 16487 tablet_bootstrap.cc:492] T d695222058a54b2eb1a3d1bbd0f7d4b9 P 458e6bc4a3114c98b3583d8cad507d39: No bootstrap required, opened a new log
I20250623 01:58:12.263286 16487 ts_tablet_manager.cc:1397] T d695222058a54b2eb1a3d1bbd0f7d4b9 P 458e6bc4a3114c98b3583d8cad507d39: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.008s
I20250623 01:58:12.263350 16486 ts_tablet_manager.cc:1428] T d695222058a54b2eb1a3d1bbd0f7d4b9 P b45591d1dbb44c5d9215a04523ed6125: Time spent starting tablet: real 0.009s user 0.003s sys 0.001s
I20250623 01:58:12.265762 16487 raft_consensus.cc:357] T d695222058a54b2eb1a3d1bbd0f7d4b9 P 458e6bc4a3114c98b3583d8cad507d39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45591d1dbb44c5d9215a04523ed6125" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37671 } } peers { permanent_uuid: "458e6bc4a3114c98b3583d8cad507d39" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 37965 } } peers { permanent_uuid: "acdd72e995154240b079652dd709dcb0" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 43645 } }
I20250623 01:58:12.266412 16487 raft_consensus.cc:383] T d695222058a54b2eb1a3d1bbd0f7d4b9 P 458e6bc4a3114c98b3583d8cad507d39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:12.266685 16487 raft_consensus.cc:738] T d695222058a54b2eb1a3d1bbd0f7d4b9 P 458e6bc4a3114c98b3583d8cad507d39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 458e6bc4a3114c98b3583d8cad507d39, State: Initialized, Role: FOLLOWER
I20250623 01:58:12.267233 16487 consensus_queue.cc:260] T d695222058a54b2eb1a3d1bbd0f7d4b9 P 458e6bc4a3114c98b3583d8cad507d39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45591d1dbb44c5d9215a04523ed6125" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37671 } } peers { permanent_uuid: "458e6bc4a3114c98b3583d8cad507d39" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 37965 } } peers { permanent_uuid: "acdd72e995154240b079652dd709dcb0" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 43645 } }
I20250623 01:58:12.269162 16487 ts_tablet_manager.cc:1428] T d695222058a54b2eb1a3d1bbd0f7d4b9 P 458e6bc4a3114c98b3583d8cad507d39: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20250623 01:58:12.382876 16491 raft_consensus.cc:491] T d695222058a54b2eb1a3d1bbd0f7d4b9 P b45591d1dbb44c5d9215a04523ed6125 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:12.383417 16491 raft_consensus.cc:513] T d695222058a54b2eb1a3d1bbd0f7d4b9 P b45591d1dbb44c5d9215a04523ed6125 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45591d1dbb44c5d9215a04523ed6125" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37671 } } peers { permanent_uuid: "458e6bc4a3114c98b3583d8cad507d39" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 37965 } } peers { permanent_uuid: "acdd72e995154240b079652dd709dcb0" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 43645 } }
I20250623 01:58:12.386466 16491 leader_election.cc:290] T d695222058a54b2eb1a3d1bbd0f7d4b9 P b45591d1dbb44c5d9215a04523ed6125 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 458e6bc4a3114c98b3583d8cad507d39 (127.14.43.130:37965), acdd72e995154240b079652dd709dcb0 (127.14.43.131:43645)
I20250623 01:58:12.398466 16288 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d695222058a54b2eb1a3d1bbd0f7d4b9" candidate_uuid: "b45591d1dbb44c5d9215a04523ed6125" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "458e6bc4a3114c98b3583d8cad507d39" is_pre_election: true
I20250623 01:58:12.399231 16288 raft_consensus.cc:2466] T d695222058a54b2eb1a3d1bbd0f7d4b9 P 458e6bc4a3114c98b3583d8cad507d39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b45591d1dbb44c5d9215a04523ed6125 in term 0.
I20250623 01:58:12.400992 16089 leader_election.cc:304] T d695222058a54b2eb1a3d1bbd0f7d4b9 P b45591d1dbb44c5d9215a04523ed6125 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 458e6bc4a3114c98b3583d8cad507d39, b45591d1dbb44c5d9215a04523ed6125; no voters:
I20250623 01:58:12.401753 16491 raft_consensus.cc:2802] T d695222058a54b2eb1a3d1bbd0f7d4b9 P b45591d1dbb44c5d9215a04523ed6125 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:12.402094 16491 raft_consensus.cc:491] T d695222058a54b2eb1a3d1bbd0f7d4b9 P b45591d1dbb44c5d9215a04523ed6125 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:12.402386 16491 raft_consensus.cc:3058] T d695222058a54b2eb1a3d1bbd0f7d4b9 P b45591d1dbb44c5d9215a04523ed6125 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:12.402667 16421 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d695222058a54b2eb1a3d1bbd0f7d4b9" candidate_uuid: "b45591d1dbb44c5d9215a04523ed6125" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acdd72e995154240b079652dd709dcb0" is_pre_election: true
I20250623 01:58:12.403111 16421 raft_consensus.cc:2466] T d695222058a54b2eb1a3d1bbd0f7d4b9 P acdd72e995154240b079652dd709dcb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b45591d1dbb44c5d9215a04523ed6125 in term 0.
I20250623 01:58:12.408634 16491 raft_consensus.cc:513] T d695222058a54b2eb1a3d1bbd0f7d4b9 P b45591d1dbb44c5d9215a04523ed6125 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45591d1dbb44c5d9215a04523ed6125" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37671 } } peers { permanent_uuid: "458e6bc4a3114c98b3583d8cad507d39" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 37965 } } peers { permanent_uuid: "acdd72e995154240b079652dd709dcb0" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 43645 } }
I20250623 01:58:12.410336 16491 leader_election.cc:290] T d695222058a54b2eb1a3d1bbd0f7d4b9 P b45591d1dbb44c5d9215a04523ed6125 [CANDIDATE]: Term 1 election: Requested vote from peers 458e6bc4a3114c98b3583d8cad507d39 (127.14.43.130:37965), acdd72e995154240b079652dd709dcb0 (127.14.43.131:43645)
I20250623 01:58:12.411053 16288 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d695222058a54b2eb1a3d1bbd0f7d4b9" candidate_uuid: "b45591d1dbb44c5d9215a04523ed6125" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "458e6bc4a3114c98b3583d8cad507d39"
I20250623 01:58:12.411254 16421 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d695222058a54b2eb1a3d1bbd0f7d4b9" candidate_uuid: "b45591d1dbb44c5d9215a04523ed6125" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acdd72e995154240b079652dd709dcb0"
I20250623 01:58:12.411476 16288 raft_consensus.cc:3058] T d695222058a54b2eb1a3d1bbd0f7d4b9 P 458e6bc4a3114c98b3583d8cad507d39 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:12.411749 16421 raft_consensus.cc:3058] T d695222058a54b2eb1a3d1bbd0f7d4b9 P acdd72e995154240b079652dd709dcb0 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:12.415771 16288 raft_consensus.cc:2466] T d695222058a54b2eb1a3d1bbd0f7d4b9 P 458e6bc4a3114c98b3583d8cad507d39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b45591d1dbb44c5d9215a04523ed6125 in term 1.
I20250623 01:58:12.416574 16089 leader_election.cc:304] T d695222058a54b2eb1a3d1bbd0f7d4b9 P b45591d1dbb44c5d9215a04523ed6125 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 458e6bc4a3114c98b3583d8cad507d39, b45591d1dbb44c5d9215a04523ed6125; no voters:
I20250623 01:58:12.417232 16491 raft_consensus.cc:2802] T d695222058a54b2eb1a3d1bbd0f7d4b9 P b45591d1dbb44c5d9215a04523ed6125 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:12.417588 16421 raft_consensus.cc:2466] T d695222058a54b2eb1a3d1bbd0f7d4b9 P acdd72e995154240b079652dd709dcb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b45591d1dbb44c5d9215a04523ed6125 in term 1.
I20250623 01:58:12.419050 16491 raft_consensus.cc:695] T d695222058a54b2eb1a3d1bbd0f7d4b9 P b45591d1dbb44c5d9215a04523ed6125 [term 1 LEADER]: Becoming Leader. State: Replica: b45591d1dbb44c5d9215a04523ed6125, State: Running, Role: LEADER
I20250623 01:58:12.419759 16491 consensus_queue.cc:237] T d695222058a54b2eb1a3d1bbd0f7d4b9 P b45591d1dbb44c5d9215a04523ed6125 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45591d1dbb44c5d9215a04523ed6125" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37671 } } peers { permanent_uuid: "458e6bc4a3114c98b3583d8cad507d39" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 37965 } } peers { permanent_uuid: "acdd72e995154240b079652dd709dcb0" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 43645 } }
I20250623 01:58:12.430778 16012 catalog_manager.cc:5582] T d695222058a54b2eb1a3d1bbd0f7d4b9 P b45591d1dbb44c5d9215a04523ed6125 reported cstate change: term changed from 0 to 1, leader changed from <none> to b45591d1dbb44c5d9215a04523ed6125 (127.14.43.129). New cstate: current_term: 1 leader_uuid: "b45591d1dbb44c5d9215a04523ed6125" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b45591d1dbb44c5d9215a04523ed6125" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "458e6bc4a3114c98b3583d8cad507d39" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 37965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acdd72e995154240b079652dd709dcb0" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 43645 } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:12.618350 16493 consensus_queue.cc:1035] T bba9194608a945d99fd47609b3762184 P 458e6bc4a3114c98b3583d8cad507d39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acdd72e995154240b079652dd709dcb0" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 43645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:12.638773 16493 consensus_queue.cc:1035] T bba9194608a945d99fd47609b3762184 P 458e6bc4a3114c98b3583d8cad507d39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b45591d1dbb44c5d9215a04523ed6125" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 37671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 01:58:12.786691 16506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:58:12.787227 16506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:58:12.817688 16506 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250623 01:58:12.886763 16491 consensus_queue.cc:1035] T d695222058a54b2eb1a3d1bbd0f7d4b9 P b45591d1dbb44c5d9215a04523ed6125 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acdd72e995154240b079652dd709dcb0" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 43645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:12.898372 16491 consensus_queue.cc:1035] T d695222058a54b2eb1a3d1bbd0f7d4b9 P b45591d1dbb44c5d9215a04523ed6125 [LEADER]: Connected to new peer: Peer: permanent_uuid: "458e6bc4a3114c98b3583d8cad507d39" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 37965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 01:58:13.480545 16196 debug-util.cc:398] Leaking SignalData structure 0x7b08000ac1e0 after lost signal to thread 16072
W20250623 01:58:13.481721 16196 debug-util.cc:398] Leaking SignalData structure 0x7b08000c81c0 after lost signal to thread 16199
W20250623 01:58:14.054390 16506 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.199s user 0.517s sys 0.676s
W20250623 01:58:14.054774 16506 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.199s user 0.517s sys 0.676s
W20250623 01:58:15.454138 16531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:58:15.454753 16531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:58:15.485235 16531 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250623 01:58:16.650104 16531 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.125s user 0.427s sys 0.696s
W20250623 01:58:16.650410 16531 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.126s user 0.430s sys 0.696s
W20250623 01:58:18.012197 16546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:58:18.012773 16546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:58:18.043408 16546 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250623 01:58:19.253093 16546 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.172s user 0.522s sys 0.647s
W20250623 01:58:19.253407 16546 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.172s user 0.522s sys 0.647s
W20250623 01:58:20.612506 16562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:58:20.613082 16562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:58:20.644726 16562 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250623 01:58:21.806778 16562 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.121s user 0.457s sys 0.663s
W20250623 01:58:21.807055 16562 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.121s user 0.457s sys 0.663s
I20250623 01:58:22.873988 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 16071
I20250623 01:58:22.895733 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 16204
I20250623 01:58:22.915915 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 16337
I20250623 01:58:22.935307 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 15979
2025-06-23T01:58:22Z chronyd exiting
[ OK ] AdminCliTest.TestDescribeTableColumnFlags (18258 ms)
[ RUN ] AdminCliTest.TestAuthzResetCacheNotAuthorized
I20250623 01:58:22.987624 14510 test_util.cc:276] Using random seed: -1946731184
I20250623 01:58:22.991576 14510 ts_itest-base.cc:115] Starting cluster with:
I20250623 01:58:22.991750 14510 ts_itest-base.cc:116] --------------
I20250623 01:58:22.991897 14510 ts_itest-base.cc:117] 3 tablet servers
I20250623 01:58:22.992028 14510 ts_itest-base.cc:118] 3 replicas per TS
I20250623 01:58:22.992167 14510 ts_itest-base.cc:119] --------------
2025-06-23T01:58:23Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-23T01:58:23Z Disabled control of system clock
I20250623 01:58:23.023910 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.43.190:44279
--webserver_interface=127.14.43.190
--webserver_port=0
--builtin_ntp_servers=127.14.43.148:46621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.43.190:44279
--superuser_acl=no-such-user with env {}
W20250623 01:58:23.307219 16584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:58:23.307768 16584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:58:23.308213 16584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:58:23.339032 16584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250623 01:58:23.339315 16584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:58:23.339570 16584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250623 01:58:23.339774 16584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250623 01:58:23.374461 16584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:46621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.43.190:44279
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.43.190:44279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.14.43.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:58:23.375670 16584 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:58:23.377265 16584 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:58:23.394028 16591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:58:23.394074 16590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:58:23.395246 16584 server_base.cc:1048] running on GCE node
W20250623 01:58:23.394038 16593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:58:24.482396 16584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:58:24.484879 16584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:58:24.486229 16584 hybrid_clock.cc:648] HybridClock initialized: now 1750643904486187 us; error 50 us; skew 500 ppm
I20250623 01:58:24.486994 16584 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:58:24.492959 16584 webserver.cc:469] Webserver started at http://127.14.43.190:45629/ using document root <none> and password file <none>
I20250623 01:58:24.493868 16584 fs_manager.cc:362] Metadata directory not provided
I20250623 01:58:24.494091 16584 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:58:24.494530 16584 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:58:24.498811 16584 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "285b24e9d5b1463aac8ade1bd310a76d"
format_stamp: "Formatted at 2025-06-23 01:58:24 on dist-test-slave-9rpg"
I20250623 01:58:24.499842 16584 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "285b24e9d5b1463aac8ade1bd310a76d"
format_stamp: "Formatted at 2025-06-23 01:58:24 on dist-test-slave-9rpg"
I20250623 01:58:24.506642 16584 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250623 01:58:24.511984 16601 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:24.512912 16584 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20250623 01:58:24.513222 16584 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
uuid: "285b24e9d5b1463aac8ade1bd310a76d"
format_stamp: "Formatted at 2025-06-23 01:58:24 on dist-test-slave-9rpg"
I20250623 01:58:24.513537 16584 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:58:24.572654 16584 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:58:24.574136 16584 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:58:24.574551 16584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:58:24.640396 16584 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.190:44279
I20250623 01:58:24.640470 16652 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.190:44279 every 8 connection(s)
I20250623 01:58:24.643013 16584 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250623 01:58:24.644598 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 16584
I20250623 01:58:24.645123 14510 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250623 01:58:24.648689 16653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:24.670113 16653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 285b24e9d5b1463aac8ade1bd310a76d: Bootstrap starting.
I20250623 01:58:24.675681 16653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 285b24e9d5b1463aac8ade1bd310a76d: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:24.677291 16653 log.cc:826] T 00000000000000000000000000000000 P 285b24e9d5b1463aac8ade1bd310a76d: Log is configured to *not* fsync() on all Append() calls
I20250623 01:58:24.681634 16653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 285b24e9d5b1463aac8ade1bd310a76d: No bootstrap required, opened a new log
I20250623 01:58:24.698925 16653 raft_consensus.cc:357] T 00000000000000000000000000000000 P 285b24e9d5b1463aac8ade1bd310a76d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285b24e9d5b1463aac8ade1bd310a76d" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 44279 } }
I20250623 01:58:24.699553 16653 raft_consensus.cc:383] T 00000000000000000000000000000000 P 285b24e9d5b1463aac8ade1bd310a76d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:24.699805 16653 raft_consensus.cc:738] T 00000000000000000000000000000000 P 285b24e9d5b1463aac8ade1bd310a76d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285b24e9d5b1463aac8ade1bd310a76d, State: Initialized, Role: FOLLOWER
I20250623 01:58:24.700472 16653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 285b24e9d5b1463aac8ade1bd310a76d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285b24e9d5b1463aac8ade1bd310a76d" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 44279 } }
I20250623 01:58:24.700978 16653 raft_consensus.cc:397] T 00000000000000000000000000000000 P 285b24e9d5b1463aac8ade1bd310a76d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250623 01:58:24.701232 16653 raft_consensus.cc:491] T 00000000000000000000000000000000 P 285b24e9d5b1463aac8ade1bd310a76d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250623 01:58:24.701508 16653 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 285b24e9d5b1463aac8ade1bd310a76d [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:24.705574 16653 raft_consensus.cc:513] T 00000000000000000000000000000000 P 285b24e9d5b1463aac8ade1bd310a76d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285b24e9d5b1463aac8ade1bd310a76d" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 44279 } }
I20250623 01:58:24.706274 16653 leader_election.cc:304] T 00000000000000000000000000000000 P 285b24e9d5b1463aac8ade1bd310a76d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 285b24e9d5b1463aac8ade1bd310a76d; no voters:
I20250623 01:58:24.707799 16653 leader_election.cc:290] T 00000000000000000000000000000000 P 285b24e9d5b1463aac8ade1bd310a76d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250623 01:58:24.708446 16658 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 285b24e9d5b1463aac8ade1bd310a76d [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:24.710422 16658 raft_consensus.cc:695] T 00000000000000000000000000000000 P 285b24e9d5b1463aac8ade1bd310a76d [term 1 LEADER]: Becoming Leader. State: Replica: 285b24e9d5b1463aac8ade1bd310a76d, State: Running, Role: LEADER
I20250623 01:58:24.711052 16658 consensus_queue.cc:237] T 00000000000000000000000000000000 P 285b24e9d5b1463aac8ade1bd310a76d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285b24e9d5b1463aac8ade1bd310a76d" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 44279 } }
I20250623 01:58:24.711536 16653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 285b24e9d5b1463aac8ade1bd310a76d [sys.catalog]: configured and running, proceeding with master startup.
I20250623 01:58:24.721854 16660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 285b24e9d5b1463aac8ade1bd310a76d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 285b24e9d5b1463aac8ade1bd310a76d. Latest consensus state: current_term: 1 leader_uuid: "285b24e9d5b1463aac8ade1bd310a76d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285b24e9d5b1463aac8ade1bd310a76d" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 44279 } } }
I20250623 01:58:24.722239 16659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 285b24e9d5b1463aac8ade1bd310a76d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "285b24e9d5b1463aac8ade1bd310a76d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285b24e9d5b1463aac8ade1bd310a76d" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 44279 } } }
I20250623 01:58:24.722573 16660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 285b24e9d5b1463aac8ade1bd310a76d [sys.catalog]: This master's current role is: LEADER
I20250623 01:58:24.722774 16659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 285b24e9d5b1463aac8ade1bd310a76d [sys.catalog]: This master's current role is: LEADER
I20250623 01:58:24.725098 16668 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250623 01:58:24.734784 16668 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250623 01:58:24.749867 16668 catalog_manager.cc:1349] Generated new cluster ID: 4a26b484221d44d4908cdee71d8880b4
I20250623 01:58:24.750180 16668 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250623 01:58:24.769614 16668 catalog_manager.cc:1372] Generated new certificate authority record
I20250623 01:58:24.771051 16668 catalog_manager.cc:1506] Loading token signing keys...
I20250623 01:58:24.785981 16668 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 285b24e9d5b1463aac8ade1bd310a76d: Generated new TSK 0
I20250623 01:58:24.786895 16668 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250623 01:58:24.802783 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.43.129:0
--local_ip_for_outbound_sockets=127.14.43.129
--webserver_interface=127.14.43.129
--webserver_port=0
--tserver_master_addrs=127.14.43.190:44279
--builtin_ntp_servers=127.14.43.148:46621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250623 01:58:25.089227 16677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:58:25.089726 16677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:58:25.090250 16677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:58:25.120705 16677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:58:25.121536 16677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.43.129
I20250623 01:58:25.155391 16677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:46621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.43.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.14.43.129
--webserver_port=0
--tserver_master_addrs=127.14.43.190:44279
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.43.129
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:58:25.156646 16677 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:58:25.158262 16677 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:58:25.174149 16686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:58:25.174258 16684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:58:25.175544 16683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:58:26.249895 16685 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250623 01:58:26.249982 16677 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250623 01:58:26.254283 16677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:58:26.256934 16677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:58:26.258396 16677 hybrid_clock.cc:648] HybridClock initialized: now 1750643906258348 us; error 50 us; skew 500 ppm
I20250623 01:58:26.259454 16677 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:58:26.269310 16677 webserver.cc:469] Webserver started at http://127.14.43.129:40919/ using document root <none> and password file <none>
I20250623 01:58:26.270542 16677 fs_manager.cc:362] Metadata directory not provided
I20250623 01:58:26.270856 16677 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:58:26.271441 16677 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:58:26.277776 16677 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "8300706e97c140b8a9f8a0e463bee402"
format_stamp: "Formatted at 2025-06-23 01:58:26 on dist-test-slave-9rpg"
I20250623 01:58:26.279247 16677 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "8300706e97c140b8a9f8a0e463bee402"
format_stamp: "Formatted at 2025-06-23 01:58:26 on dist-test-slave-9rpg"
I20250623 01:58:26.288614 16677 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.001s sys 0.008s
I20250623 01:58:26.296343 16693 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:26.297616 16677 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20250623 01:58:26.298192 16677 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "8300706e97c140b8a9f8a0e463bee402"
format_stamp: "Formatted at 2025-06-23 01:58:26 on dist-test-slave-9rpg"
I20250623 01:58:26.298650 16677 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:58:26.352391 16677 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:58:26.353808 16677 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:58:26.354244 16677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:58:26.356918 16677 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:58:26.361195 16677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250623 01:58:26.361403 16677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:26.361632 16677 ts_tablet_manager.cc:610] Registered 0 tablets
I20250623 01:58:26.361792 16677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:26.504791 16677 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.129:41343
I20250623 01:58:26.504904 16805 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.129:41343 every 8 connection(s)
I20250623 01:58:26.507333 16677 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250623 01:58:26.509361 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 16677
I20250623 01:58:26.509864 14510 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250623 01:58:26.516553 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.43.130:0
--local_ip_for_outbound_sockets=127.14.43.130
--webserver_interface=127.14.43.130
--webserver_port=0
--tserver_master_addrs=127.14.43.190:44279
--builtin_ntp_servers=127.14.43.148:46621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250623 01:58:26.527534 16806 heartbeater.cc:344] Connected to a master server at 127.14.43.190:44279
I20250623 01:58:26.528033 16806 heartbeater.cc:461] Registering TS with master...
I20250623 01:58:26.529253 16806 heartbeater.cc:507] Master 127.14.43.190:44279 requested a full tablet report, sending...
I20250623 01:58:26.531978 16618 ts_manager.cc:194] Registered new tserver with Master: 8300706e97c140b8a9f8a0e463bee402 (127.14.43.129:41343)
I20250623 01:58:26.533787 16618 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.43.129:60295
W20250623 01:58:26.798732 16810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:58:26.799237 16810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:58:26.799738 16810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:58:26.830554 16810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:58:26.831410 16810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.43.130
I20250623 01:58:26.865193 16810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:46621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.43.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.14.43.130
--webserver_port=0
--tserver_master_addrs=127.14.43.190:44279
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.43.130
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:58:26.866453 16810 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:58:26.868088 16810 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:58:26.882679 16816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:58:27.537248 16806 heartbeater.cc:499] Master 127.14.43.190:44279 was elected leader, sending a full tablet report...
W20250623 01:58:26.882926 16817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:58:26.885286 16810 server_base.cc:1048] running on GCE node
W20250623 01:58:26.883519 16819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:58:27.952679 16810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:58:27.954814 16810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:58:27.956177 16810 hybrid_clock.cc:648] HybridClock initialized: now 1750643907956141 us; error 47 us; skew 500 ppm
I20250623 01:58:27.956988 16810 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:58:27.966622 16810 webserver.cc:469] Webserver started at http://127.14.43.130:43587/ using document root <none> and password file <none>
I20250623 01:58:27.967506 16810 fs_manager.cc:362] Metadata directory not provided
I20250623 01:58:27.967707 16810 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:58:27.968142 16810 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:58:27.972498 16810 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "d4b978e8d8fa4718b402ce62ea9a5f43"
format_stamp: "Formatted at 2025-06-23 01:58:27 on dist-test-slave-9rpg"
I20250623 01:58:27.973547 16810 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "d4b978e8d8fa4718b402ce62ea9a5f43"
format_stamp: "Formatted at 2025-06-23 01:58:27 on dist-test-slave-9rpg"
I20250623 01:58:27.980486 16810 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20250623 01:58:27.985976 16826 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:27.987077 16810 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250623 01:58:27.987385 16810 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d4b978e8d8fa4718b402ce62ea9a5f43"
format_stamp: "Formatted at 2025-06-23 01:58:27 on dist-test-slave-9rpg"
I20250623 01:58:27.987733 16810 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:58:28.056710 16810 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:58:28.058271 16810 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:58:28.058676 16810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:58:28.061065 16810 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:58:28.065016 16810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250623 01:58:28.065234 16810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:28.065472 16810 ts_tablet_manager.cc:610] Registered 0 tablets
I20250623 01:58:28.065629 16810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:28.191177 16810 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.130:33153
I20250623 01:58:28.191293 16938 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.130:33153 every 8 connection(s)
I20250623 01:58:28.193575 16810 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250623 01:58:28.200908 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 16810
I20250623 01:58:28.201265 14510 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250623 01:58:28.207002 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.43.131:0
--local_ip_for_outbound_sockets=127.14.43.131
--webserver_interface=127.14.43.131
--webserver_port=0
--tserver_master_addrs=127.14.43.190:44279
--builtin_ntp_servers=127.14.43.148:46621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250623 01:58:28.213960 16939 heartbeater.cc:344] Connected to a master server at 127.14.43.190:44279
I20250623 01:58:28.214397 16939 heartbeater.cc:461] Registering TS with master...
I20250623 01:58:28.215356 16939 heartbeater.cc:507] Master 127.14.43.190:44279 requested a full tablet report, sending...
I20250623 01:58:28.217345 16618 ts_manager.cc:194] Registered new tserver with Master: d4b978e8d8fa4718b402ce62ea9a5f43 (127.14.43.130:33153)
I20250623 01:58:28.218597 16618 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.43.130:58703
W20250623 01:58:28.498874 16943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:58:28.499351 16943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:58:28.499825 16943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:58:28.530445 16943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:58:28.531288 16943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.43.131
I20250623 01:58:28.565994 16943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:46621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.43.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.14.43.131
--webserver_port=0
--tserver_master_addrs=127.14.43.190:44279
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.43.131
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:58:28.567291 16943 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:58:28.568886 16943 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:58:28.584108 16949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:58:29.221455 16939 heartbeater.cc:499] Master 127.14.43.190:44279 was elected leader, sending a full tablet report...
W20250623 01:58:28.584710 16952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:58:28.585583 16943 server_base.cc:1048] running on GCE node
W20250623 01:58:28.585644 16950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:58:29.649580 16943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:58:29.651859 16943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:58:29.653225 16943 hybrid_clock.cc:648] HybridClock initialized: now 1750643909653180 us; error 58 us; skew 500 ppm
I20250623 01:58:29.654059 16943 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:58:29.659874 16943 webserver.cc:469] Webserver started at http://127.14.43.131:40079/ using document root <none> and password file <none>
I20250623 01:58:29.660804 16943 fs_manager.cc:362] Metadata directory not provided
I20250623 01:58:29.661019 16943 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:58:29.661446 16943 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:58:29.665803 16943 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "fae3c3bd6e3f4fc4a20a298bf0d2a513"
format_stamp: "Formatted at 2025-06-23 01:58:29 on dist-test-slave-9rpg"
I20250623 01:58:29.666900 16943 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "fae3c3bd6e3f4fc4a20a298bf0d2a513"
format_stamp: "Formatted at 2025-06-23 01:58:29 on dist-test-slave-9rpg"
I20250623 01:58:29.673708 16943 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20250623 01:58:29.679032 16959 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:29.680048 16943 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250623 01:58:29.680353 16943 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "fae3c3bd6e3f4fc4a20a298bf0d2a513"
format_stamp: "Formatted at 2025-06-23 01:58:29 on dist-test-slave-9rpg"
I20250623 01:58:29.680708 16943 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:58:29.732131 16943 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:58:29.733538 16943 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:58:29.733955 16943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:58:29.736430 16943 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:58:29.740374 16943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250623 01:58:29.740581 16943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:29.740839 16943 ts_tablet_manager.cc:610] Registered 0 tablets
I20250623 01:58:29.740996 16943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:29.866402 16943 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.131:42499
I20250623 01:58:29.866510 17071 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.131:42499 every 8 connection(s)
I20250623 01:58:29.868831 16943 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250623 01:58:29.872737 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 16943
I20250623 01:58:29.873195 14510 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250623 01:58:29.888691 17072 heartbeater.cc:344] Connected to a master server at 127.14.43.190:44279
I20250623 01:58:29.889099 17072 heartbeater.cc:461] Registering TS with master...
I20250623 01:58:29.890053 17072 heartbeater.cc:507] Master 127.14.43.190:44279 requested a full tablet report, sending...
I20250623 01:58:29.892122 16618 ts_manager.cc:194] Registered new tserver with Master: fae3c3bd6e3f4fc4a20a298bf0d2a513 (127.14.43.131:42499)
I20250623 01:58:29.893255 14510 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250623 01:58:29.893368 16618 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.43.131:51621
I20250623 01:58:29.924670 16618 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60924:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250623 01:58:29.942415 16618 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 01:58:29.991715 16874 tablet_service.cc:1468] Processing CreateTablet for tablet dcb8f495db684b03b67927d7be9f3d48 (DEFAULT_TABLE table=TestTable [id=61c5e99ef70a4a00992ef09322d0b704]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 01:58:29.993768 16874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcb8f495db684b03b67927d7be9f3d48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:29.994238 16741 tablet_service.cc:1468] Processing CreateTablet for tablet dcb8f495db684b03b67927d7be9f3d48 (DEFAULT_TABLE table=TestTable [id=61c5e99ef70a4a00992ef09322d0b704]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 01:58:29.995703 16741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcb8f495db684b03b67927d7be9f3d48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:29.996091 17007 tablet_service.cc:1468] Processing CreateTablet for tablet dcb8f495db684b03b67927d7be9f3d48 (DEFAULT_TABLE table=TestTable [id=61c5e99ef70a4a00992ef09322d0b704]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 01:58:29.997778 17007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcb8f495db684b03b67927d7be9f3d48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:30.017140 17091 tablet_bootstrap.cc:492] T dcb8f495db684b03b67927d7be9f3d48 P 8300706e97c140b8a9f8a0e463bee402: Bootstrap starting.
I20250623 01:58:30.022927 17091 tablet_bootstrap.cc:654] T dcb8f495db684b03b67927d7be9f3d48 P 8300706e97c140b8a9f8a0e463bee402: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:30.023370 17092 tablet_bootstrap.cc:492] T dcb8f495db684b03b67927d7be9f3d48 P d4b978e8d8fa4718b402ce62ea9a5f43: Bootstrap starting.
I20250623 01:58:30.025663 17093 tablet_bootstrap.cc:492] T dcb8f495db684b03b67927d7be9f3d48 P fae3c3bd6e3f4fc4a20a298bf0d2a513: Bootstrap starting.
I20250623 01:58:30.025941 17091 log.cc:826] T dcb8f495db684b03b67927d7be9f3d48 P 8300706e97c140b8a9f8a0e463bee402: Log is configured to *not* fsync() on all Append() calls
I20250623 01:58:30.030550 17092 tablet_bootstrap.cc:654] T dcb8f495db684b03b67927d7be9f3d48 P d4b978e8d8fa4718b402ce62ea9a5f43: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:30.032212 17092 log.cc:826] T dcb8f495db684b03b67927d7be9f3d48 P d4b978e8d8fa4718b402ce62ea9a5f43: Log is configured to *not* fsync() on all Append() calls
I20250623 01:58:30.032333 17091 tablet_bootstrap.cc:492] T dcb8f495db684b03b67927d7be9f3d48 P 8300706e97c140b8a9f8a0e463bee402: No bootstrap required, opened a new log
I20250623 01:58:30.032409 17093 tablet_bootstrap.cc:654] T dcb8f495db684b03b67927d7be9f3d48 P fae3c3bd6e3f4fc4a20a298bf0d2a513: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:30.032840 17091 ts_tablet_manager.cc:1397] T dcb8f495db684b03b67927d7be9f3d48 P 8300706e97c140b8a9f8a0e463bee402: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.000s
I20250623 01:58:30.034615 17093 log.cc:826] T dcb8f495db684b03b67927d7be9f3d48 P fae3c3bd6e3f4fc4a20a298bf0d2a513: Log is configured to *not* fsync() on all Append() calls
I20250623 01:58:30.037206 17092 tablet_bootstrap.cc:492] T dcb8f495db684b03b67927d7be9f3d48 P d4b978e8d8fa4718b402ce62ea9a5f43: No bootstrap required, opened a new log
I20250623 01:58:30.037696 17092 ts_tablet_manager.cc:1397] T dcb8f495db684b03b67927d7be9f3d48 P d4b978e8d8fa4718b402ce62ea9a5f43: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20250623 01:58:30.039912 17093 tablet_bootstrap.cc:492] T dcb8f495db684b03b67927d7be9f3d48 P fae3c3bd6e3f4fc4a20a298bf0d2a513: No bootstrap required, opened a new log
I20250623 01:58:30.040416 17093 ts_tablet_manager.cc:1397] T dcb8f495db684b03b67927d7be9f3d48 P fae3c3bd6e3f4fc4a20a298bf0d2a513: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.004s
I20250623 01:58:30.054971 17092 raft_consensus.cc:357] T dcb8f495db684b03b67927d7be9f3d48 P d4b978e8d8fa4718b402ce62ea9a5f43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b978e8d8fa4718b402ce62ea9a5f43" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 33153 } } peers { permanent_uuid: "8300706e97c140b8a9f8a0e463bee402" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 41343 } } peers { permanent_uuid: "fae3c3bd6e3f4fc4a20a298bf0d2a513" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 42499 } }
I20250623 01:58:30.055645 17092 raft_consensus.cc:383] T dcb8f495db684b03b67927d7be9f3d48 P d4b978e8d8fa4718b402ce62ea9a5f43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:30.055974 17092 raft_consensus.cc:738] T dcb8f495db684b03b67927d7be9f3d48 P d4b978e8d8fa4718b402ce62ea9a5f43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4b978e8d8fa4718b402ce62ea9a5f43, State: Initialized, Role: FOLLOWER
I20250623 01:58:30.056674 17092 consensus_queue.cc:260] T dcb8f495db684b03b67927d7be9f3d48 P d4b978e8d8fa4718b402ce62ea9a5f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b978e8d8fa4718b402ce62ea9a5f43" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 33153 } } peers { permanent_uuid: "8300706e97c140b8a9f8a0e463bee402" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 41343 } } peers { permanent_uuid: "fae3c3bd6e3f4fc4a20a298bf0d2a513" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 42499 } }
I20250623 01:58:30.058411 17091 raft_consensus.cc:357] T dcb8f495db684b03b67927d7be9f3d48 P 8300706e97c140b8a9f8a0e463bee402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b978e8d8fa4718b402ce62ea9a5f43" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 33153 } } peers { permanent_uuid: "8300706e97c140b8a9f8a0e463bee402" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 41343 } } peers { permanent_uuid: "fae3c3bd6e3f4fc4a20a298bf0d2a513" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 42499 } }
I20250623 01:58:30.059321 17091 raft_consensus.cc:383] T dcb8f495db684b03b67927d7be9f3d48 P 8300706e97c140b8a9f8a0e463bee402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:30.059628 17091 raft_consensus.cc:738] T dcb8f495db684b03b67927d7be9f3d48 P 8300706e97c140b8a9f8a0e463bee402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8300706e97c140b8a9f8a0e463bee402, State: Initialized, Role: FOLLOWER
I20250623 01:58:30.060524 17091 consensus_queue.cc:260] T dcb8f495db684b03b67927d7be9f3d48 P 8300706e97c140b8a9f8a0e463bee402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b978e8d8fa4718b402ce62ea9a5f43" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 33153 } } peers { permanent_uuid: "8300706e97c140b8a9f8a0e463bee402" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 41343 } } peers { permanent_uuid: "fae3c3bd6e3f4fc4a20a298bf0d2a513" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 42499 } }
I20250623 01:58:30.062614 17092 ts_tablet_manager.cc:1428] T dcb8f495db684b03b67927d7be9f3d48 P d4b978e8d8fa4718b402ce62ea9a5f43: Time spent starting tablet: real 0.025s user 0.013s sys 0.013s
I20250623 01:58:30.065208 17091 ts_tablet_manager.cc:1428] T dcb8f495db684b03b67927d7be9f3d48 P 8300706e97c140b8a9f8a0e463bee402: Time spent starting tablet: real 0.032s user 0.031s sys 0.002s
I20250623 01:58:30.067395 17093 raft_consensus.cc:357] T dcb8f495db684b03b67927d7be9f3d48 P fae3c3bd6e3f4fc4a20a298bf0d2a513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b978e8d8fa4718b402ce62ea9a5f43" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 33153 } } peers { permanent_uuid: "8300706e97c140b8a9f8a0e463bee402" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 41343 } } peers { permanent_uuid: "fae3c3bd6e3f4fc4a20a298bf0d2a513" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 42499 } }
I20250623 01:58:30.068231 17093 raft_consensus.cc:383] T dcb8f495db684b03b67927d7be9f3d48 P fae3c3bd6e3f4fc4a20a298bf0d2a513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:30.068532 17093 raft_consensus.cc:738] T dcb8f495db684b03b67927d7be9f3d48 P fae3c3bd6e3f4fc4a20a298bf0d2a513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fae3c3bd6e3f4fc4a20a298bf0d2a513, State: Initialized, Role: FOLLOWER
I20250623 01:58:30.069370 17093 consensus_queue.cc:260] T dcb8f495db684b03b67927d7be9f3d48 P fae3c3bd6e3f4fc4a20a298bf0d2a513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b978e8d8fa4718b402ce62ea9a5f43" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 33153 } } peers { permanent_uuid: "8300706e97c140b8a9f8a0e463bee402" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 41343 } } peers { permanent_uuid: "fae3c3bd6e3f4fc4a20a298bf0d2a513" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 42499 } }
I20250623 01:58:30.073004 17093 ts_tablet_manager.cc:1428] T dcb8f495db684b03b67927d7be9f3d48 P fae3c3bd6e3f4fc4a20a298bf0d2a513: Time spent starting tablet: real 0.032s user 0.029s sys 0.000s
I20250623 01:58:30.074419 17072 heartbeater.cc:499] Master 127.14.43.190:44279 was elected leader, sending a full tablet report...
W20250623 01:58:30.122968 17073 tablet.cc:2378] T dcb8f495db684b03b67927d7be9f3d48 P fae3c3bd6e3f4fc4a20a298bf0d2a513: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250623 01:58:30.188117 17098 raft_consensus.cc:491] T dcb8f495db684b03b67927d7be9f3d48 P 8300706e97c140b8a9f8a0e463bee402 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:58:30.188562 17098 raft_consensus.cc:513] T dcb8f495db684b03b67927d7be9f3d48 P 8300706e97c140b8a9f8a0e463bee402 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b978e8d8fa4718b402ce62ea9a5f43" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 33153 } } peers { permanent_uuid: "8300706e97c140b8a9f8a0e463bee402" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 41343 } } peers { permanent_uuid: "fae3c3bd6e3f4fc4a20a298bf0d2a513" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 42499 } }
I20250623 01:58:30.191141 17098 leader_election.cc:290] T dcb8f495db684b03b67927d7be9f3d48 P 8300706e97c140b8a9f8a0e463bee402 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4b978e8d8fa4718b402ce62ea9a5f43 (127.14.43.130:33153), fae3c3bd6e3f4fc4a20a298bf0d2a513 (127.14.43.131:42499)
W20250623 01:58:30.199385 16940 tablet.cc:2378] T dcb8f495db684b03b67927d7be9f3d48 P d4b978e8d8fa4718b402ce62ea9a5f43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250623 01:58:30.203895 16894 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dcb8f495db684b03b67927d7be9f3d48" candidate_uuid: "8300706e97c140b8a9f8a0e463bee402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4b978e8d8fa4718b402ce62ea9a5f43" is_pre_election: true
I20250623 01:58:30.204166 17027 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dcb8f495db684b03b67927d7be9f3d48" candidate_uuid: "8300706e97c140b8a9f8a0e463bee402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fae3c3bd6e3f4fc4a20a298bf0d2a513" is_pre_election: true
I20250623 01:58:30.204573 16894 raft_consensus.cc:2466] T dcb8f495db684b03b67927d7be9f3d48 P d4b978e8d8fa4718b402ce62ea9a5f43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8300706e97c140b8a9f8a0e463bee402 in term 0.
I20250623 01:58:30.204955 17027 raft_consensus.cc:2466] T dcb8f495db684b03b67927d7be9f3d48 P fae3c3bd6e3f4fc4a20a298bf0d2a513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8300706e97c140b8a9f8a0e463bee402 in term 0.
I20250623 01:58:30.205677 16695 leader_election.cc:304] T dcb8f495db684b03b67927d7be9f3d48 P 8300706e97c140b8a9f8a0e463bee402 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8300706e97c140b8a9f8a0e463bee402, d4b978e8d8fa4718b402ce62ea9a5f43; no voters:
I20250623 01:58:30.206426 17098 raft_consensus.cc:2802] T dcb8f495db684b03b67927d7be9f3d48 P 8300706e97c140b8a9f8a0e463bee402 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:58:30.206758 17098 raft_consensus.cc:491] T dcb8f495db684b03b67927d7be9f3d48 P 8300706e97c140b8a9f8a0e463bee402 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:58:30.207003 17098 raft_consensus.cc:3058] T dcb8f495db684b03b67927d7be9f3d48 P 8300706e97c140b8a9f8a0e463bee402 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:30.211282 17098 raft_consensus.cc:513] T dcb8f495db684b03b67927d7be9f3d48 P 8300706e97c140b8a9f8a0e463bee402 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b978e8d8fa4718b402ce62ea9a5f43" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 33153 } } peers { permanent_uuid: "8300706e97c140b8a9f8a0e463bee402" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 41343 } } peers { permanent_uuid: "fae3c3bd6e3f4fc4a20a298bf0d2a513" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 42499 } }
I20250623 01:58:30.212631 17098 leader_election.cc:290] T dcb8f495db684b03b67927d7be9f3d48 P 8300706e97c140b8a9f8a0e463bee402 [CANDIDATE]: Term 1 election: Requested vote from peers d4b978e8d8fa4718b402ce62ea9a5f43 (127.14.43.130:33153), fae3c3bd6e3f4fc4a20a298bf0d2a513 (127.14.43.131:42499)
I20250623 01:58:30.213469 16894 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dcb8f495db684b03b67927d7be9f3d48" candidate_uuid: "8300706e97c140b8a9f8a0e463bee402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4b978e8d8fa4718b402ce62ea9a5f43"
I20250623 01:58:30.213630 17027 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dcb8f495db684b03b67927d7be9f3d48" candidate_uuid: "8300706e97c140b8a9f8a0e463bee402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fae3c3bd6e3f4fc4a20a298bf0d2a513"
I20250623 01:58:30.213866 16894 raft_consensus.cc:3058] T dcb8f495db684b03b67927d7be9f3d48 P d4b978e8d8fa4718b402ce62ea9a5f43 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:30.214167 17027 raft_consensus.cc:3058] T dcb8f495db684b03b67927d7be9f3d48 P fae3c3bd6e3f4fc4a20a298bf0d2a513 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:30.218653 16894 raft_consensus.cc:2466] T dcb8f495db684b03b67927d7be9f3d48 P d4b978e8d8fa4718b402ce62ea9a5f43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8300706e97c140b8a9f8a0e463bee402 in term 1.
I20250623 01:58:30.219573 16695 leader_election.cc:304] T dcb8f495db684b03b67927d7be9f3d48 P 8300706e97c140b8a9f8a0e463bee402 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8300706e97c140b8a9f8a0e463bee402, d4b978e8d8fa4718b402ce62ea9a5f43; no voters:
I20250623 01:58:30.220150 17098 raft_consensus.cc:2802] T dcb8f495db684b03b67927d7be9f3d48 P 8300706e97c140b8a9f8a0e463bee402 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:30.222503 17027 raft_consensus.cc:2466] T dcb8f495db684b03b67927d7be9f3d48 P fae3c3bd6e3f4fc4a20a298bf0d2a513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8300706e97c140b8a9f8a0e463bee402 in term 1.
I20250623 01:58:30.222584 17098 raft_consensus.cc:695] T dcb8f495db684b03b67927d7be9f3d48 P 8300706e97c140b8a9f8a0e463bee402 [term 1 LEADER]: Becoming Leader. State: Replica: 8300706e97c140b8a9f8a0e463bee402, State: Running, Role: LEADER
I20250623 01:58:30.223501 17098 consensus_queue.cc:237] T dcb8f495db684b03b67927d7be9f3d48 P 8300706e97c140b8a9f8a0e463bee402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b978e8d8fa4718b402ce62ea9a5f43" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 33153 } } peers { permanent_uuid: "8300706e97c140b8a9f8a0e463bee402" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 41343 } } peers { permanent_uuid: "fae3c3bd6e3f4fc4a20a298bf0d2a513" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 42499 } }
I20250623 01:58:30.233968 16618 catalog_manager.cc:5582] T dcb8f495db684b03b67927d7be9f3d48 P 8300706e97c140b8a9f8a0e463bee402 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8300706e97c140b8a9f8a0e463bee402 (127.14.43.129). New cstate: current_term: 1 leader_uuid: "8300706e97c140b8a9f8a0e463bee402" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b978e8d8fa4718b402ce62ea9a5f43" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 33153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8300706e97c140b8a9f8a0e463bee402" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 41343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fae3c3bd6e3f4fc4a20a298bf0d2a513" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 42499 } health_report { overall_health: UNKNOWN } } }
W20250623 01:58:30.265704 16807 tablet.cc:2378] T dcb8f495db684b03b67927d7be9f3d48 P 8300706e97c140b8a9f8a0e463bee402: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250623 01:58:30.287128 14510 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250623 01:58:30.290403 14510 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 8300706e97c140b8a9f8a0e463bee402 to finish bootstrapping
I20250623 01:58:30.302294 14510 ts_itest-base.cc:246] Waiting for 1 tablets on tserver d4b978e8d8fa4718b402ce62ea9a5f43 to finish bootstrapping
I20250623 01:58:30.312002 14510 ts_itest-base.cc:246] Waiting for 1 tablets on tserver fae3c3bd6e3f4fc4a20a298bf0d2a513 to finish bootstrapping
I20250623 01:58:30.611373 17098 consensus_queue.cc:1035] T dcb8f495db684b03b67927d7be9f3d48 P 8300706e97c140b8a9f8a0e463bee402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fae3c3bd6e3f4fc4a20a298bf0d2a513" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 42499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:58:30.631795 17098 consensus_queue.cc:1035] T dcb8f495db684b03b67927d7be9f3d48 P 8300706e97c140b8a9f8a0e463bee402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4b978e8d8fa4718b402ce62ea9a5f43" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 33153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 01:58:31.844343 16618 server_base.cc:1130] Unauthorized access attempt to method kudu.master.MasterService.RefreshAuthzCache from {username='slave'} at 127.0.0.1:60940
I20250623 01:58:32.869374 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 16677
I20250623 01:58:32.889343 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 16810
I20250623 01:58:32.910688 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 16943
I20250623 01:58:32.930130 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 16584
2025-06-23T01:58:32Z chronyd exiting
[ OK ] AdminCliTest.TestAuthzResetCacheNotAuthorized (9990 ms)
[ RUN ] AdminCliTest.TestRebuildTables
I20250623 01:58:32.978268 14510 test_util.cc:276] Using random seed: -1936740543
I20250623 01:58:32.982331 14510 ts_itest-base.cc:115] Starting cluster with:
I20250623 01:58:32.982549 14510 ts_itest-base.cc:116] --------------
I20250623 01:58:32.982712 14510 ts_itest-base.cc:117] 3 tablet servers
I20250623 01:58:32.982887 14510 ts_itest-base.cc:118] 3 replicas per TS
I20250623 01:58:32.983067 14510 ts_itest-base.cc:119] --------------
2025-06-23T01:58:32Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-23T01:58:33Z Disabled control of system clock
I20250623 01:58:33.015132 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.43.190:33475
--webserver_interface=127.14.43.190
--webserver_port=0
--builtin_ntp_servers=127.14.43.148:43657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.43.190:33475 with env {}
W20250623 01:58:33.295603 17140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:58:33.296175 17140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:58:33.296612 17140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:58:33.326975 17140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250623 01:58:33.327301 17140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:58:33.327549 17140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250623 01:58:33.327790 17140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250623 01:58:33.362046 17140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:43657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.43.190:33475
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.43.190:33475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.14.43.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:58:33.363389 17140 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:58:33.364948 17140 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:58:33.378975 17147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:58:33.379022 17149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:58:33.379446 17146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:58:34.467773 17148 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250623 01:58:34.467967 17140 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250623 01:58:34.471812 17140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:58:34.474439 17140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:58:34.475800 17140 hybrid_clock.cc:648] HybridClock initialized: now 1750643914475769 us; error 43 us; skew 500 ppm
I20250623 01:58:34.476589 17140 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:58:34.487450 17140 webserver.cc:469] Webserver started at http://127.14.43.190:40887/ using document root <none> and password file <none>
I20250623 01:58:34.488396 17140 fs_manager.cc:362] Metadata directory not provided
I20250623 01:58:34.488626 17140 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:58:34.489100 17140 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:58:34.493552 17140 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "99a1ba904ecf487b894f3609083cce41"
format_stamp: "Formatted at 2025-06-23 01:58:34 on dist-test-slave-9rpg"
I20250623 01:58:34.494653 17140 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "99a1ba904ecf487b894f3609083cce41"
format_stamp: "Formatted at 2025-06-23 01:58:34 on dist-test-slave-9rpg"
I20250623 01:58:34.501642 17140 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250623 01:58:34.507145 17157 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:34.508138 17140 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250623 01:58:34.508461 17140 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
uuid: "99a1ba904ecf487b894f3609083cce41"
format_stamp: "Formatted at 2025-06-23 01:58:34 on dist-test-slave-9rpg"
I20250623 01:58:34.508791 17140 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:58:34.565809 17140 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:58:34.567260 17140 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:58:34.567683 17140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:58:34.634804 17140 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.190:33475
I20250623 01:58:34.634882 17208 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.190:33475 every 8 connection(s)
I20250623 01:58:34.637403 17140 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250623 01:58:34.642395 17209 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:34.646255 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 17140
I20250623 01:58:34.646745 14510 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250623 01:58:34.664136 17209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41: Bootstrap starting.
I20250623 01:58:34.670248 17209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:34.671900 17209 log.cc:826] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41: Log is configured to *not* fsync() on all Append() calls
I20250623 01:58:34.676165 17209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41: No bootstrap required, opened a new log
I20250623 01:58:34.693086 17209 raft_consensus.cc:357] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a1ba904ecf487b894f3609083cce41" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 33475 } }
I20250623 01:58:34.693737 17209 raft_consensus.cc:383] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:34.693933 17209 raft_consensus.cc:738] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a1ba904ecf487b894f3609083cce41, State: Initialized, Role: FOLLOWER
I20250623 01:58:34.694586 17209 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a1ba904ecf487b894f3609083cce41" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 33475 } }
I20250623 01:58:34.695063 17209 raft_consensus.cc:397] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250623 01:58:34.695292 17209 raft_consensus.cc:491] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250623 01:58:34.695536 17209 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:34.699595 17209 raft_consensus.cc:513] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a1ba904ecf487b894f3609083cce41" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 33475 } }
I20250623 01:58:34.700274 17209 leader_election.cc:304] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99a1ba904ecf487b894f3609083cce41; no voters:
I20250623 01:58:34.701826 17209 leader_election.cc:290] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250623 01:58:34.702471 17214 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:34.704365 17214 raft_consensus.cc:695] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [term 1 LEADER]: Becoming Leader. State: Replica: 99a1ba904ecf487b894f3609083cce41, State: Running, Role: LEADER
I20250623 01:58:34.705088 17214 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a1ba904ecf487b894f3609083cce41" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 33475 } }
I20250623 01:58:34.706105 17209 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [sys.catalog]: configured and running, proceeding with master startup.
I20250623 01:58:34.714735 17216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99a1ba904ecf487b894f3609083cce41. Latest consensus state: current_term: 1 leader_uuid: "99a1ba904ecf487b894f3609083cce41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a1ba904ecf487b894f3609083cce41" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 33475 } } }
I20250623 01:58:34.714668 17215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99a1ba904ecf487b894f3609083cce41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a1ba904ecf487b894f3609083cce41" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 33475 } } }
I20250623 01:58:34.715394 17216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [sys.catalog]: This master's current role is: LEADER
I20250623 01:58:34.715476 17215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [sys.catalog]: This master's current role is: LEADER
I20250623 01:58:34.719233 17222 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250623 01:58:34.731302 17222 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250623 01:58:34.746670 17222 catalog_manager.cc:1349] Generated new cluster ID: 820e7a8f9f4b48d3a79721cb8afe844e
I20250623 01:58:34.746963 17222 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250623 01:58:34.765559 17222 catalog_manager.cc:1372] Generated new certificate authority record
I20250623 01:58:34.767117 17222 catalog_manager.cc:1506] Loading token signing keys...
I20250623 01:58:34.779091 17222 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41: Generated new TSK 0
I20250623 01:58:34.780251 17222 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250623 01:58:34.793509 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.43.129:0
--local_ip_for_outbound_sockets=127.14.43.129
--webserver_interface=127.14.43.129
--webserver_port=0
--tserver_master_addrs=127.14.43.190:33475
--builtin_ntp_servers=127.14.43.148:43657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250623 01:58:35.082873 17233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:58:35.083390 17233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:58:35.083909 17233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:58:35.114897 17233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:58:35.115725 17233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.43.129
I20250623 01:58:35.149792 17233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:43657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.43.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.14.43.129
--webserver_port=0
--tserver_master_addrs=127.14.43.190:33475
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.43.129
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:58:35.151083 17233 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:58:35.152648 17233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:58:35.169341 17240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:58:35.171736 17242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:58:35.169821 17239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:58:36.309127 17241 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1138 milliseconds
I20250623 01:58:36.309221 17233 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250623 01:58:36.310413 17233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:58:36.312889 17233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:58:36.314348 17233 hybrid_clock.cc:648] HybridClock initialized: now 1750643916314250 us; error 81 us; skew 500 ppm
I20250623 01:58:36.315635 17233 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:58:36.326104 17233 webserver.cc:469] Webserver started at http://127.14.43.129:37409/ using document root <none> and password file <none>
I20250623 01:58:36.328030 17233 fs_manager.cc:362] Metadata directory not provided
I20250623 01:58:36.328362 17233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:58:36.328924 17233 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:58:36.334271 17233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "419f59c561fa4aa8ab1a5a311af7a9ed"
format_stamp: "Formatted at 2025-06-23 01:58:36 on dist-test-slave-9rpg"
I20250623 01:58:36.335623 17233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "419f59c561fa4aa8ab1a5a311af7a9ed"
format_stamp: "Formatted at 2025-06-23 01:58:36 on dist-test-slave-9rpg"
I20250623 01:58:36.343988 17233 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20250623 01:58:36.350454 17249 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:36.351678 17233 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250623 01:58:36.352043 17233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "419f59c561fa4aa8ab1a5a311af7a9ed"
format_stamp: "Formatted at 2025-06-23 01:58:36 on dist-test-slave-9rpg"
I20250623 01:58:36.352505 17233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:58:36.404420 17233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:58:36.405833 17233 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:58:36.406252 17233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:58:36.409085 17233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:58:36.413093 17233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250623 01:58:36.413285 17233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:36.413519 17233 ts_tablet_manager.cc:610] Registered 0 tablets
I20250623 01:58:36.413671 17233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:36.567714 17233 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.129:44313
I20250623 01:58:36.567880 17361 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.129:44313 every 8 connection(s)
I20250623 01:58:36.570420 17233 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250623 01:58:36.578375 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 17233
I20250623 01:58:36.578831 14510 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250623 01:58:36.585549 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.43.130:0
--local_ip_for_outbound_sockets=127.14.43.130
--webserver_interface=127.14.43.130
--webserver_port=0
--tserver_master_addrs=127.14.43.190:33475
--builtin_ntp_servers=127.14.43.148:43657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250623 01:58:36.594563 17362 heartbeater.cc:344] Connected to a master server at 127.14.43.190:33475
I20250623 01:58:36.594970 17362 heartbeater.cc:461] Registering TS with master...
I20250623 01:58:36.595947 17362 heartbeater.cc:507] Master 127.14.43.190:33475 requested a full tablet report, sending...
I20250623 01:58:36.598284 17174 ts_manager.cc:194] Registered new tserver with Master: 419f59c561fa4aa8ab1a5a311af7a9ed (127.14.43.129:44313)
I20250623 01:58:36.600124 17174 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.43.129:49183
W20250623 01:58:36.893592 17366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:58:36.894093 17366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:58:36.894537 17366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:58:36.924785 17366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:58:36.925591 17366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.43.130
I20250623 01:58:36.959513 17366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:43657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.43.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.14.43.130
--webserver_port=0
--tserver_master_addrs=127.14.43.190:33475
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.43.130
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:58:36.960728 17366 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:58:36.962363 17366 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:58:36.977538 17373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:58:37.603281 17362 heartbeater.cc:499] Master 127.14.43.190:33475 was elected leader, sending a full tablet report...
W20250623 01:58:36.977584 17375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:58:36.979523 17366 server_base.cc:1048] running on GCE node
W20250623 01:58:36.978754 17372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:58:38.052683 17366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:58:38.055250 17366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:58:38.056679 17366 hybrid_clock.cc:648] HybridClock initialized: now 1750643918056656 us; error 43 us; skew 500 ppm
I20250623 01:58:38.057448 17366 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:58:38.063724 17366 webserver.cc:469] Webserver started at http://127.14.43.130:33091/ using document root <none> and password file <none>
I20250623 01:58:38.064567 17366 fs_manager.cc:362] Metadata directory not provided
I20250623 01:58:38.064750 17366 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:58:38.065146 17366 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:58:38.069380 17366 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "e3ec3497126848969a0372dcacc06c59"
format_stamp: "Formatted at 2025-06-23 01:58:38 on dist-test-slave-9rpg"
I20250623 01:58:38.070448 17366 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "e3ec3497126848969a0372dcacc06c59"
format_stamp: "Formatted at 2025-06-23 01:58:38 on dist-test-slave-9rpg"
I20250623 01:58:38.077227 17366 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20250623 01:58:38.082629 17382 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:38.083715 17366 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250623 01:58:38.084029 17366 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "e3ec3497126848969a0372dcacc06c59"
format_stamp: "Formatted at 2025-06-23 01:58:38 on dist-test-slave-9rpg"
I20250623 01:58:38.084343 17366 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:58:38.126643 17366 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:58:38.128018 17366 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:58:38.128412 17366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:58:38.130828 17366 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:58:38.134681 17366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250623 01:58:38.134878 17366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:38.135111 17366 ts_tablet_manager.cc:610] Registered 0 tablets
I20250623 01:58:38.135260 17366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:38.260170 17366 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.130:44383
I20250623 01:58:38.260267 17494 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.130:44383 every 8 connection(s)
I20250623 01:58:38.262622 17366 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250623 01:58:38.273082 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 17366
I20250623 01:58:38.273484 14510 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250623 01:58:38.279325 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.43.131:0
--local_ip_for_outbound_sockets=127.14.43.131
--webserver_interface=127.14.43.131
--webserver_port=0
--tserver_master_addrs=127.14.43.190:33475
--builtin_ntp_servers=127.14.43.148:43657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250623 01:58:38.282186 17495 heartbeater.cc:344] Connected to a master server at 127.14.43.190:33475
I20250623 01:58:38.282580 17495 heartbeater.cc:461] Registering TS with master...
I20250623 01:58:38.283535 17495 heartbeater.cc:507] Master 127.14.43.190:33475 requested a full tablet report, sending...
I20250623 01:58:38.285627 17174 ts_manager.cc:194] Registered new tserver with Master: e3ec3497126848969a0372dcacc06c59 (127.14.43.130:44383)
I20250623 01:58:38.286911 17174 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.43.130:55405
W20250623 01:58:38.568058 17499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:58:38.568694 17499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:58:38.569460 17499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:58:38.605052 17499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:58:38.605875 17499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.43.131
I20250623 01:58:38.642146 17499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:43657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.43.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.14.43.131
--webserver_port=0
--tserver_master_addrs=127.14.43.190:33475
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.43.131
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:58:38.643430 17499 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:58:38.645035 17499 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:58:38.660164 17506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:58:39.289970 17495 heartbeater.cc:499] Master 127.14.43.190:33475 was elected leader, sending a full tablet report...
W20250623 01:58:38.660655 17505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:58:38.660276 17508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:58:38.662210 17499 server_base.cc:1048] running on GCE node
I20250623 01:58:39.727080 17499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:58:39.729287 17499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:58:39.730623 17499 hybrid_clock.cc:648] HybridClock initialized: now 1750643919730584 us; error 50 us; skew 500 ppm
I20250623 01:58:39.731390 17499 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:58:39.741240 17499 webserver.cc:469] Webserver started at http://127.14.43.131:40355/ using document root <none> and password file <none>
I20250623 01:58:39.742163 17499 fs_manager.cc:362] Metadata directory not provided
I20250623 01:58:39.742383 17499 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:58:39.742806 17499 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:58:39.747085 17499 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "41149896fc354b3dacb5d1b32f6788f7"
format_stamp: "Formatted at 2025-06-23 01:58:39 on dist-test-slave-9rpg"
I20250623 01:58:39.748100 17499 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "41149896fc354b3dacb5d1b32f6788f7"
format_stamp: "Formatted at 2025-06-23 01:58:39 on dist-test-slave-9rpg"
I20250623 01:58:39.754909 17499 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250623 01:58:39.760067 17515 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:39.761032 17499 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20250623 01:58:39.761319 17499 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "41149896fc354b3dacb5d1b32f6788f7"
format_stamp: "Formatted at 2025-06-23 01:58:39 on dist-test-slave-9rpg"
I20250623 01:58:39.761652 17499 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:58:39.825934 17499 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:58:39.827369 17499 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:58:39.827766 17499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:58:39.830124 17499 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:58:39.833899 17499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250623 01:58:39.834115 17499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250623 01:58:39.834345 17499 ts_tablet_manager.cc:610] Registered 0 tablets
I20250623 01:58:39.834499 17499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:39.957643 17499 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.131:45611
I20250623 01:58:39.957741 17627 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.131:45611 every 8 connection(s)
I20250623 01:58:39.960100 17499 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250623 01:58:39.963891 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 17499
I20250623 01:58:39.964367 14510 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250623 01:58:39.980276 17628 heartbeater.cc:344] Connected to a master server at 127.14.43.190:33475
I20250623 01:58:39.980692 17628 heartbeater.cc:461] Registering TS with master...
I20250623 01:58:39.981695 17628 heartbeater.cc:507] Master 127.14.43.190:33475 requested a full tablet report, sending...
I20250623 01:58:39.983979 17173 ts_manager.cc:194] Registered new tserver with Master: 41149896fc354b3dacb5d1b32f6788f7 (127.14.43.131:45611)
I20250623 01:58:39.985368 17173 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.43.131:45563
I20250623 01:58:39.998190 14510 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250623 01:58:40.031703 14510 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250623 01:58:40.032044 14510 test_util.cc:276] Using random seed: -1929686757
I20250623 01:58:40.070716 17173 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42708:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 01:58:40.110651 17430 tablet_service.cc:1468] Processing CreateTablet for tablet 8ae52d3edd7748be9366f4baea5b1472 (DEFAULT_TABLE table=TestTable [id=2219689c6e5d4972a4cbb9fae7d27f5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 01:58:40.112110 17430 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ae52d3edd7748be9366f4baea5b1472. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:40.130425 17648 tablet_bootstrap.cc:492] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59: Bootstrap starting.
I20250623 01:58:40.135815 17648 tablet_bootstrap.cc:654] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:40.137459 17648 log.cc:826] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59: Log is configured to *not* fsync() on all Append() calls
I20250623 01:58:40.141963 17648 tablet_bootstrap.cc:492] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59: No bootstrap required, opened a new log
I20250623 01:58:40.142405 17648 ts_tablet_manager.cc:1397] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20250623 01:58:40.159345 17648 raft_consensus.cc:357] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } }
I20250623 01:58:40.160120 17648 raft_consensus.cc:383] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:40.160419 17648 raft_consensus.cc:738] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3ec3497126848969a0372dcacc06c59, State: Initialized, Role: FOLLOWER
I20250623 01:58:40.161281 17648 consensus_queue.cc:260] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } }
I20250623 01:58:40.161955 17648 raft_consensus.cc:397] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250623 01:58:40.162340 17648 raft_consensus.cc:491] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250623 01:58:40.162758 17648 raft_consensus.cc:3058] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:40.168648 17648 raft_consensus.cc:513] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } }
I20250623 01:58:40.169552 17648 leader_election.cc:304] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3ec3497126848969a0372dcacc06c59; no voters:
I20250623 01:58:40.172621 17648 leader_election.cc:290] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250623 01:58:40.173023 17650 raft_consensus.cc:2802] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:40.175300 17650 raft_consensus.cc:695] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [term 1 LEADER]: Becoming Leader. State: Replica: e3ec3497126848969a0372dcacc06c59, State: Running, Role: LEADER
I20250623 01:58:40.176224 17650 consensus_queue.cc:237] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } }
I20250623 01:58:40.183768 17648 ts_tablet_manager.cc:1428] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59: Time spent starting tablet: real 0.041s user 0.026s sys 0.006s
I20250623 01:58:40.185869 17173 catalog_manager.cc:5582] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3ec3497126848969a0372dcacc06c59 (127.14.43.130). New cstate: current_term: 1 leader_uuid: "e3ec3497126848969a0372dcacc06c59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } health_report { overall_health: HEALTHY } } }
I20250623 01:58:40.374073 14510 test_util.cc:276] Using random seed: -1929344739
I20250623 01:58:40.394651 17168 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42720:
name: "TestTable1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 01:58:40.421788 17563 tablet_service.cc:1468] Processing CreateTablet for tablet 62858b6460fa4faa8c4bb9a06c000863 (DEFAULT_TABLE table=TestTable1 [id=eb20ab5775044132b3af8ca0247c6ee8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 01:58:40.423278 17563 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62858b6460fa4faa8c4bb9a06c000863. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:40.441913 17669 tablet_bootstrap.cc:492] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7: Bootstrap starting.
I20250623 01:58:40.447450 17669 tablet_bootstrap.cc:654] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:40.449095 17669 log.cc:826] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7: Log is configured to *not* fsync() on all Append() calls
I20250623 01:58:40.453212 17669 tablet_bootstrap.cc:492] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7: No bootstrap required, opened a new log
I20250623 01:58:40.453589 17669 ts_tablet_manager.cc:1397] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20250623 01:58:40.470719 17669 raft_consensus.cc:357] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } }
I20250623 01:58:40.471310 17669 raft_consensus.cc:383] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:40.471532 17669 raft_consensus.cc:738] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41149896fc354b3dacb5d1b32f6788f7, State: Initialized, Role: FOLLOWER
I20250623 01:58:40.472187 17669 consensus_queue.cc:260] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } }
I20250623 01:58:40.472673 17669 raft_consensus.cc:397] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250623 01:58:40.472920 17669 raft_consensus.cc:491] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250623 01:58:40.473201 17669 raft_consensus.cc:3058] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:40.477273 17669 raft_consensus.cc:513] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } }
I20250623 01:58:40.477928 17669 leader_election.cc:304] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41149896fc354b3dacb5d1b32f6788f7; no voters:
I20250623 01:58:40.479826 17669 leader_election.cc:290] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250623 01:58:40.480190 17671 raft_consensus.cc:2802] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:40.482640 17628 heartbeater.cc:499] Master 127.14.43.190:33475 was elected leader, sending a full tablet report...
I20250623 01:58:40.482995 17671 raft_consensus.cc:695] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [term 1 LEADER]: Becoming Leader. State: Replica: 41149896fc354b3dacb5d1b32f6788f7, State: Running, Role: LEADER
I20250623 01:58:40.483566 17669 ts_tablet_manager.cc:1428] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7: Time spent starting tablet: real 0.030s user 0.026s sys 0.005s
I20250623 01:58:40.483693 17671 consensus_queue.cc:237] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } }
I20250623 01:58:40.493853 17168 catalog_manager.cc:5582] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 41149896fc354b3dacb5d1b32f6788f7 (127.14.43.131). New cstate: current_term: 1 leader_uuid: "41149896fc354b3dacb5d1b32f6788f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } health_report { overall_health: HEALTHY } } }
I20250623 01:58:40.665740 14510 test_util.cc:276] Using random seed: -1929053071
I20250623 01:58:40.689267 17171 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42728:
name: "TestTable2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 01:58:40.718528 17297 tablet_service.cc:1468] Processing CreateTablet for tablet d1cb2e476ce046cbac6fbd6faae2a046 (DEFAULT_TABLE table=TestTable2 [id=a9b73e9cdc214b82bd52a7b58d2e5e5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 01:58:40.719970 17297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1cb2e476ce046cbac6fbd6faae2a046. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:40.738034 17690 tablet_bootstrap.cc:492] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed: Bootstrap starting.
I20250623 01:58:40.743502 17690 tablet_bootstrap.cc:654] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed: Neither blocks nor log segments found. Creating new log.
I20250623 01:58:40.745152 17690 log.cc:826] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed: Log is configured to *not* fsync() on all Append() calls
I20250623 01:58:40.749389 17690 tablet_bootstrap.cc:492] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed: No bootstrap required, opened a new log
I20250623 01:58:40.749784 17690 ts_tablet_manager.cc:1397] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20250623 01:58:40.766664 17690 raft_consensus.cc:357] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } }
I20250623 01:58:40.767228 17690 raft_consensus.cc:383] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:58:40.767472 17690 raft_consensus.cc:738] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 419f59c561fa4aa8ab1a5a311af7a9ed, State: Initialized, Role: FOLLOWER
I20250623 01:58:40.768123 17690 consensus_queue.cc:260] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } }
I20250623 01:58:40.768602 17690 raft_consensus.cc:397] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250623 01:58:40.768857 17690 raft_consensus.cc:491] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250623 01:58:40.769134 17690 raft_consensus.cc:3058] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:58:40.773092 17690 raft_consensus.cc:513] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } }
I20250623 01:58:40.773751 17690 leader_election.cc:304] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 419f59c561fa4aa8ab1a5a311af7a9ed; no voters:
I20250623 01:58:40.775655 17690 leader_election.cc:290] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20250623 01:58:40.776055 17692 raft_consensus.cc:2802] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:58:40.778308 17692 raft_consensus.cc:695] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 1 LEADER]: Becoming Leader. State: Replica: 419f59c561fa4aa8ab1a5a311af7a9ed, State: Running, Role: LEADER
I20250623 01:58:40.779114 17692 consensus_queue.cc:237] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } }
I20250623 01:58:40.779614 17690 ts_tablet_manager.cc:1428] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed: Time spent starting tablet: real 0.030s user 0.023s sys 0.005s
I20250623 01:58:40.787293 17171 catalog_manager.cc:5582] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 419f59c561fa4aa8ab1a5a311af7a9ed (127.14.43.129). New cstate: current_term: 1 leader_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } health_report { overall_health: HEALTHY } } }
I20250623 01:58:40.948385 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 17140
W20250623 01:58:41.188366 17358 debug-util.cc:398] Leaking SignalData structure 0x7b08000acdc0 after lost signal to thread 17234
W20250623 01:58:41.214664 17495 heartbeater.cc:646] Failed to heartbeat to 127.14.43.190:33475 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.43.190:33475: connect: Connection refused (error 111)
W20250623 01:58:41.522653 17628 heartbeater.cc:646] Failed to heartbeat to 127.14.43.190:33475 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.43.190:33475: connect: Connection refused (error 111)
W20250623 01:58:41.819713 17362 heartbeater.cc:646] Failed to heartbeat to 127.14.43.190:33475 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.43.190:33475: connect: Connection refused (error 111)
I20250623 01:58:45.958231 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 17233
I20250623 01:58:45.981918 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 17366
I20250623 01:58:46.001803 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 17499
I20250623 01:58:46.023488 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.43.190:33475
--webserver_interface=127.14.43.190
--webserver_port=40887
--builtin_ntp_servers=127.14.43.148:43657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.43.190:33475 with env {}
W20250623 01:58:46.309242 17769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:58:46.309815 17769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:58:46.310315 17769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:58:46.341535 17769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250623 01:58:46.341858 17769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:58:46.342137 17769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250623 01:58:46.342368 17769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250623 01:58:46.390743 17769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:43657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.43.190:33475
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.43.190:33475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.14.43.190
--webserver_port=40887
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:58:46.392326 17769 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:58:46.394438 17769 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:58:46.411520 17775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:58:46.413053 17776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:58:46.415359 17778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:58:46.416565 17769 server_base.cc:1048] running on GCE node
I20250623 01:58:47.497625 17769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:58:47.500159 17769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:58:47.501499 17769 hybrid_clock.cc:648] HybridClock initialized: now 1750643927501458 us; error 56 us; skew 500 ppm
I20250623 01:58:47.502343 17769 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:58:47.508270 17769 webserver.cc:469] Webserver started at http://127.14.43.190:40887/ using document root <none> and password file <none>
I20250623 01:58:47.509142 17769 fs_manager.cc:362] Metadata directory not provided
I20250623 01:58:47.509325 17769 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:58:47.516892 17769 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20250623 01:58:47.522469 17785 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:47.523485 17769 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250623 01:58:47.523766 17769 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
uuid: "99a1ba904ecf487b894f3609083cce41"
format_stamp: "Formatted at 2025-06-23 01:58:34 on dist-test-slave-9rpg"
I20250623 01:58:47.525616 17769 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:58:47.589802 17769 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:58:47.591245 17769 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:58:47.591648 17769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:58:47.657738 17769 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.190:33475
I20250623 01:58:47.657809 17836 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.190:33475 every 8 connection(s)
I20250623 01:58:47.660565 17769 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250623 01:58:47.664038 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 17769
I20250623 01:58:47.665880 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.43.129:44313
--local_ip_for_outbound_sockets=127.14.43.129
--tserver_master_addrs=127.14.43.190:33475
--webserver_port=37409
--webserver_interface=127.14.43.129
--builtin_ntp_servers=127.14.43.148:43657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250623 01:58:47.673215 17837 sys_catalog.cc:263] Verifying existing consensus state
I20250623 01:58:47.678224 17837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41: Bootstrap starting.
I20250623 01:58:47.687811 17837 log.cc:826] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41: Log is configured to *not* fsync() on all Append() calls
I20250623 01:58:47.735885 17837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41: Bootstrap replayed 1/1 log segments. Stats: ops{read=19 overwritten=0 applied=19 ignored=0} inserts{seen=13 ignored=0} mutations{seen=11 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 01:58:47.736668 17837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41: Bootstrap complete.
I20250623 01:58:47.757639 17837 raft_consensus.cc:357] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a1ba904ecf487b894f3609083cce41" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 33475 } }
I20250623 01:58:47.759776 17837 raft_consensus.cc:738] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a1ba904ecf487b894f3609083cce41, State: Initialized, Role: FOLLOWER
I20250623 01:58:47.760562 17837 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 19, Last appended: 2.19, Last appended by leader: 19, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a1ba904ecf487b894f3609083cce41" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 33475 } }
I20250623 01:58:47.761086 17837 raft_consensus.cc:397] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250623 01:58:47.761346 17837 raft_consensus.cc:491] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250623 01:58:47.761624 17837 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:58:47.766909 17837 raft_consensus.cc:513] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a1ba904ecf487b894f3609083cce41" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 33475 } }
I20250623 01:58:47.767518 17837 leader_election.cc:304] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99a1ba904ecf487b894f3609083cce41; no voters:
I20250623 01:58:47.769402 17837 leader_election.cc:290] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250623 01:58:47.769938 17841 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:58:47.772673 17841 raft_consensus.cc:695] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [term 3 LEADER]: Becoming Leader. State: Replica: 99a1ba904ecf487b894f3609083cce41, State: Running, Role: LEADER
I20250623 01:58:47.773392 17837 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [sys.catalog]: configured and running, proceeding with master startup.
I20250623 01:58:47.773486 17841 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 2.19, Last appended by leader: 19, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a1ba904ecf487b894f3609083cce41" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 33475 } }
I20250623 01:58:47.784083 17842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "99a1ba904ecf487b894f3609083cce41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a1ba904ecf487b894f3609083cce41" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 33475 } } }
I20250623 01:58:47.784795 17842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [sys.catalog]: This master's current role is: LEADER
I20250623 01:58:47.787098 17843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99a1ba904ecf487b894f3609083cce41. Latest consensus state: current_term: 3 leader_uuid: "99a1ba904ecf487b894f3609083cce41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a1ba904ecf487b894f3609083cce41" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 33475 } } }
I20250623 01:58:47.787844 17843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [sys.catalog]: This master's current role is: LEADER
I20250623 01:58:47.789553 17848 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250623 01:58:47.802920 17848 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=a9b73e9cdc214b82bd52a7b58d2e5e5f]
I20250623 01:58:47.805307 17848 catalog_manager.cc:671] Loaded metadata for table TestTable [id=aa109a0399cc4f4297ef2e2348a3434c]
I20250623 01:58:47.806905 17848 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=aaa2b1d5e1b343f2b9def29de1bc6d7c]
I20250623 01:58:47.814654 17848 tablet_loader.cc:96] loaded metadata for tablet 62858b6460fa4faa8c4bb9a06c000863 (table TestTable1 [id=aaa2b1d5e1b343f2b9def29de1bc6d7c])
I20250623 01:58:47.816015 17848 tablet_loader.cc:96] loaded metadata for tablet 8ae52d3edd7748be9366f4baea5b1472 (table TestTable [id=aa109a0399cc4f4297ef2e2348a3434c])
I20250623 01:58:47.817407 17848 tablet_loader.cc:96] loaded metadata for tablet d1cb2e476ce046cbac6fbd6faae2a046 (table TestTable2 [id=a9b73e9cdc214b82bd52a7b58d2e5e5f])
I20250623 01:58:47.818856 17848 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250623 01:58:47.823958 17848 catalog_manager.cc:1261] Loaded cluster ID: 820e7a8f9f4b48d3a79721cb8afe844e
I20250623 01:58:47.824302 17848 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250623 01:58:47.832441 17848 catalog_manager.cc:1506] Loading token signing keys...
I20250623 01:58:47.837700 17848 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41: Loaded TSK: 0
I20250623 01:58:47.839457 17848 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250623 01:58:47.985864 17839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:58:47.986421 17839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:58:47.986896 17839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:58:48.017895 17839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:58:48.018908 17839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.43.129
I20250623 01:58:48.053450 17839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:43657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.43.129:44313
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.14.43.129
--webserver_port=37409
--tserver_master_addrs=127.14.43.190:33475
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.43.129
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:58:48.054759 17839 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:58:48.056385 17839 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:58:48.072949 17864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:58:48.074072 17865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:58:48.076215 17839 server_base.cc:1048] running on GCE node
W20250623 01:58:48.074858 17867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:58:49.145706 17839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:58:49.148281 17839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:58:49.149725 17839 hybrid_clock.cc:648] HybridClock initialized: now 1750643929149665 us; error 77 us; skew 500 ppm
I20250623 01:58:49.150542 17839 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:58:49.156872 17839 webserver.cc:469] Webserver started at http://127.14.43.129:37409/ using document root <none> and password file <none>
I20250623 01:58:49.157756 17839 fs_manager.cc:362] Metadata directory not provided
I20250623 01:58:49.157969 17839 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:58:49.165838 17839 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20250623 01:58:49.170524 17874 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:49.171576 17839 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250623 01:58:49.171870 17839 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "419f59c561fa4aa8ab1a5a311af7a9ed"
format_stamp: "Formatted at 2025-06-23 01:58:36 on dist-test-slave-9rpg"
I20250623 01:58:49.173751 17839 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:58:49.229429 17839 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:58:49.230907 17839 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:58:49.231338 17839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:58:49.233865 17839 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:58:49.240137 17881 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250623 01:58:49.247479 17839 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250623 01:58:49.247706 17839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.002s sys 0.000s
I20250623 01:58:49.247983 17839 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250623 01:58:49.252600 17839 ts_tablet_manager.cc:610] Registered 1 tablets
I20250623 01:58:49.252853 17839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.003s sys 0.000s
I20250623 01:58:49.253237 17881 tablet_bootstrap.cc:492] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed: Bootstrap starting.
I20250623 01:58:49.327961 17881 log.cc:826] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed: Log is configured to *not* fsync() on all Append() calls
I20250623 01:58:49.428786 17839 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.129:44313
I20250623 01:58:49.428969 17988 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.129:44313 every 8 connection(s)
I20250623 01:58:49.432396 17839 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250623 01:58:49.434510 17881 tablet_bootstrap.cc:492] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 01:58:49.435611 17881 tablet_bootstrap.cc:492] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed: Bootstrap complete.
I20250623 01:58:49.436501 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 17839
I20250623 01:58:49.438205 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.43.130:44383
--local_ip_for_outbound_sockets=127.14.43.130
--tserver_master_addrs=127.14.43.190:33475
--webserver_port=33091
--webserver_interface=127.14.43.130
--builtin_ntp_servers=127.14.43.148:43657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250623 01:58:49.451391 17881 ts_tablet_manager.cc:1397] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed: Time spent bootstrapping tablet: real 0.199s user 0.131s sys 0.047s
I20250623 01:58:49.474558 17881 raft_consensus.cc:357] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } }
I20250623 01:58:49.477537 17881 raft_consensus.cc:738] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 419f59c561fa4aa8ab1a5a311af7a9ed, State: Initialized, Role: FOLLOWER
I20250623 01:58:49.478717 17881 consensus_queue.cc:260] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } }
I20250623 01:58:49.479357 17881 raft_consensus.cc:397] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250623 01:58:49.479708 17881 raft_consensus.cc:491] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250623 01:58:49.480190 17881 raft_consensus.cc:3058] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:49.486877 17989 heartbeater.cc:344] Connected to a master server at 127.14.43.190:33475
I20250623 01:58:49.487345 17989 heartbeater.cc:461] Registering TS with master...
I20250623 01:58:49.488505 17989 heartbeater.cc:507] Master 127.14.43.190:33475 requested a full tablet report, sending...
I20250623 01:58:49.488431 17881 raft_consensus.cc:513] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } }
I20250623 01:58:49.489408 17881 leader_election.cc:304] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 419f59c561fa4aa8ab1a5a311af7a9ed; no voters:
I20250623 01:58:49.492079 17881 leader_election.cc:290] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed [CANDIDATE]: Term 2 election: Requested vote from peers
I20250623 01:58:49.492503 17994 raft_consensus.cc:2802] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:58:49.493660 17802 ts_manager.cc:194] Registered new tserver with Master: 419f59c561fa4aa8ab1a5a311af7a9ed (127.14.43.129:44313)
I20250623 01:58:49.497802 17994 raft_consensus.cc:695] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 LEADER]: Becoming Leader. State: Replica: 419f59c561fa4aa8ab1a5a311af7a9ed, State: Running, Role: LEADER
I20250623 01:58:49.498687 17994 consensus_queue.cc:237] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } }
I20250623 01:58:49.501243 17802 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.43.129:52939
I20250623 01:58:49.503163 17881 ts_tablet_manager.cc:1428] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed: Time spent starting tablet: real 0.051s user 0.037s sys 0.004s
I20250623 01:58:49.507627 17989 heartbeater.cc:499] Master 127.14.43.190:33475 was elected leader, sending a full tablet report...
I20250623 01:58:49.516537 17802 catalog_manager.cc:5582] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } health_report { overall_health: HEALTHY } } }
W20250623 01:58:49.777837 17993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:58:49.778367 17993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:58:49.778841 17993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:58:49.809513 17993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:58:49.810395 17993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.43.130
I20250623 01:58:49.844827 17993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:43657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.43.130:44383
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.14.43.130
--webserver_port=33091
--tserver_master_addrs=127.14.43.190:33475
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.43.130
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:58:49.846189 17993 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:58:49.847715 17993 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:58:49.862730 18009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:58:49.863128 18011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:58:49.864475 17993 server_base.cc:1048] running on GCE node
W20250623 01:58:49.863432 18008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:58:50.927747 17993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:58:50.929888 17993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:58:50.931280 17993 hybrid_clock.cc:648] HybridClock initialized: now 1750643930931243 us; error 49 us; skew 500 ppm
I20250623 01:58:50.932039 17993 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:58:50.937848 17993 webserver.cc:469] Webserver started at http://127.14.43.130:33091/ using document root <none> and password file <none>
I20250623 01:58:50.938740 17993 fs_manager.cc:362] Metadata directory not provided
I20250623 01:58:50.938964 17993 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:58:50.946456 17993 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20250623 01:58:50.950848 18018 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:50.951936 17993 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250623 01:58:50.952229 17993 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "e3ec3497126848969a0372dcacc06c59"
format_stamp: "Formatted at 2025-06-23 01:58:38 on dist-test-slave-9rpg"
I20250623 01:58:50.954116 17993 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:58:51.004220 17993 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:58:51.005697 17993 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:58:51.006147 17993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:58:51.008504 17993 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:58:51.013927 18025 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250623 01:58:51.021143 17993 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250623 01:58:51.021394 17993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.002s sys 0.000s
I20250623 01:58:51.021662 17993 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250623 01:58:51.026176 17993 ts_tablet_manager.cc:610] Registered 1 tablets
I20250623 01:58:51.026365 17993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.005s sys 0.000s
I20250623 01:58:51.026726 18025 tablet_bootstrap.cc:492] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59: Bootstrap starting.
I20250623 01:58:51.082160 18025 log.cc:826] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59: Log is configured to *not* fsync() on all Append() calls
I20250623 01:58:51.168450 18025 tablet_bootstrap.cc:492] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 01:58:51.169190 18025 tablet_bootstrap.cc:492] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59: Bootstrap complete.
I20250623 01:58:51.170742 18025 ts_tablet_manager.cc:1397] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59: Time spent bootstrapping tablet: real 0.144s user 0.102s sys 0.039s
I20250623 01:58:51.186833 18025 raft_consensus.cc:357] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } }
I20250623 01:58:51.189218 18025 raft_consensus.cc:738] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3ec3497126848969a0372dcacc06c59, State: Initialized, Role: FOLLOWER
I20250623 01:58:51.190047 18025 consensus_queue.cc:260] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } }
I20250623 01:58:51.190565 18025 raft_consensus.cc:397] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250623 01:58:51.190894 18025 raft_consensus.cc:491] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250623 01:58:51.191243 18025 raft_consensus.cc:3058] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:51.194315 17993 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.130:44383
I20250623 01:58:51.194485 18132 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.130:44383 every 8 connection(s)
I20250623 01:58:51.196697 17993 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250623 01:58:51.196583 18025 raft_consensus.cc:513] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } }
I20250623 01:58:51.197196 18025 leader_election.cc:304] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3ec3497126848969a0372dcacc06c59; no voters:
I20250623 01:58:51.199671 18025 leader_election.cc:290] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250623 01:58:51.200802 18134 raft_consensus.cc:2802] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:58:51.203562 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 17993
I20250623 01:58:51.205451 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.43.131:45611
--local_ip_for_outbound_sockets=127.14.43.131
--tserver_master_addrs=127.14.43.190:33475
--webserver_port=40355
--webserver_interface=127.14.43.131
--builtin_ntp_servers=127.14.43.148:43657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250623 01:58:51.231546 18134 raft_consensus.cc:695] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [term 2 LEADER]: Becoming Leader. State: Replica: e3ec3497126848969a0372dcacc06c59, State: Running, Role: LEADER
I20250623 01:58:51.232580 18134 consensus_queue.cc:237] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } }
I20250623 01:58:51.241374 18025 ts_tablet_manager.cc:1428] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59: Time spent starting tablet: real 0.070s user 0.032s sys 0.034s
I20250623 01:58:51.246206 18133 heartbeater.cc:344] Connected to a master server at 127.14.43.190:33475
I20250623 01:58:51.246712 18133 heartbeater.cc:461] Registering TS with master...
I20250623 01:58:51.247908 18133 heartbeater.cc:507] Master 127.14.43.190:33475 requested a full tablet report, sending...
I20250623 01:58:51.251863 17802 ts_manager.cc:194] Registered new tserver with Master: e3ec3497126848969a0372dcacc06c59 (127.14.43.130:44383)
I20250623 01:58:51.253095 17802 catalog_manager.cc:5582] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 reported cstate change: term changed from 0 to 2, leader changed from <none> to e3ec3497126848969a0372dcacc06c59 (127.14.43.130), VOTER e3ec3497126848969a0372dcacc06c59 (127.14.43.130) added. New cstate: current_term: 2 leader_uuid: "e3ec3497126848969a0372dcacc06c59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } health_report { overall_health: HEALTHY } } }
I20250623 01:58:51.265218 17802 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.43.130:55849
I20250623 01:58:51.269388 18133 heartbeater.cc:499] Master 127.14.43.190:33475 was elected leader, sending a full tablet report...
I20250623 01:58:51.279174 18088 consensus_queue.cc:237] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: NON_VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: true } }
I20250623 01:58:51.282600 18134 raft_consensus.cc:2953] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index -1 to 10, NON_VOTER 419f59c561fa4aa8ab1a5a311af7a9ed (127.14.43.129) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: NON_VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: true } } }
I20250623 01:58:51.289397 17788 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8ae52d3edd7748be9366f4baea5b1472 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250623 01:58:51.292452 17802 catalog_manager.cc:5582] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 reported cstate change: config changed from index -1 to 10, NON_VOTER 419f59c561fa4aa8ab1a5a311af7a9ed (127.14.43.129) added. New cstate: current_term: 2 leader_uuid: "e3ec3497126848969a0372dcacc06c59" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: NON_VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250623 01:58:51.301546 17802 catalog_manager.cc:5260] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8ae52d3edd7748be9366f4baea5b1472 with cas_config_opid_index 10: no extra replica candidate found for tablet 8ae52d3edd7748be9366f4baea5b1472 (table TestTable [id=aa109a0399cc4f4297ef2e2348a3434c]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (2) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
W20250623 01:58:51.301594 18022 consensus_peers.cc:487] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 -> Peer 419f59c561fa4aa8ab1a5a311af7a9ed (127.14.43.129:44313): Couldn't send request to peer 419f59c561fa4aa8ab1a5a311af7a9ed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ae52d3edd7748be9366f4baea5b1472. This is attempt 1: this message will repeat every 5th retry.
W20250623 01:58:51.524242 18140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:58:51.524749 18140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:58:51.525243 18140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:58:51.556679 18140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:58:51.557577 18140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.43.131
I20250623 01:58:51.593938 18140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:43657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.43.131:45611
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.14.43.131
--webserver_port=40355
--tserver_master_addrs=127.14.43.190:33475
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.43.131
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:58:51.595270 18140 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:58:51.596880 18140 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:58:51.611814 18155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:58:51.819181 18162 ts_tablet_manager.cc:927] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed: Initiating tablet copy from peer e3ec3497126848969a0372dcacc06c59 (127.14.43.130:44383)
I20250623 01:58:51.821156 18162 tablet_copy_client.cc:323] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed: tablet copy: Beginning tablet copy session from remote peer at address 127.14.43.130:44383
I20250623 01:58:51.831178 18108 tablet_copy_service.cc:140] P e3ec3497126848969a0372dcacc06c59: Received BeginTabletCopySession request for tablet 8ae52d3edd7748be9366f4baea5b1472 from peer 419f59c561fa4aa8ab1a5a311af7a9ed ({username='slave'} at 127.14.43.129:38903)
I20250623 01:58:51.831877 18108 tablet_copy_service.cc:161] P e3ec3497126848969a0372dcacc06c59: Beginning new tablet copy session on tablet 8ae52d3edd7748be9366f4baea5b1472 from peer 419f59c561fa4aa8ab1a5a311af7a9ed at {username='slave'} at 127.14.43.129:38903: session id = 419f59c561fa4aa8ab1a5a311af7a9ed-8ae52d3edd7748be9366f4baea5b1472
I20250623 01:58:51.839534 18108 tablet_copy_source_session.cc:215] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59: Tablet Copy: opened 0 blocks and 1 log segments
I20250623 01:58:51.843750 18162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ae52d3edd7748be9366f4baea5b1472. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:51.853827 18162 tablet_copy_client.cc:806] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed: tablet copy: Starting download of 0 data blocks...
I20250623 01:58:51.854403 18162 tablet_copy_client.cc:670] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed: tablet copy: Starting download of 1 WAL segments...
I20250623 01:58:51.859143 18162 tablet_copy_client.cc:538] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250623 01:58:51.864415 18162 tablet_bootstrap.cc:492] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed: Bootstrap starting.
I20250623 01:58:51.950635 18162 tablet_bootstrap.cc:492] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 01:58:51.951253 18162 tablet_bootstrap.cc:492] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed: Bootstrap complete.
I20250623 01:58:51.951663 18162 ts_tablet_manager.cc:1397] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed: Time spent bootstrapping tablet: real 0.087s user 0.079s sys 0.008s
I20250623 01:58:51.953276 18162 raft_consensus.cc:357] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: NON_VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: true } }
I20250623 01:58:51.953675 18162 raft_consensus.cc:738] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 419f59c561fa4aa8ab1a5a311af7a9ed, State: Initialized, Role: LEARNER
I20250623 01:58:51.954135 18162 consensus_queue.cc:260] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: NON_VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: true } }
I20250623 01:58:51.956960 18162 ts_tablet_manager.cc:1428] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20250623 01:58:51.958892 18108 tablet_copy_service.cc:342] P e3ec3497126848969a0372dcacc06c59: Request end of tablet copy session 419f59c561fa4aa8ab1a5a311af7a9ed-8ae52d3edd7748be9366f4baea5b1472 received from {username='slave'} at 127.14.43.129:38903
I20250623 01:58:51.959224 18108 tablet_copy_service.cc:434] P e3ec3497126848969a0372dcacc06c59: ending tablet copy session 419f59c561fa4aa8ab1a5a311af7a9ed-8ae52d3edd7748be9366f4baea5b1472 on tablet 8ae52d3edd7748be9366f4baea5b1472 with peer 419f59c561fa4aa8ab1a5a311af7a9ed
I20250623 01:58:52.249076 17944 raft_consensus.cc:1215] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 LEARNER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
W20250623 01:58:51.613523 18156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:58:51.613379 18158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:58:51.613999 18140 server_base.cc:1048] running on GCE node
I20250623 01:58:52.697134 18140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:58:52.699273 18140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:58:52.700613 18140 hybrid_clock.cc:648] HybridClock initialized: now 1750643932700573 us; error 49 us; skew 500 ppm
I20250623 01:58:52.701390 18140 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:58:52.710268 18140 webserver.cc:469] Webserver started at http://127.14.43.131:40355/ using document root <none> and password file <none>
I20250623 01:58:52.711161 18140 fs_manager.cc:362] Metadata directory not provided
I20250623 01:58:52.711395 18140 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:58:52.718824 18140 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20250623 01:58:52.723564 18174 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:52.724587 18140 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20250623 01:58:52.724870 18140 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "41149896fc354b3dacb5d1b32f6788f7"
format_stamp: "Formatted at 2025-06-23 01:58:39 on dist-test-slave-9rpg"
I20250623 01:58:52.726732 18140 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:58:52.776782 18140 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:58:52.778223 18140 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:58:52.778643 18140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:58:52.781086 18140 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:58:52.786489 18181 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250623 01:58:52.793661 18140 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250623 01:58:52.793876 18140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.002s sys 0.000s
I20250623 01:58:52.794178 18140 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250623 01:58:52.798571 18140 ts_tablet_manager.cc:610] Registered 1 tablets
I20250623 01:58:52.798758 18140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.003s sys 0.001s
I20250623 01:58:52.799093 18181 tablet_bootstrap.cc:492] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7: Bootstrap starting.
I20250623 01:58:52.852128 18181 log.cc:826] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7: Log is configured to *not* fsync() on all Append() calls
I20250623 01:58:52.865413 18217 raft_consensus.cc:1062] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59: attempting to promote NON_VOTER 419f59c561fa4aa8ab1a5a311af7a9ed to VOTER
I20250623 01:58:52.867266 18217 consensus_queue.cc:237] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } }
I20250623 01:58:52.877071 17944 raft_consensus.cc:1273] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 LEARNER]: Refusing update from remote peer e3ec3497126848969a0372dcacc06c59: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250623 01:58:52.878587 18217 consensus_queue.cc:1035] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250623 01:58:52.891511 18217 raft_consensus.cc:2953] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 419f59c561fa4aa8ab1a5a311af7a9ed (127.14.43.129) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } } }
I20250623 01:58:52.892812 17944 raft_consensus.cc:2953] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 419f59c561fa4aa8ab1a5a311af7a9ed (127.14.43.129) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } } }
I20250623 01:58:52.905212 17801 catalog_manager.cc:5582] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 reported cstate change: config changed from index 10 to 11, 419f59c561fa4aa8ab1a5a311af7a9ed (127.14.43.129) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "e3ec3497126848969a0372dcacc06c59" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250623 01:58:52.946066 18181 tablet_bootstrap.cc:492] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 01:58:52.946813 18181 tablet_bootstrap.cc:492] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7: Bootstrap complete.
I20250623 01:58:52.947992 18181 ts_tablet_manager.cc:1397] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7: Time spent bootstrapping tablet: real 0.149s user 0.086s sys 0.059s
I20250623 01:58:52.962437 18181 raft_consensus.cc:357] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } }
I20250623 01:58:52.965468 18181 raft_consensus.cc:738] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41149896fc354b3dacb5d1b32f6788f7, State: Initialized, Role: FOLLOWER
I20250623 01:58:52.966305 18181 consensus_queue.cc:260] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } }
I20250623 01:58:52.966770 18181 raft_consensus.cc:397] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250623 01:58:52.967020 18181 raft_consensus.cc:491] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250623 01:58:52.967314 18181 raft_consensus.cc:3058] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [term 1 FOLLOWER]: Advancing to term 2
I20250623 01:58:52.972920 18181 raft_consensus.cc:513] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } }
I20250623 01:58:52.973582 18181 leader_election.cc:304] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41149896fc354b3dacb5d1b32f6788f7; no voters:
I20250623 01:58:52.976176 18181 leader_election.cc:290] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250623 01:58:52.976544 18282 raft_consensus.cc:2802] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 01:58:52.980463 18181 ts_tablet_manager.cc:1428] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7: Time spent starting tablet: real 0.032s user 0.024s sys 0.006s
I20250623 01:58:52.981490 18282 raft_consensus.cc:695] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [term 2 LEADER]: Becoming Leader. State: Replica: 41149896fc354b3dacb5d1b32f6788f7, State: Running, Role: LEADER
I20250623 01:58:52.982566 18282 consensus_queue.cc:237] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } }
I20250623 01:58:52.999552 18140 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.131:45611
I20250623 01:58:53.000147 18302 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.131:45611 every 8 connection(s)
I20250623 01:58:53.001962 18140 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250623 01:58:53.003418 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 18140
I20250623 01:58:53.027825 18303 heartbeater.cc:344] Connected to a master server at 127.14.43.190:33475
I20250623 01:58:53.028324 18303 heartbeater.cc:461] Registering TS with master...
I20250623 01:58:53.029366 18303 heartbeater.cc:507] Master 127.14.43.190:33475 requested a full tablet report, sending...
I20250623 01:58:53.032505 17801 ts_manager.cc:194] Registered new tserver with Master: 41149896fc354b3dacb5d1b32f6788f7 (127.14.43.131:45611)
I20250623 01:58:53.033428 17801 catalog_manager.cc:5582] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 reported cstate change: term changed from 0 to 2, leader changed from <none> to 41149896fc354b3dacb5d1b32f6788f7 (127.14.43.131), VOTER 41149896fc354b3dacb5d1b32f6788f7 (127.14.43.131) added. New cstate: current_term: 2 leader_uuid: "41149896fc354b3dacb5d1b32f6788f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } health_report { overall_health: HEALTHY } } }
I20250623 01:58:53.042663 17801 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.43.131:40789
I20250623 01:58:53.045791 14510 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250623 01:58:53.046911 18303 heartbeater.cc:499] Master 127.14.43.190:33475 was elected leader, sending a full tablet report...
I20250623 01:58:53.050822 14510 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
W20250623 01:58:53.054404 14510 ts_itest-base.cc:209] found only 2 out of 3 replicas of tablet 8ae52d3edd7748be9366f4baea5b1472: tablet_id: "8ae52d3edd7748be9366f4baea5b1472" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER } interned_replicas { ts_info_idx: 1 role: FOLLOWER }
I20250623 01:58:53.057157 18253 consensus_queue.cc:237] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } } peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: NON_VOTER last_known_addr { host: "127.14.43.130" port: 44383 } attrs { promote: true } }
I20250623 01:58:53.060210 18286 raft_consensus.cc:2953] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index -1 to 8, NON_VOTER e3ec3497126848969a0372dcacc06c59 (127.14.43.130) added. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } } peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: NON_VOTER last_known_addr { host: "127.14.43.130" port: 44383 } attrs { promote: true } } }
I20250623 01:58:53.067957 17787 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 62858b6460fa4faa8c4bb9a06c000863 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250623 01:58:53.069808 18177 consensus_peers.cc:487] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 -> Peer e3ec3497126848969a0372dcacc06c59 (127.14.43.130:44383): Couldn't send request to peer e3ec3497126848969a0372dcacc06c59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62858b6460fa4faa8c4bb9a06c000863. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:58:53.070520 17801 catalog_manager.cc:5582] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 reported cstate change: config changed from index -1 to 8, NON_VOTER e3ec3497126848969a0372dcacc06c59 (127.14.43.130) added. New cstate: current_term: 2 leader_uuid: "41149896fc354b3dacb5d1b32f6788f7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: NON_VOTER last_known_addr { host: "127.14.43.130" port: 44383 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:53.078963 18253 consensus_queue.cc:237] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } } peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: NON_VOTER last_known_addr { host: "127.14.43.130" port: 44383 } attrs { promote: true } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: NON_VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: true } }
I20250623 01:58:53.082046 18286 raft_consensus.cc:2953] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 419f59c561fa4aa8ab1a5a311af7a9ed (127.14.43.129) added. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } } peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: NON_VOTER last_known_addr { host: "127.14.43.130" port: 44383 } attrs { promote: true } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: NON_VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: true } } }
W20250623 01:58:53.083890 18177 consensus_peers.cc:487] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 -> Peer e3ec3497126848969a0372dcacc06c59 (127.14.43.130:44383): Couldn't send request to peer e3ec3497126848969a0372dcacc06c59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62858b6460fa4faa8c4bb9a06c000863. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:58:53.088529 17787 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 62858b6460fa4faa8c4bb9a06c000863 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250623 01:58:53.091550 17801 catalog_manager.cc:5582] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 reported cstate change: config changed from index 8 to 9, NON_VOTER 419f59c561fa4aa8ab1a5a311af7a9ed (127.14.43.129) added. New cstate: current_term: 2 leader_uuid: "41149896fc354b3dacb5d1b32f6788f7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: NON_VOTER last_known_addr { host: "127.14.43.130" port: 44383 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: NON_VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250623 01:58:53.092820 18178 consensus_peers.cc:487] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 -> Peer 419f59c561fa4aa8ab1a5a311af7a9ed (127.14.43.129:44313): Couldn't send request to peer 419f59c561fa4aa8ab1a5a311af7a9ed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62858b6460fa4faa8c4bb9a06c000863. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:58:53.132189 18088 consensus_queue.cc:237] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } } peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: NON_VOTER last_known_addr { host: "127.14.43.131" port: 45611 } attrs { promote: true } }
I20250623 01:58:53.136535 17944 raft_consensus.cc:1273] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 FOLLOWER]: Refusing update from remote peer e3ec3497126848969a0372dcacc06c59: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250623 01:58:53.137696 18222 consensus_queue.cc:1035] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250623 01:58:53.142659 18217 raft_consensus.cc:2953] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 41149896fc354b3dacb5d1b32f6788f7 (127.14.43.131) added. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } } peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: NON_VOTER last_known_addr { host: "127.14.43.131" port: 45611 } attrs { promote: true } } }
I20250623 01:58:53.144279 17944 raft_consensus.cc:2953] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 41149896fc354b3dacb5d1b32f6788f7 (127.14.43.131) added. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } } peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: NON_VOTER last_known_addr { host: "127.14.43.131" port: 45611 } attrs { promote: true } } }
W20250623 01:58:53.146104 18020 consensus_peers.cc:487] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 -> Peer 41149896fc354b3dacb5d1b32f6788f7 (127.14.43.131:45611): Couldn't send request to peer 41149896fc354b3dacb5d1b32f6788f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ae52d3edd7748be9366f4baea5b1472. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:58:53.148934 17788 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8ae52d3edd7748be9366f4baea5b1472 with cas_config_opid_index 11: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 4)
I20250623 01:58:53.152101 17801 catalog_manager.cc:5582] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 reported cstate change: config changed from index 11 to 12, NON_VOTER 41149896fc354b3dacb5d1b32f6788f7 (127.14.43.131) added. New cstate: current_term: 2 leader_uuid: "e3ec3497126848969a0372dcacc06c59" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: NON_VOTER last_known_addr { host: "127.14.43.131" port: 45611 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:53.580207 18314 ts_tablet_manager.cc:927] T 62858b6460fa4faa8c4bb9a06c000863 P e3ec3497126848969a0372dcacc06c59: Initiating tablet copy from peer 41149896fc354b3dacb5d1b32f6788f7 (127.14.43.131:45611)
I20250623 01:58:53.582228 18314 tablet_copy_client.cc:323] T 62858b6460fa4faa8c4bb9a06c000863 P e3ec3497126848969a0372dcacc06c59: tablet copy: Beginning tablet copy session from remote peer at address 127.14.43.131:45611
I20250623 01:58:53.583825 18273 tablet_copy_service.cc:140] P 41149896fc354b3dacb5d1b32f6788f7: Received BeginTabletCopySession request for tablet 62858b6460fa4faa8c4bb9a06c000863 from peer e3ec3497126848969a0372dcacc06c59 ({username='slave'} at 127.14.43.130:33093)
I20250623 01:58:53.584285 18273 tablet_copy_service.cc:161] P 41149896fc354b3dacb5d1b32f6788f7: Beginning new tablet copy session on tablet 62858b6460fa4faa8c4bb9a06c000863 from peer e3ec3497126848969a0372dcacc06c59 at {username='slave'} at 127.14.43.130:33093: session id = e3ec3497126848969a0372dcacc06c59-62858b6460fa4faa8c4bb9a06c000863
I20250623 01:58:53.585508 18316 ts_tablet_manager.cc:927] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed: Initiating tablet copy from peer 41149896fc354b3dacb5d1b32f6788f7 (127.14.43.131:45611)
I20250623 01:58:53.587225 18316 tablet_copy_client.cc:323] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed: tablet copy: Beginning tablet copy session from remote peer at address 127.14.43.131:45611
I20250623 01:58:53.591032 18273 tablet_copy_source_session.cc:215] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7: Tablet Copy: opened 0 blocks and 1 log segments
I20250623 01:58:53.594211 18314 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62858b6460fa4faa8c4bb9a06c000863. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:53.597941 18273 tablet_copy_service.cc:140] P 41149896fc354b3dacb5d1b32f6788f7: Received BeginTabletCopySession request for tablet 62858b6460fa4faa8c4bb9a06c000863 from peer 419f59c561fa4aa8ab1a5a311af7a9ed ({username='slave'} at 127.14.43.129:55435)
I20250623 01:58:53.598363 18273 tablet_copy_service.cc:161] P 41149896fc354b3dacb5d1b32f6788f7: Beginning new tablet copy session on tablet 62858b6460fa4faa8c4bb9a06c000863 from peer 419f59c561fa4aa8ab1a5a311af7a9ed at {username='slave'} at 127.14.43.129:55435: session id = 419f59c561fa4aa8ab1a5a311af7a9ed-62858b6460fa4faa8c4bb9a06c000863
I20250623 01:58:53.598992 17787 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8ae52d3edd7748be9366f4baea5b1472 with cas_config_opid_index 10: aborting the task: latest config opid_index 12; task opid_index 10
I20250623 01:58:53.602569 18273 tablet_copy_source_session.cc:215] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7: Tablet Copy: opened 0 blocks and 1 log segments
I20250623 01:58:53.604732 18316 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62858b6460fa4faa8c4bb9a06c000863. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:53.605341 18314 tablet_copy_client.cc:806] T 62858b6460fa4faa8c4bb9a06c000863 P e3ec3497126848969a0372dcacc06c59: tablet copy: Starting download of 0 data blocks...
I20250623 01:58:53.605804 18314 tablet_copy_client.cc:670] T 62858b6460fa4faa8c4bb9a06c000863 P e3ec3497126848969a0372dcacc06c59: tablet copy: Starting download of 1 WAL segments...
I20250623 01:58:53.609689 18314 tablet_copy_client.cc:538] T 62858b6460fa4faa8c4bb9a06c000863 P e3ec3497126848969a0372dcacc06c59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250623 01:58:53.615121 18316 tablet_copy_client.cc:806] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed: tablet copy: Starting download of 0 data blocks...
I20250623 01:58:53.615563 18314 tablet_bootstrap.cc:492] T 62858b6460fa4faa8c4bb9a06c000863 P e3ec3497126848969a0372dcacc06c59: Bootstrap starting.
I20250623 01:58:53.615586 18316 tablet_copy_client.cc:670] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed: tablet copy: Starting download of 1 WAL segments...
I20250623 01:58:53.618863 18316 tablet_copy_client.cc:538] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250623 01:58:53.623903 18316 tablet_bootstrap.cc:492] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed: Bootstrap starting.
I20250623 01:58:53.652115 18321 ts_tablet_manager.cc:927] T 8ae52d3edd7748be9366f4baea5b1472 P 41149896fc354b3dacb5d1b32f6788f7: Initiating tablet copy from peer e3ec3497126848969a0372dcacc06c59 (127.14.43.130:44383)
I20250623 01:58:53.654562 18321 tablet_copy_client.cc:323] T 8ae52d3edd7748be9366f4baea5b1472 P 41149896fc354b3dacb5d1b32f6788f7: tablet copy: Beginning tablet copy session from remote peer at address 127.14.43.130:44383
I20250623 01:58:53.656338 18108 tablet_copy_service.cc:140] P e3ec3497126848969a0372dcacc06c59: Received BeginTabletCopySession request for tablet 8ae52d3edd7748be9366f4baea5b1472 from peer 41149896fc354b3dacb5d1b32f6788f7 ({username='slave'} at 127.14.43.131:53245)
I20250623 01:58:53.656803 18108 tablet_copy_service.cc:161] P e3ec3497126848969a0372dcacc06c59: Beginning new tablet copy session on tablet 8ae52d3edd7748be9366f4baea5b1472 from peer 41149896fc354b3dacb5d1b32f6788f7 at {username='slave'} at 127.14.43.131:53245: session id = 41149896fc354b3dacb5d1b32f6788f7-8ae52d3edd7748be9366f4baea5b1472
I20250623 01:58:53.664148 18108 tablet_copy_source_session.cc:215] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59: Tablet Copy: opened 0 blocks and 1 log segments
I20250623 01:58:53.667726 18321 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ae52d3edd7748be9366f4baea5b1472. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:58:53.687167 18321 tablet_copy_client.cc:806] T 8ae52d3edd7748be9366f4baea5b1472 P 41149896fc354b3dacb5d1b32f6788f7: tablet copy: Starting download of 0 data blocks...
I20250623 01:58:53.687798 18321 tablet_copy_client.cc:670] T 8ae52d3edd7748be9366f4baea5b1472 P 41149896fc354b3dacb5d1b32f6788f7: tablet copy: Starting download of 1 WAL segments...
I20250623 01:58:53.692162 18321 tablet_copy_client.cc:538] T 8ae52d3edd7748be9366f4baea5b1472 P 41149896fc354b3dacb5d1b32f6788f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250623 01:58:53.700284 18321 tablet_bootstrap.cc:492] T 8ae52d3edd7748be9366f4baea5b1472 P 41149896fc354b3dacb5d1b32f6788f7: Bootstrap starting.
I20250623 01:58:53.711395 18314 tablet_bootstrap.cc:492] T 62858b6460fa4faa8c4bb9a06c000863 P e3ec3497126848969a0372dcacc06c59: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 01:58:53.712224 18314 tablet_bootstrap.cc:492] T 62858b6460fa4faa8c4bb9a06c000863 P e3ec3497126848969a0372dcacc06c59: Bootstrap complete.
I20250623 01:58:53.712839 18314 ts_tablet_manager.cc:1397] T 62858b6460fa4faa8c4bb9a06c000863 P e3ec3497126848969a0372dcacc06c59: Time spent bootstrapping tablet: real 0.097s user 0.082s sys 0.010s
I20250623 01:58:53.716475 18314 raft_consensus.cc:357] T 62858b6460fa4faa8c4bb9a06c000863 P e3ec3497126848969a0372dcacc06c59 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } } peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: NON_VOTER last_known_addr { host: "127.14.43.130" port: 44383 } attrs { promote: true } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: NON_VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: true } }
I20250623 01:58:53.717183 18314 raft_consensus.cc:738] T 62858b6460fa4faa8c4bb9a06c000863 P e3ec3497126848969a0372dcacc06c59 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: e3ec3497126848969a0372dcacc06c59, State: Initialized, Role: LEARNER
I20250623 01:58:53.717780 18316 tablet_bootstrap.cc:492] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 01:58:53.717734 18314 consensus_queue.cc:260] T 62858b6460fa4faa8c4bb9a06c000863 P e3ec3497126848969a0372dcacc06c59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } } peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: NON_VOTER last_known_addr { host: "127.14.43.130" port: 44383 } attrs { promote: true } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: NON_VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: true } }
I20250623 01:58:53.718608 18316 tablet_bootstrap.cc:492] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed: Bootstrap complete.
I20250623 01:58:53.719182 18316 ts_tablet_manager.cc:1397] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed: Time spent bootstrapping tablet: real 0.095s user 0.082s sys 0.012s
I20250623 01:58:53.719640 18314 ts_tablet_manager.cc:1428] T 62858b6460fa4faa8c4bb9a06c000863 P e3ec3497126848969a0372dcacc06c59: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250623 01:58:53.721271 18273 tablet_copy_service.cc:342] P 41149896fc354b3dacb5d1b32f6788f7: Request end of tablet copy session e3ec3497126848969a0372dcacc06c59-62858b6460fa4faa8c4bb9a06c000863 received from {username='slave'} at 127.14.43.130:33093
I20250623 01:58:53.721688 18273 tablet_copy_service.cc:434] P 41149896fc354b3dacb5d1b32f6788f7: ending tablet copy session e3ec3497126848969a0372dcacc06c59-62858b6460fa4faa8c4bb9a06c000863 on tablet 62858b6460fa4faa8c4bb9a06c000863 with peer e3ec3497126848969a0372dcacc06c59
I20250623 01:58:53.721307 18316 raft_consensus.cc:357] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } } peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: NON_VOTER last_known_addr { host: "127.14.43.130" port: 44383 } attrs { promote: true } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: NON_VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: true } }
I20250623 01:58:53.721935 18316 raft_consensus.cc:738] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 419f59c561fa4aa8ab1a5a311af7a9ed, State: Initialized, Role: LEARNER
I20250623 01:58:53.722436 18316 consensus_queue.cc:260] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } } peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: NON_VOTER last_known_addr { host: "127.14.43.130" port: 44383 } attrs { promote: true } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: NON_VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: true } }
I20250623 01:58:53.728602 18316 ts_tablet_manager.cc:1428] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed: Time spent starting tablet: real 0.009s user 0.004s sys 0.003s
I20250623 01:58:53.730158 18273 tablet_copy_service.cc:342] P 41149896fc354b3dacb5d1b32f6788f7: Request end of tablet copy session 419f59c561fa4aa8ab1a5a311af7a9ed-62858b6460fa4faa8c4bb9a06c000863 received from {username='slave'} at 127.14.43.129:55435
I20250623 01:58:53.730520 18273 tablet_copy_service.cc:434] P 41149896fc354b3dacb5d1b32f6788f7: ending tablet copy session 419f59c561fa4aa8ab1a5a311af7a9ed-62858b6460fa4faa8c4bb9a06c000863 on tablet 62858b6460fa4faa8c4bb9a06c000863 with peer 419f59c561fa4aa8ab1a5a311af7a9ed
I20250623 01:58:53.803529 18321 tablet_bootstrap.cc:492] T 8ae52d3edd7748be9366f4baea5b1472 P 41149896fc354b3dacb5d1b32f6788f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 01:58:53.804152 18321 tablet_bootstrap.cc:492] T 8ae52d3edd7748be9366f4baea5b1472 P 41149896fc354b3dacb5d1b32f6788f7: Bootstrap complete.
I20250623 01:58:53.804576 18321 ts_tablet_manager.cc:1397] T 8ae52d3edd7748be9366f4baea5b1472 P 41149896fc354b3dacb5d1b32f6788f7: Time spent bootstrapping tablet: real 0.104s user 0.088s sys 0.012s
I20250623 01:58:53.806282 18321 raft_consensus.cc:357] T 8ae52d3edd7748be9366f4baea5b1472 P 41149896fc354b3dacb5d1b32f6788f7 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } } peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: NON_VOTER last_known_addr { host: "127.14.43.131" port: 45611 } attrs { promote: true } }
I20250623 01:58:53.806713 18321 raft_consensus.cc:738] T 8ae52d3edd7748be9366f4baea5b1472 P 41149896fc354b3dacb5d1b32f6788f7 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 41149896fc354b3dacb5d1b32f6788f7, State: Initialized, Role: LEARNER
I20250623 01:58:53.807065 18321 consensus_queue.cc:260] T 8ae52d3edd7748be9366f4baea5b1472 P 41149896fc354b3dacb5d1b32f6788f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } } peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: NON_VOTER last_known_addr { host: "127.14.43.131" port: 45611 } attrs { promote: true } }
I20250623 01:58:53.808251 18321 ts_tablet_manager.cc:1428] T 8ae52d3edd7748be9366f4baea5b1472 P 41149896fc354b3dacb5d1b32f6788f7: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20250623 01:58:53.809536 18108 tablet_copy_service.cc:342] P e3ec3497126848969a0372dcacc06c59: Request end of tablet copy session 41149896fc354b3dacb5d1b32f6788f7-8ae52d3edd7748be9366f4baea5b1472 received from {username='slave'} at 127.14.43.131:53245
I20250623 01:58:53.809839 18108 tablet_copy_service.cc:434] P e3ec3497126848969a0372dcacc06c59: ending tablet copy session 41149896fc354b3dacb5d1b32f6788f7-8ae52d3edd7748be9366f4baea5b1472 on tablet 8ae52d3edd7748be9366f4baea5b1472 with peer 41149896fc354b3dacb5d1b32f6788f7
I20250623 01:58:54.048552 18253 raft_consensus.cc:1215] T 8ae52d3edd7748be9366f4baea5b1472 P 41149896fc354b3dacb5d1b32f6788f7 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
I20250623 01:58:54.059032 14510 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 419f59c561fa4aa8ab1a5a311af7a9ed to finish bootstrapping
I20250623 01:58:54.072454 14510 ts_itest-base.cc:246] Waiting for 1 tablets on tserver e3ec3497126848969a0372dcacc06c59 to finish bootstrapping
I20250623 01:58:54.083858 14510 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 41149896fc354b3dacb5d1b32f6788f7 to finish bootstrapping
I20250623 01:58:54.155440 18088 raft_consensus.cc:1215] T 62858b6460fa4faa8c4bb9a06c000863 P e3ec3497126848969a0372dcacc06c59 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20250623 01:58:54.196188 17944 raft_consensus.cc:1215] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20250623 01:58:54.333650 18226 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250623 01:58:54.340024 17923 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250623 01:58:54.342208 18068 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250623 01:58:54.434281 18327 raft_consensus.cc:1062] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59: attempting to promote NON_VOTER 41149896fc354b3dacb5d1b32f6788f7 to VOTER
I20250623 01:58:54.436192 18327 consensus_queue.cc:237] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } } peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } attrs { promote: false } }
I20250623 01:58:54.448879 18253 raft_consensus.cc:1273] T 8ae52d3edd7748be9366f4baea5b1472 P 41149896fc354b3dacb5d1b32f6788f7 [term 2 LEARNER]: Refusing update from remote peer e3ec3497126848969a0372dcacc06c59: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250623 01:58:54.451843 17944 raft_consensus.cc:1273] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 FOLLOWER]: Refusing update from remote peer e3ec3497126848969a0372dcacc06c59: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250623 01:58:54.454147 18217 consensus_queue.cc:1035] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250623 01:58:54.455744 18217 consensus_queue.cc:1035] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250623 01:58:54.471889 18326 raft_consensus.cc:2953] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 41149896fc354b3dacb5d1b32f6788f7 (127.14.43.131) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } } peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } attrs { promote: false } } }
I20250623 01:58:54.473589 17943 raft_consensus.cc:2953] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 41149896fc354b3dacb5d1b32f6788f7 (127.14.43.131) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } } peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } attrs { promote: false } } }
I20250623 01:58:54.486693 17801 catalog_manager.cc:5582] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 reported cstate change: config changed from index 12 to 13, 41149896fc354b3dacb5d1b32f6788f7 (127.14.43.131) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "e3ec3497126848969a0372dcacc06c59" committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250623 01:58:54.517241 18253 raft_consensus.cc:2953] T 8ae52d3edd7748be9366f4baea5b1472 P 41149896fc354b3dacb5d1b32f6788f7 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 41149896fc354b3dacb5d1b32f6788f7 (127.14.43.131) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } } peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } attrs { promote: false } } }
I20250623 01:58:54.538623 18339 raft_consensus.cc:1062] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7: attempting to promote NON_VOTER e3ec3497126848969a0372dcacc06c59 to VOTER
I20250623 01:58:54.540597 18339 consensus_queue.cc:237] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } } peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } attrs { promote: false } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: NON_VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: true } }
I20250623 01:58:54.560914 17943 raft_consensus.cc:1273] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 LEARNER]: Refusing update from remote peer 41149896fc354b3dacb5d1b32f6788f7: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250623 01:58:54.563489 18088 raft_consensus.cc:1273] T 62858b6460fa4faa8c4bb9a06c000863 P e3ec3497126848969a0372dcacc06c59 [term 2 LEARNER]: Refusing update from remote peer 41149896fc354b3dacb5d1b32f6788f7: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250623 01:58:54.564937 18282 consensus_queue.cc:1035] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: NON_VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250623 01:58:54.566427 18282 consensus_queue.cc:1035] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250623 01:58:54.581681 18339 raft_consensus.cc:1025] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [term 2 LEADER]: attempt to promote peer 419f59c561fa4aa8ab1a5a311af7a9ed: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250623 01:58:54.587252 18338 raft_consensus.cc:2953] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, e3ec3497126848969a0372dcacc06c59 (127.14.43.130) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } } peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } attrs { promote: false } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: NON_VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: true } } }
I20250623 01:58:54.588418 18088 raft_consensus.cc:2953] T 62858b6460fa4faa8c4bb9a06c000863 P e3ec3497126848969a0372dcacc06c59 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, e3ec3497126848969a0372dcacc06c59 (127.14.43.130) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } } peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } attrs { promote: false } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: NON_VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: true } } }
I20250623 01:58:54.597797 17943 raft_consensus.cc:2953] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 LEARNER]: Committing config change with OpId 2.10: config changed from index 9 to 10, e3ec3497126848969a0372dcacc06c59 (127.14.43.130) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } } peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } attrs { promote: false } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: NON_VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: true } } }
I20250623 01:58:54.598564 17802 catalog_manager.cc:5582] T 62858b6460fa4faa8c4bb9a06c000863 P e3ec3497126848969a0372dcacc06c59 reported cstate change: config changed from index 9 to 10, e3ec3497126848969a0372dcacc06c59 (127.14.43.130) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "41149896fc354b3dacb5d1b32f6788f7" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } } peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } attrs { promote: false } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: NON_VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: true } } }
I20250623 01:58:54.607163 18339 raft_consensus.cc:1062] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7: attempting to promote NON_VOTER 419f59c561fa4aa8ab1a5a311af7a9ed to VOTER
I20250623 01:58:54.609032 18339 consensus_queue.cc:237] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } } peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } attrs { promote: false } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } }
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
99a1ba904ecf487b894f3609083cce41 | 127.14.43.190:33475 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.14.43.148:43657 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
I20250623 01:58:54.613390 17943 raft_consensus.cc:1273] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 LEARNER]: Refusing update from remote peer 41149896fc354b3dacb5d1b32f6788f7: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
41149896fc354b3dacb5d1b32f6788f7 | 127.14.43.131:45611 | HEALTHY | <none> | 1 | 0
419f59c561fa4aa8ab1a5a311af7a9ed | 127.14.43.129:44313 | HEALTHY | <none> | 1 | 0
e3ec3497126848969a0372dcacc06c59 | 127.14.43.130:44383 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
I20250623 01:58:54.614404 18282 consensus_queue.cc:1035] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.14.43.129 | experimental | 127.14.43.129:44313
I20250623 01:58:54.616451 18088 raft_consensus.cc:1273] T 62858b6460fa4faa8c4bb9a06c000863 P e3ec3497126848969a0372dcacc06c59 [term 2 FOLLOWER]: Refusing update from remote peer 41149896fc354b3dacb5d1b32f6788f7: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
local_ip_for_outbound_sockets | 127.14.43.130 | experimental | 127.14.43.130:44383
local_ip_for_outbound_sockets | 127.14.43.131 | experimental | 127.14.43.131:45611
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.14.43.129:44313
server_dump_info_path | /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.14.43.130:44383
server_dump_info_path | /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.14.43.131:45611
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.14.43.148:43657 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
I20250623 01:58:54.617651 18282 consensus_queue.cc:1035] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable1 | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable2 | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 3
Tablets | 3
Replicas | 7
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250623 01:58:54.621994 18338 raft_consensus.cc:2953] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 419f59c561fa4aa8ab1a5a311af7a9ed (127.14.43.129) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } } peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } attrs { promote: false } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } } }
I20250623 01:58:54.623392 17943 raft_consensus.cc:2953] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 419f59c561fa4aa8ab1a5a311af7a9ed (127.14.43.129) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } } peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } attrs { promote: false } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } } }
I20250623 01:58:54.625643 18088 raft_consensus.cc:2953] T 62858b6460fa4faa8c4bb9a06c000863 P e3ec3497126848969a0372dcacc06c59 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 419f59c561fa4aa8ab1a5a311af7a9ed (127.14.43.129) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } } peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } attrs { promote: false } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } } }
I20250623 01:58:54.635417 17801 catalog_manager.cc:5582] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed reported cstate change: config changed from index 10 to 11, 419f59c561fa4aa8ab1a5a311af7a9ed (127.14.43.129) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "41149896fc354b3dacb5d1b32f6788f7" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } } peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } attrs { promote: false } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } } }
I20250623 01:58:54.636953 14510 log_verifier.cc:126] Checking tablet 62858b6460fa4faa8c4bb9a06c000863
I20250623 01:58:54.731132 14510 log_verifier.cc:177] Verified matching terms for 11 ops in tablet 62858b6460fa4faa8c4bb9a06c000863
I20250623 01:58:54.731539 14510 log_verifier.cc:126] Checking tablet 8ae52d3edd7748be9366f4baea5b1472
I20250623 01:58:54.815886 14510 log_verifier.cc:177] Verified matching terms for 13 ops in tablet 8ae52d3edd7748be9366f4baea5b1472
I20250623 01:58:54.816128 14510 log_verifier.cc:126] Checking tablet d1cb2e476ce046cbac6fbd6faae2a046
I20250623 01:58:54.839946 14510 log_verifier.cc:177] Verified matching terms for 7 ops in tablet d1cb2e476ce046cbac6fbd6faae2a046
I20250623 01:58:54.840302 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 17769
I20250623 01:58:54.863647 14510 minidump.cc:252] Setting minidump size limit to 20M
I20250623 01:58:54.864892 14510 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:58:54.865895 14510 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:58:54.875566 18370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:58:54.875587 18369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:58:55.040879 14510 server_base.cc:1048] running on GCE node
W20250623 01:58:55.040868 18372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:58:55.042503 14510 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250623 01:58:55.042790 14510 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250623 01:58:55.042984 14510 hybrid_clock.cc:648] HybridClock initialized: now 1750643935042964 us; error 0 us; skew 500 ppm
I20250623 01:58:55.043808 14510 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:58:55.047293 14510 webserver.cc:469] Webserver started at http://0.0.0.0:38251/ using document root <none> and password file <none>
I20250623 01:58:55.048357 14510 fs_manager.cc:362] Metadata directory not provided
I20250623 01:58:55.048607 14510 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:58:55.055301 14510 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20250623 01:58:55.059779 18377 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:58:55.060801 14510 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250623 01:58:55.061183 14510 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
uuid: "99a1ba904ecf487b894f3609083cce41"
format_stamp: "Formatted at 2025-06-23 01:58:34 on dist-test-slave-9rpg"
I20250623 01:58:55.063512 14510 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:58:55.089702 14510 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:58:55.091154 14510 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:58:55.091552 14510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:58:55.099783 14510 sys_catalog.cc:263] Verifying existing consensus state
W20250623 01:58:55.103121 14510 sys_catalog.cc:243] For a single master config, on-disk Raft master: 127.14.43.190:33475 exists but no master address supplied!
I20250623 01:58:55.104993 14510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41: Bootstrap starting.
I20250623 01:58:55.147629 14510 log.cc:826] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41: Log is configured to *not* fsync() on all Append() calls
I20250623 01:58:55.212394 14510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41: Bootstrap replayed 1/1 log segments. Stats: ops{read=31 overwritten=0 applied=31 ignored=0} inserts{seen=13 ignored=0} mutations{seen=22 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 01:58:55.213155 14510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41: Bootstrap complete.
I20250623 01:58:55.226258 14510 raft_consensus.cc:357] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a1ba904ecf487b894f3609083cce41" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 33475 } }
I20250623 01:58:55.226836 14510 raft_consensus.cc:738] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a1ba904ecf487b894f3609083cce41, State: Initialized, Role: FOLLOWER
I20250623 01:58:55.227578 14510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a1ba904ecf487b894f3609083cce41" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 33475 } }
I20250623 01:58:55.228034 14510 raft_consensus.cc:397] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [term 3 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250623 01:58:55.228283 14510 raft_consensus.cc:491] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [term 3 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250623 01:58:55.228586 14510 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [term 3 FOLLOWER]: Advancing to term 4
I20250623 01:58:55.234176 14510 raft_consensus.cc:513] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a1ba904ecf487b894f3609083cce41" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 33475 } }
I20250623 01:58:55.234860 14510 leader_election.cc:304] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99a1ba904ecf487b894f3609083cce41; no voters:
I20250623 01:58:55.235913 14510 leader_election.cc:290] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [CANDIDATE]: Term 4 election: Requested vote from peers
I20250623 01:58:55.236192 18384 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 01:58:55.237402 18384 raft_consensus.cc:695] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [term 4 LEADER]: Becoming Leader. State: Replica: 99a1ba904ecf487b894f3609083cce41, State: Running, Role: LEADER
I20250623 01:58:55.238215 18384 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 4, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a1ba904ecf487b894f3609083cce41" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 33475 } }
I20250623 01:58:55.244084 18385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 leader_uuid: "99a1ba904ecf487b894f3609083cce41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a1ba904ecf487b894f3609083cce41" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 33475 } } }
I20250623 01:58:55.244529 18385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [sys.catalog]: This master's current role is: LEADER
I20250623 01:58:55.245450 18386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99a1ba904ecf487b894f3609083cce41. Latest consensus state: current_term: 4 leader_uuid: "99a1ba904ecf487b894f3609083cce41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a1ba904ecf487b894f3609083cce41" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 33475 } } }
I20250623 01:58:55.245872 18386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [sys.catalog]: This master's current role is: LEADER
I20250623 01:58:55.271142 14510 tablet_replica.cc:331] stopping tablet replica
I20250623 01:58:55.271726 14510 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [term 4 LEADER]: Raft consensus shutting down.
I20250623 01:58:55.272143 14510 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250623 01:58:55.274266 14510 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250623 01:58:55.274725 14510 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250623 01:58:55.299546 14510 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
W20250623 01:58:55.661616 18133 heartbeater.cc:646] Failed to heartbeat to 127.14.43.190:33475 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.43.190:33475: connect: Connection refused (error 111)
W20250623 01:58:55.663872 18303 heartbeater.cc:646] Failed to heartbeat to 127.14.43.190:33475 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.43.190:33475: connect: Connection refused (error 111)
W20250623 01:58:55.666319 17989 heartbeater.cc:646] Failed to heartbeat to 127.14.43.190:33475 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.43.190:33475: connect: Connection refused (error 111)
I20250623 01:59:00.274186 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 17839
I20250623 01:59:00.295015 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 17993
I20250623 01:59:00.318079 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 18140
I20250623 01:59:00.340941 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.43.190:33475
--webserver_interface=127.14.43.190
--webserver_port=40887
--builtin_ntp_servers=127.14.43.148:43657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.43.190:33475 with env {}
W20250623 01:59:00.633414 18461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:59:00.633988 18461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:59:00.634471 18461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:59:00.665112 18461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250623 01:59:00.665407 18461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:59:00.665658 18461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250623 01:59:00.665887 18461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250623 01:59:00.700172 18461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:43657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.43.190:33475
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.43.190:33475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.14.43.190
--webserver_port=40887
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:59:00.701475 18461 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:59:00.703109 18461 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:59:00.718726 18467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:59:00.718739 18470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:59:00.718801 18468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:59:01.800973 18469 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250623 01:59:01.801108 18461 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250623 01:59:01.804783 18461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:59:01.807932 18461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:59:01.809378 18461 hybrid_clock.cc:648] HybridClock initialized: now 1750643941809338 us; error 62 us; skew 500 ppm
I20250623 01:59:01.810221 18461 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:59:01.819429 18461 webserver.cc:469] Webserver started at http://127.14.43.190:40887/ using document root <none> and password file <none>
I20250623 01:59:01.820369 18461 fs_manager.cc:362] Metadata directory not provided
I20250623 01:59:01.820595 18461 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:59:01.828444 18461 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20250623 01:59:01.832898 18477 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:59:01.833998 18461 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250623 01:59:01.834362 18461 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
uuid: "99a1ba904ecf487b894f3609083cce41"
format_stamp: "Formatted at 2025-06-23 01:58:34 on dist-test-slave-9rpg"
I20250623 01:59:01.836302 18461 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:59:01.881413 18461 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:59:01.882902 18461 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:59:01.883376 18461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:59:01.951403 18461 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.190:33475
I20250623 01:59:01.951478 18528 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.190:33475 every 8 connection(s)
I20250623 01:59:01.954226 18461 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250623 01:59:01.956266 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 18461
I20250623 01:59:01.958115 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.43.129:44313
--local_ip_for_outbound_sockets=127.14.43.129
--tserver_master_addrs=127.14.43.190:33475
--webserver_port=37409
--webserver_interface=127.14.43.129
--builtin_ntp_servers=127.14.43.148:43657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250623 01:59:01.964469 18529 sys_catalog.cc:263] Verifying existing consensus state
I20250623 01:59:01.969215 18529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41: Bootstrap starting.
I20250623 01:59:01.978994 18529 log.cc:826] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41: Log is configured to *not* fsync() on all Append() calls
I20250623 01:59:02.057461 18529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41: Bootstrap replayed 1/1 log segments. Stats: ops{read=35 overwritten=0 applied=35 ignored=0} inserts{seen=15 ignored=0} mutations{seen=24 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 01:59:02.058270 18529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41: Bootstrap complete.
I20250623 01:59:02.076856 18529 raft_consensus.cc:357] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a1ba904ecf487b894f3609083cce41" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 33475 } }
I20250623 01:59:02.078931 18529 raft_consensus.cc:738] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a1ba904ecf487b894f3609083cce41, State: Initialized, Role: FOLLOWER
I20250623 01:59:02.079752 18529 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 35, Last appended: 5.35, Last appended by leader: 35, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a1ba904ecf487b894f3609083cce41" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 33475 } }
I20250623 01:59:02.080246 18529 raft_consensus.cc:397] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [term 5 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250623 01:59:02.080500 18529 raft_consensus.cc:491] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [term 5 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250623 01:59:02.080765 18529 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [term 5 FOLLOWER]: Advancing to term 6
I20250623 01:59:02.085719 18529 raft_consensus.cc:513] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a1ba904ecf487b894f3609083cce41" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 33475 } }
I20250623 01:59:02.086402 18529 leader_election.cc:304] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99a1ba904ecf487b894f3609083cce41; no voters:
I20250623 01:59:02.088335 18529 leader_election.cc:290] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [CANDIDATE]: Term 6 election: Requested vote from peers
I20250623 01:59:02.088795 18533 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [term 6 FOLLOWER]: Leader election won for term 6
I20250623 01:59:02.091673 18533 raft_consensus.cc:695] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [term 6 LEADER]: Becoming Leader. State: Replica: 99a1ba904ecf487b894f3609083cce41, State: Running, Role: LEADER
I20250623 01:59:02.092531 18533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 5.35, Last appended by leader: 35, Current term: 6, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a1ba904ecf487b894f3609083cce41" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 33475 } }
I20250623 01:59:02.092887 18529 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [sys.catalog]: configured and running, proceeding with master startup.
I20250623 01:59:02.102777 18534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 leader_uuid: "99a1ba904ecf487b894f3609083cce41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a1ba904ecf487b894f3609083cce41" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 33475 } } }
I20250623 01:59:02.103454 18534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [sys.catalog]: This master's current role is: LEADER
I20250623 01:59:02.105752 18535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99a1ba904ecf487b894f3609083cce41. Latest consensus state: current_term: 6 leader_uuid: "99a1ba904ecf487b894f3609083cce41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a1ba904ecf487b894f3609083cce41" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 33475 } } }
I20250623 01:59:02.107189 18535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41 [sys.catalog]: This master's current role is: LEADER
I20250623 01:59:02.112838 18540 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250623 01:59:02.125569 18540 catalog_manager.cc:671] Loaded metadata for table TestTable [id=44d246c7602b478183f2b259f067a7c7]
I20250623 01:59:02.127435 18540 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=a9b73e9cdc214b82bd52a7b58d2e5e5f]
I20250623 01:59:02.129158 18540 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=aaa2b1d5e1b343f2b9def29de1bc6d7c]
I20250623 01:59:02.137591 18540 tablet_loader.cc:96] loaded metadata for tablet 62858b6460fa4faa8c4bb9a06c000863 (table TestTable1 [id=aaa2b1d5e1b343f2b9def29de1bc6d7c])
I20250623 01:59:02.140235 18540 tablet_loader.cc:96] loaded metadata for tablet 8ae52d3edd7748be9366f4baea5b1472 (table TestTable [id=44d246c7602b478183f2b259f067a7c7])
I20250623 01:59:02.141618 18540 tablet_loader.cc:96] loaded metadata for tablet d1cb2e476ce046cbac6fbd6faae2a046 (table TestTable2 [id=a9b73e9cdc214b82bd52a7b58d2e5e5f])
I20250623 01:59:02.143114 18540 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250623 01:59:02.148897 18540 catalog_manager.cc:1261] Loaded cluster ID: 820e7a8f9f4b48d3a79721cb8afe844e
I20250623 01:59:02.149276 18540 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250623 01:59:02.157963 18540 catalog_manager.cc:1506] Loading token signing keys...
I20250623 01:59:02.162423 18550 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250623 01:59:02.163741 18540 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 99a1ba904ecf487b894f3609083cce41: Loaded TSK: 0
I20250623 01:59:02.164899 18540 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250623 01:59:02.288678 18531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:59:02.289192 18531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:59:02.289700 18531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:59:02.321272 18531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:59:02.322228 18531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.43.129
I20250623 01:59:02.356825 18531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:43657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.43.129:44313
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.14.43.129
--webserver_port=37409
--tserver_master_addrs=127.14.43.190:33475
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.43.129
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:59:02.358180 18531 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:59:02.359859 18531 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:59:02.376603 18559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:59:02.378309 18556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:59:02.376605 18557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:59:03.480893 18558 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1101 milliseconds
I20250623 01:59:03.481009 18531 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250623 01:59:03.482306 18531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:59:03.484859 18531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:59:03.486296 18531 hybrid_clock.cc:648] HybridClock initialized: now 1750643943486240 us; error 89 us; skew 500 ppm
I20250623 01:59:03.487109 18531 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:59:03.493847 18531 webserver.cc:469] Webserver started at http://127.14.43.129:37409/ using document root <none> and password file <none>
I20250623 01:59:03.495151 18531 fs_manager.cc:362] Metadata directory not provided
I20250623 01:59:03.495381 18531 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:59:03.503274 18531 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20250623 01:59:03.508280 18566 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:59:03.509390 18531 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250623 01:59:03.509702 18531 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "419f59c561fa4aa8ab1a5a311af7a9ed"
format_stamp: "Formatted at 2025-06-23 01:58:36 on dist-test-slave-9rpg"
I20250623 01:59:03.511713 18531 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:59:03.567989 18531 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:59:03.569478 18531 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:59:03.569902 18531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:59:03.572759 18531 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:59:03.579144 18573 ts_tablet_manager.cc:542] Loading tablet metadata (0/3 complete)
I20250623 01:59:03.594835 18531 ts_tablet_manager.cc:579] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20250623 01:59:03.595173 18531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.018s user 0.000s sys 0.003s
I20250623 01:59:03.595511 18531 ts_tablet_manager.cc:594] Registering tablets (0/3 complete)
I20250623 01:59:03.603528 18573 tablet_bootstrap.cc:492] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed: Bootstrap starting.
I20250623 01:59:03.613814 18531 ts_tablet_manager.cc:610] Registered 3 tablets
I20250623 01:59:03.614133 18531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.019s user 0.018s sys 0.001s
I20250623 01:59:03.662421 18573 log.cc:826] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed: Log is configured to *not* fsync() on all Append() calls
I20250623 01:59:03.789227 18573 tablet_bootstrap.cc:492] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 01:59:03.790632 18573 tablet_bootstrap.cc:492] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed: Bootstrap complete.
I20250623 01:59:03.792418 18573 ts_tablet_manager.cc:1397] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed: Time spent bootstrapping tablet: real 0.189s user 0.159s sys 0.025s
I20250623 01:59:03.815012 18573 raft_consensus.cc:357] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } } peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } attrs { promote: false } }
I20250623 01:59:03.818557 18573 raft_consensus.cc:738] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 419f59c561fa4aa8ab1a5a311af7a9ed, State: Initialized, Role: FOLLOWER
I20250623 01:59:03.819837 18573 consensus_queue.cc:260] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } } peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } attrs { promote: false } }
I20250623 01:59:03.825031 18573 ts_tablet_manager.cc:1428] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed: Time spent starting tablet: real 0.032s user 0.023s sys 0.006s
I20250623 01:59:03.825906 18573 tablet_bootstrap.cc:492] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed: Bootstrap starting.
I20250623 01:59:03.829994 18531 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.129:44313
I20250623 01:59:03.830348 18682 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.129:44313 every 8 connection(s)
I20250623 01:59:03.832767 18531 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250623 01:59:03.841339 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 18531
I20250623 01:59:03.843578 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.43.130:44383
--local_ip_for_outbound_sockets=127.14.43.130
--tserver_master_addrs=127.14.43.190:33475
--webserver_port=33091
--webserver_interface=127.14.43.130
--builtin_ntp_servers=127.14.43.148:43657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250623 01:59:03.875367 18683 heartbeater.cc:344] Connected to a master server at 127.14.43.190:33475
I20250623 01:59:03.875856 18683 heartbeater.cc:461] Registering TS with master...
I20250623 01:59:03.877110 18683 heartbeater.cc:507] Master 127.14.43.190:33475 requested a full tablet report, sending...
I20250623 01:59:03.883296 18494 ts_manager.cc:194] Registered new tserver with Master: 419f59c561fa4aa8ab1a5a311af7a9ed (127.14.43.129:44313)
I20250623 01:59:03.888576 18494 catalog_manager.cc:5582] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed reported cstate change: config changed from index -1 to 13, term changed from 0 to 2, VOTER 41149896fc354b3dacb5d1b32f6788f7 (127.14.43.131) added, VOTER 419f59c561fa4aa8ab1a5a311af7a9ed (127.14.43.129) added, VOTER e3ec3497126848969a0372dcacc06c59 (127.14.43.130) added. New cstate: current_term: 2 committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } } peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } attrs { promote: false } } }
I20250623 01:59:03.947628 18494 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.43.129:51999
I20250623 01:59:03.980777 18573 tablet_bootstrap.cc:492] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 01:59:03.981786 18573 tablet_bootstrap.cc:492] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed: Bootstrap complete.
I20250623 01:59:03.983343 18573 ts_tablet_manager.cc:1397] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed: Time spent bootstrapping tablet: real 0.158s user 0.122s sys 0.016s
I20250623 01:59:03.985646 18573 raft_consensus.cc:357] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } } peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } attrs { promote: false } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } }
I20250623 01:59:03.986382 18573 raft_consensus.cc:738] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 419f59c561fa4aa8ab1a5a311af7a9ed, State: Initialized, Role: FOLLOWER
I20250623 01:59:03.987069 18573 consensus_queue.cc:260] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } } peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } attrs { promote: false } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } }
I20250623 01:59:03.988274 18683 heartbeater.cc:499] Master 127.14.43.190:33475 was elected leader, sending a full tablet report...
I20250623 01:59:03.999372 18573 ts_tablet_manager.cc:1428] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed: Time spent starting tablet: real 0.016s user 0.004s sys 0.000s
I20250623 01:59:04.005847 18573 tablet_bootstrap.cc:492] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed: Bootstrap starting.
I20250623 01:59:04.085865 18573 tablet_bootstrap.cc:492] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 01:59:04.086583 18573 tablet_bootstrap.cc:492] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed: Bootstrap complete.
I20250623 01:59:04.087589 18573 ts_tablet_manager.cc:1397] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed: Time spent bootstrapping tablet: real 0.082s user 0.069s sys 0.012s
I20250623 01:59:04.089046 18573 raft_consensus.cc:357] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } }
I20250623 01:59:04.089407 18573 raft_consensus.cc:738] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 419f59c561fa4aa8ab1a5a311af7a9ed, State: Initialized, Role: FOLLOWER
I20250623 01:59:04.089886 18573 consensus_queue.cc:260] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } }
I20250623 01:59:04.090327 18573 raft_consensus.cc:397] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250623 01:59:04.090605 18573 raft_consensus.cc:491] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250623 01:59:04.090896 18573 raft_consensus.cc:3058] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:59:04.096084 18573 raft_consensus.cc:513] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } }
I20250623 01:59:04.096725 18573 leader_election.cc:304] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 419f59c561fa4aa8ab1a5a311af7a9ed; no voters:
I20250623 01:59:04.097261 18573 leader_election.cc:290] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed [CANDIDATE]: Term 3 election: Requested vote from peers
I20250623 01:59:04.097491 18678 raft_consensus.cc:2802] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:59:04.105759 18678 raft_consensus.cc:695] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 3 LEADER]: Becoming Leader. State: Replica: 419f59c561fa4aa8ab1a5a311af7a9ed, State: Running, Role: LEADER
I20250623 01:59:04.106753 18678 consensus_queue.cc:237] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } }
I20250623 01:59:04.111572 18573 ts_tablet_manager.cc:1428] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed: Time spent starting tablet: real 0.024s user 0.014s sys 0.008s
I20250623 01:59:04.124862 18494 catalog_manager.cc:5582] T d1cb2e476ce046cbac6fbd6faae2a046 P 419f59c561fa4aa8ab1a5a311af7a9ed reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } health_report { overall_health: HEALTHY } } }
W20250623 01:59:04.237609 18687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:59:04.238162 18687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:59:04.238659 18687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:59:04.270141 18687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:59:04.271060 18687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.43.130
I20250623 01:59:04.305809 18687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:43657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.43.130:44383
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.14.43.130
--webserver_port=33091
--tserver_master_addrs=127.14.43.190:33475
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.43.130
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:59:04.307124 18687 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:59:04.308728 18687 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:59:04.330045 18704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:59:05.031584 18709 raft_consensus.cc:491] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:05.031963 18709 raft_consensus.cc:513] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } } peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } attrs { promote: false } }
I20250623 01:59:05.034039 18709 leader_election.cc:290] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e3ec3497126848969a0372dcacc06c59 (127.14.43.130:44383), 41149896fc354b3dacb5d1b32f6788f7 (127.14.43.131:45611)
W20250623 01:59:05.037762 18568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.43.131:45611: connect: Connection refused (error 111)
W20250623 01:59:05.041543 18569 leader_election.cc:336] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e3ec3497126848969a0372dcacc06c59 (127.14.43.130:44383): Network error: Client connection negotiation failed: client connection to 127.14.43.130:44383: connect: Connection refused (error 111)
W20250623 01:59:05.042093 18568 leader_election.cc:336] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 41149896fc354b3dacb5d1b32f6788f7 (127.14.43.131:45611): Network error: Client connection negotiation failed: client connection to 127.14.43.131:45611: connect: Connection refused (error 111)
I20250623 01:59:05.042367 18568 leader_election.cc:304] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 419f59c561fa4aa8ab1a5a311af7a9ed; no voters: 41149896fc354b3dacb5d1b32f6788f7, e3ec3497126848969a0372dcacc06c59
I20250623 01:59:05.042958 18709 raft_consensus.cc:2747] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250623 01:59:04.330586 18703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:59:04.333194 18687 server_base.cc:1048] running on GCE node
W20250623 01:59:04.333463 18706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:59:05.422784 18687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:59:05.424970 18687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:59:05.426344 18687 hybrid_clock.cc:648] HybridClock initialized: now 1750643945426312 us; error 49 us; skew 500 ppm
I20250623 01:59:05.427166 18687 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:59:05.433107 18687 webserver.cc:469] Webserver started at http://127.14.43.130:33091/ using document root <none> and password file <none>
I20250623 01:59:05.434063 18687 fs_manager.cc:362] Metadata directory not provided
I20250623 01:59:05.434296 18687 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:59:05.442042 18687 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20250623 01:59:05.443851 18709 raft_consensus.cc:491] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:05.444341 18709 raft_consensus.cc:513] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } } peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } attrs { promote: false } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } }
I20250623 01:59:05.445910 18709 leader_election.cc:290] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 41149896fc354b3dacb5d1b32f6788f7 (127.14.43.131:45611), e3ec3497126848969a0372dcacc06c59 (127.14.43.130:44383)
I20250623 01:59:05.447492 18717 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:59:05.448825 18687 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
W20250623 01:59:05.449227 18568 leader_election.cc:336] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 41149896fc354b3dacb5d1b32f6788f7 (127.14.43.131:45611): Network error: Client connection negotiation failed: client connection to 127.14.43.131:45611: connect: Connection refused (error 111)
W20250623 01:59:05.449985 18569 leader_election.cc:336] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e3ec3497126848969a0372dcacc06c59 (127.14.43.130:44383): Network error: Client connection negotiation failed: client connection to 127.14.43.130:44383: connect: Connection refused (error 111)
I20250623 01:59:05.450328 18569 leader_election.cc:304] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 419f59c561fa4aa8ab1a5a311af7a9ed; no voters: 41149896fc354b3dacb5d1b32f6788f7, e3ec3497126848969a0372dcacc06c59
I20250623 01:59:05.450868 18709 raft_consensus.cc:2747] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 01:59:05.449226 18687 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "e3ec3497126848969a0372dcacc06c59"
format_stamp: "Formatted at 2025-06-23 01:58:38 on dist-test-slave-9rpg"
I20250623 01:59:05.451630 18687 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:59:05.503255 18687 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:59:05.504762 18687 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:59:05.505191 18687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:59:05.507752 18687 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:59:05.513648 18724 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250623 01:59:05.525461 18687 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250623 01:59:05.525782 18687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.002s sys 0.001s
I20250623 01:59:05.526090 18687 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250623 01:59:05.531540 18724 tablet_bootstrap.cc:492] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59: Bootstrap starting.
I20250623 01:59:05.534147 18687 ts_tablet_manager.cc:610] Registered 2 tablets
I20250623 01:59:05.534348 18687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.008s user 0.008s sys 0.000s
I20250623 01:59:05.584717 18724 log.cc:826] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59: Log is configured to *not* fsync() on all Append() calls
I20250623 01:59:05.688205 18724 tablet_bootstrap.cc:492] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 01:59:05.688956 18724 tablet_bootstrap.cc:492] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59: Bootstrap complete.
I20250623 01:59:05.690451 18724 ts_tablet_manager.cc:1397] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59: Time spent bootstrapping tablet: real 0.159s user 0.122s sys 0.036s
I20250623 01:59:05.701210 18687 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.130:44383
I20250623 01:59:05.701345 18831 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.130:44383 every 8 connection(s)
I20250623 01:59:05.703876 18687 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250623 01:59:05.705346 18724 raft_consensus.cc:357] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } } peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } attrs { promote: false } }
I20250623 01:59:05.708382 18724 raft_consensus.cc:738] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3ec3497126848969a0372dcacc06c59, State: Initialized, Role: FOLLOWER
I20250623 01:59:05.709123 18724 consensus_queue.cc:260] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } } peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } attrs { promote: false } }
I20250623 01:59:05.712438 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 18687
I20250623 01:59:05.712961 18724 ts_tablet_manager.cc:1428] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59: Time spent starting tablet: real 0.022s user 0.020s sys 0.000s
I20250623 01:59:05.713810 18724 tablet_bootstrap.cc:492] T 62858b6460fa4faa8c4bb9a06c000863 P e3ec3497126848969a0372dcacc06c59: Bootstrap starting.
I20250623 01:59:05.714227 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.43.131:45611
--local_ip_for_outbound_sockets=127.14.43.131
--tserver_master_addrs=127.14.43.190:33475
--webserver_port=40355
--webserver_interface=127.14.43.131
--builtin_ntp_servers=127.14.43.148:43657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250623 01:59:05.724953 18832 heartbeater.cc:344] Connected to a master server at 127.14.43.190:33475
I20250623 01:59:05.725445 18832 heartbeater.cc:461] Registering TS with master...
I20250623 01:59:05.726809 18832 heartbeater.cc:507] Master 127.14.43.190:33475 requested a full tablet report, sending...
I20250623 01:59:05.731232 18494 ts_manager.cc:194] Registered new tserver with Master: e3ec3497126848969a0372dcacc06c59 (127.14.43.130:44383)
I20250623 01:59:05.735353 18494 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.43.130:51845
I20250623 01:59:05.740284 18832 heartbeater.cc:499] Master 127.14.43.190:33475 was elected leader, sending a full tablet report...
I20250623 01:59:05.806142 18724 tablet_bootstrap.cc:492] T 62858b6460fa4faa8c4bb9a06c000863 P e3ec3497126848969a0372dcacc06c59: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 01:59:05.806792 18724 tablet_bootstrap.cc:492] T 62858b6460fa4faa8c4bb9a06c000863 P e3ec3497126848969a0372dcacc06c59: Bootstrap complete.
I20250623 01:59:05.807754 18724 ts_tablet_manager.cc:1397] T 62858b6460fa4faa8c4bb9a06c000863 P e3ec3497126848969a0372dcacc06c59: Time spent bootstrapping tablet: real 0.094s user 0.080s sys 0.011s
I20250623 01:59:05.809387 18724 raft_consensus.cc:357] T 62858b6460fa4faa8c4bb9a06c000863 P e3ec3497126848969a0372dcacc06c59 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } } peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } attrs { promote: false } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } }
I20250623 01:59:05.809831 18724 raft_consensus.cc:738] T 62858b6460fa4faa8c4bb9a06c000863 P e3ec3497126848969a0372dcacc06c59 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3ec3497126848969a0372dcacc06c59, State: Initialized, Role: FOLLOWER
I20250623 01:59:05.810339 18724 consensus_queue.cc:260] T 62858b6460fa4faa8c4bb9a06c000863 P e3ec3497126848969a0372dcacc06c59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } } peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } attrs { promote: false } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } }
I20250623 01:59:05.812000 18724 ts_tablet_manager.cc:1428] T 62858b6460fa4faa8c4bb9a06c000863 P e3ec3497126848969a0372dcacc06c59: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20250623 01:59:06.025305 18837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:59:06.025818 18837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:59:06.026365 18837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:59:06.058145 18837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:59:06.058995 18837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.43.131
I20250623 01:59:06.094405 18837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:43657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.43.131:45611
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.14.43.131
--webserver_port=40355
--tserver_master_addrs=127.14.43.190:33475
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.43.131
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:59:06.095759 18837 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:59:06.097450 18837 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:59:06.112432 18844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:59:06.938275 18850 raft_consensus.cc:491] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:06.938696 18850 raft_consensus.cc:513] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } } peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } attrs { promote: false } }
I20250623 01:59:06.940220 18850 leader_election.cc:290] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e3ec3497126848969a0372dcacc06c59 (127.14.43.130:44383), 41149896fc354b3dacb5d1b32f6788f7 (127.14.43.131:45611)
I20250623 01:59:06.978643 18787 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ae52d3edd7748be9366f4baea5b1472" candidate_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "e3ec3497126848969a0372dcacc06c59" is_pre_election: true
I20250623 01:59:06.979363 18787 raft_consensus.cc:2466] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 419f59c561fa4aa8ab1a5a311af7a9ed in term 2.
I20250623 01:59:06.980741 18569 leader_election.cc:304] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 419f59c561fa4aa8ab1a5a311af7a9ed, e3ec3497126848969a0372dcacc06c59; no voters:
I20250623 01:59:06.981442 18850 raft_consensus.cc:2802] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250623 01:59:06.981299 18568 leader_election.cc:336] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 41149896fc354b3dacb5d1b32f6788f7 (127.14.43.131:45611): Network error: Client connection negotiation failed: client connection to 127.14.43.131:45611: connect: Connection refused (error 111)
I20250623 01:59:06.981717 18850 raft_consensus.cc:491] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:06.981967 18850 raft_consensus.cc:3058] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 FOLLOWER]: Advancing to term 3
I20250623 01:59:06.986280 18850 raft_consensus.cc:513] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 3 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } } peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } attrs { promote: false } }
I20250623 01:59:06.987668 18850 leader_election.cc:290] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [CANDIDATE]: Term 3 election: Requested vote from peers e3ec3497126848969a0372dcacc06c59 (127.14.43.130:44383), 41149896fc354b3dacb5d1b32f6788f7 (127.14.43.131:45611)
I20250623 01:59:06.988399 18787 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ae52d3edd7748be9366f4baea5b1472" candidate_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "e3ec3497126848969a0372dcacc06c59"
I20250623 01:59:06.988921 18787 raft_consensus.cc:3058] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [term 2 FOLLOWER]: Advancing to term 3
W20250623 01:59:06.991470 18568 leader_election.cc:336] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 41149896fc354b3dacb5d1b32f6788f7 (127.14.43.131:45611): Network error: Client connection negotiation failed: client connection to 127.14.43.131:45611: connect: Connection refused (error 111)
I20250623 01:59:06.995932 18787 raft_consensus.cc:2466] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 419f59c561fa4aa8ab1a5a311af7a9ed in term 3.
I20250623 01:59:06.996771 18569 leader_election.cc:304] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 419f59c561fa4aa8ab1a5a311af7a9ed, e3ec3497126848969a0372dcacc06c59; no voters: 41149896fc354b3dacb5d1b32f6788f7
I20250623 01:59:06.997356 18850 raft_consensus.cc:2802] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:59:06.998497 18850 raft_consensus.cc:695] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 3 LEADER]: Becoming Leader. State: Replica: 419f59c561fa4aa8ab1a5a311af7a9ed, State: Running, Role: LEADER
I20250623 01:59:06.999197 18850 consensus_queue.cc:237] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } } peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } attrs { promote: false } }
I20250623 01:59:07.006834 18494 catalog_manager.cc:5582] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed reported cstate change: term changed from 2 to 3, leader changed from <none> to 419f59c561fa4aa8ab1a5a311af7a9ed (127.14.43.129). New cstate: current_term: 3 leader_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:07.054738 18850 raft_consensus.cc:491] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:07.055120 18850 raft_consensus.cc:513] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } } peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } attrs { promote: false } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } }
I20250623 01:59:07.056502 18850 leader_election.cc:290] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 41149896fc354b3dacb5d1b32f6788f7 (127.14.43.131:45611), e3ec3497126848969a0372dcacc06c59 (127.14.43.130:44383)
I20250623 01:59:07.057480 18787 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62858b6460fa4faa8c4bb9a06c000863" candidate_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "e3ec3497126848969a0372dcacc06c59" is_pre_election: true
I20250623 01:59:07.057943 18787 raft_consensus.cc:2466] T 62858b6460fa4faa8c4bb9a06c000863 P e3ec3497126848969a0372dcacc06c59 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 419f59c561fa4aa8ab1a5a311af7a9ed in term 2.
I20250623 01:59:07.058861 18569 leader_election.cc:304] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 419f59c561fa4aa8ab1a5a311af7a9ed, e3ec3497126848969a0372dcacc06c59; no voters:
I20250623 01:59:07.059412 18850 raft_consensus.cc:2802] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 01:59:07.059650 18850 raft_consensus.cc:491] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:07.059888 18850 raft_consensus.cc:3058] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 2 FOLLOWER]: Advancing to term 3
W20250623 01:59:07.060288 18568 leader_election.cc:336] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 41149896fc354b3dacb5d1b32f6788f7 (127.14.43.131:45611): Network error: Client connection negotiation failed: client connection to 127.14.43.131:45611: connect: Connection refused (error 111)
I20250623 01:59:07.064189 18850 raft_consensus.cc:513] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } } peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } attrs { promote: false } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } }
I20250623 01:59:07.065546 18850 leader_election.cc:290] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed [CANDIDATE]: Term 3 election: Requested vote from peers 41149896fc354b3dacb5d1b32f6788f7 (127.14.43.131:45611), e3ec3497126848969a0372dcacc06c59 (127.14.43.130:44383)
I20250623 01:59:07.066618 18787 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62858b6460fa4faa8c4bb9a06c000863" candidate_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "e3ec3497126848969a0372dcacc06c59"
I20250623 01:59:07.067133 18787 raft_consensus.cc:3058] T 62858b6460fa4faa8c4bb9a06c000863 P e3ec3497126848969a0372dcacc06c59 [term 2 FOLLOWER]: Advancing to term 3
W20250623 01:59:07.068899 18568 leader_election.cc:336] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 41149896fc354b3dacb5d1b32f6788f7 (127.14.43.131:45611): Network error: Client connection negotiation failed: client connection to 127.14.43.131:45611: connect: Connection refused (error 111)
I20250623 01:59:07.071589 18787 raft_consensus.cc:2466] T 62858b6460fa4faa8c4bb9a06c000863 P e3ec3497126848969a0372dcacc06c59 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 419f59c561fa4aa8ab1a5a311af7a9ed in term 3.
I20250623 01:59:07.072381 18569 leader_election.cc:304] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 419f59c561fa4aa8ab1a5a311af7a9ed, e3ec3497126848969a0372dcacc06c59; no voters: 41149896fc354b3dacb5d1b32f6788f7
I20250623 01:59:07.073064 18850 raft_consensus.cc:2802] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 3 FOLLOWER]: Leader election won for term 3
I20250623 01:59:07.073410 18850 raft_consensus.cc:695] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 3 LEADER]: Becoming Leader. State: Replica: 419f59c561fa4aa8ab1a5a311af7a9ed, State: Running, Role: LEADER
I20250623 01:59:07.074110 18850 consensus_queue.cc:237] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } } peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } attrs { promote: false } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } }
I20250623 01:59:07.081580 18494 catalog_manager.cc:5582] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed reported cstate change: term changed from 2 to 3, leader changed from 41149896fc354b3dacb5d1b32f6788f7 (127.14.43.131) to 419f59c561fa4aa8ab1a5a311af7a9ed (127.14.43.129). New cstate: current_term: 3 leader_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250623 01:59:06.112430 18845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:59:06.114344 18847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:59:06.114943 18837 server_base.cc:1048] running on GCE node
I20250623 01:59:07.211771 18837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:59:07.213968 18837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:59:07.215353 18837 hybrid_clock.cc:648] HybridClock initialized: now 1750643947215311 us; error 54 us; skew 500 ppm
I20250623 01:59:07.216158 18837 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:59:07.226236 18837 webserver.cc:469] Webserver started at http://127.14.43.131:40355/ using document root <none> and password file <none>
I20250623 01:59:07.227149 18837 fs_manager.cc:362] Metadata directory not provided
I20250623 01:59:07.227370 18837 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:59:07.235335 18837 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20250623 01:59:07.239945 18866 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:59:07.240958 18837 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250623 01:59:07.241286 18837 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "41149896fc354b3dacb5d1b32f6788f7"
format_stamp: "Formatted at 2025-06-23 01:58:39 on dist-test-slave-9rpg"
I20250623 01:59:07.243221 18837 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:59:07.301465 18837 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:59:07.302956 18837 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:59:07.303397 18837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:59:07.305932 18837 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:59:07.311427 18873 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250623 01:59:07.322773 18837 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250623 01:59:07.323020 18837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.001s sys 0.000s
I20250623 01:59:07.323279 18837 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250623 01:59:07.328389 18873 tablet_bootstrap.cc:492] T 8ae52d3edd7748be9366f4baea5b1472 P 41149896fc354b3dacb5d1b32f6788f7: Bootstrap starting.
I20250623 01:59:07.331121 18837 ts_tablet_manager.cc:610] Registered 2 tablets
I20250623 01:59:07.331399 18837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.008s user 0.007s sys 0.000s
I20250623 01:59:07.381143 18873 log.cc:826] T 8ae52d3edd7748be9366f4baea5b1472 P 41149896fc354b3dacb5d1b32f6788f7: Log is configured to *not* fsync() on all Append() calls
W20250623 01:59:07.382901 18568 consensus_peers.cc:487] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed -> Peer 41149896fc354b3dacb5d1b32f6788f7 (127.14.43.131:45611): Couldn't send request to peer 41149896fc354b3dacb5d1b32f6788f7. Status: Network error: Client connection negotiation failed: client connection to 127.14.43.131:45611: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250623 01:59:07.422320 18787 raft_consensus.cc:1273] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [term 3 FOLLOWER]: Refusing update from remote peer 419f59c561fa4aa8ab1a5a311af7a9ed: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20250623 01:59:07.424172 18850 consensus_queue.cc:1035] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250623 01:59:07.478365 18636 consensus_queue.cc:237] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 13, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } }
I20250623 01:59:07.484232 18787 raft_consensus.cc:1273] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [term 3 FOLLOWER]: Refusing update from remote peer 419f59c561fa4aa8ab1a5a311af7a9ed: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20250623 01:59:07.485914 18855 consensus_queue.cc:1035] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250623 01:59:07.492226 18855 raft_consensus.cc:2953] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 3 LEADER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER 41149896fc354b3dacb5d1b32f6788f7 (127.14.43.131) evicted. New config: { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } } }
I20250623 01:59:07.496093 18787 raft_consensus.cc:2953] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [term 3 FOLLOWER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER 41149896fc354b3dacb5d1b32f6788f7 (127.14.43.131) evicted. New config: { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } } }
I20250623 01:59:07.502642 18481 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 8ae52d3edd7748be9366f4baea5b1472 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250623 01:59:07.507817 18494 catalog_manager.cc:5582] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed reported cstate change: config changed from index 13 to 15, VOTER 41149896fc354b3dacb5d1b32f6788f7 (127.14.43.131) evicted. New cstate: current_term: 3 leader_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" committed_config { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250623 01:59:07.516690 18494 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet 8ae52d3edd7748be9366f4baea5b1472 on TS 41149896fc354b3dacb5d1b32f6788f7: Not found: failed to reset TS proxy: Could not find TS for UUID 41149896fc354b3dacb5d1b32f6788f7
I20250623 01:59:07.520805 18636 consensus_queue.cc:237] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 3.15, Last appended by leader: 13, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } }
I20250623 01:59:07.523649 18850 raft_consensus.cc:2953] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed [term 3 LEADER]: Committing config change with OpId 3.16: config changed from index 15 to 16, VOTER e3ec3497126848969a0372dcacc06c59 (127.14.43.130) evicted. New config: { opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } } }
I20250623 01:59:07.527390 18873 tablet_bootstrap.cc:492] T 8ae52d3edd7748be9366f4baea5b1472 P 41149896fc354b3dacb5d1b32f6788f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 01:59:07.528347 18873 tablet_bootstrap.cc:492] T 8ae52d3edd7748be9366f4baea5b1472 P 41149896fc354b3dacb5d1b32f6788f7: Bootstrap complete.
I20250623 01:59:07.530202 18873 ts_tablet_manager.cc:1397] T 8ae52d3edd7748be9366f4baea5b1472 P 41149896fc354b3dacb5d1b32f6788f7: Time spent bootstrapping tablet: real 0.202s user 0.149s sys 0.043s
I20250623 01:59:07.531626 18481 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 8ae52d3edd7748be9366f4baea5b1472 with cas_config_opid_index 15: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250623 01:59:07.535446 18494 catalog_manager.cc:5582] T 8ae52d3edd7748be9366f4baea5b1472 P 419f59c561fa4aa8ab1a5a311af7a9ed reported cstate change: config changed from index 15 to 16, VOTER e3ec3497126848969a0372dcacc06c59 (127.14.43.130) evicted. New cstate: current_term: 3 leader_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" committed_config { opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250623 01:59:07.551158 18873 raft_consensus.cc:357] T 8ae52d3edd7748be9366f4baea5b1472 P 41149896fc354b3dacb5d1b32f6788f7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } } peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } attrs { promote: false } }
I20250623 01:59:07.554234 18873 raft_consensus.cc:738] T 8ae52d3edd7748be9366f4baea5b1472 P 41149896fc354b3dacb5d1b32f6788f7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41149896fc354b3dacb5d1b32f6788f7, State: Initialized, Role: FOLLOWER
I20250623 01:59:07.555294 18873 consensus_queue.cc:260] T 8ae52d3edd7748be9366f4baea5b1472 P 41149896fc354b3dacb5d1b32f6788f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } } peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } attrs { promote: false } }
I20250623 01:59:07.559612 18767 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ae52d3edd7748be9366f4baea5b1472 with delete_type TABLET_DATA_TOMBSTONED (TS e3ec3497126848969a0372dcacc06c59 not found in new config with opid_index 16) from {username='slave'} at 127.0.0.1:58902
I20250623 01:59:07.560669 18873 ts_tablet_manager.cc:1428] T 8ae52d3edd7748be9366f4baea5b1472 P 41149896fc354b3dacb5d1b32f6788f7: Time spent starting tablet: real 0.030s user 0.024s sys 0.005s
I20250623 01:59:07.561501 18873 tablet_bootstrap.cc:492] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7: Bootstrap starting.
I20250623 01:59:07.561594 18837 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.131:45611
I20250623 01:59:07.562357 18988 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.131:45611 every 8 connection(s)
I20250623 01:59:07.563848 18989 tablet_replica.cc:331] stopping tablet replica
I20250623 01:59:07.564639 18989 raft_consensus.cc:2241] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:07.565114 18837 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250623 01:59:07.565276 18989 raft_consensus.cc:2270] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:07.568966 18989 ts_tablet_manager.cc:1905] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20250623 01:59:07.569984 18479 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 8ae52d3edd7748be9366f4baea5b1472 on TS 41149896fc354b3dacb5d1b32f6788f7 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 41149896fc354b3dacb5d1b32f6788f7
I20250623 01:59:07.574626 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 18837
I20250623 01:59:07.586248 18989 ts_tablet_manager.cc:1918] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.15
I20250623 01:59:07.586695 18989 log.cc:1199] T 8ae52d3edd7748be9366f4baea5b1472 P e3ec3497126848969a0372dcacc06c59: Deleting WAL directory at /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal/wals/8ae52d3edd7748be9366f4baea5b1472
I20250623 01:59:07.591102 18480 catalog_manager.cc:4928] TS e3ec3497126848969a0372dcacc06c59 (127.14.43.130:44383): tablet 8ae52d3edd7748be9366f4baea5b1472 (table TestTable [id=44d246c7602b478183f2b259f067a7c7]) successfully deleted
I20250623 01:59:07.606701 18990 heartbeater.cc:344] Connected to a master server at 127.14.43.190:33475
I20250623 01:59:07.607120 18990 heartbeater.cc:461] Registering TS with master...
I20250623 01:59:07.608227 18990 heartbeater.cc:507] Master 127.14.43.190:33475 requested a full tablet report, sending...
I20250623 01:59:07.611438 18494 ts_manager.cc:194] Registered new tserver with Master: 41149896fc354b3dacb5d1b32f6788f7 (127.14.43.131:45611)
I20250623 01:59:07.614390 18494 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.43.131:40523
I20250623 01:59:07.623737 14510 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250623 01:59:07.628242 14510 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
W20250623 01:59:07.631760 14510 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 8ae52d3edd7748be9366f4baea5b1472: tablet_id: "8ae52d3edd7748be9366f4baea5b1472" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
I20250623 01:59:07.632115 18916 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ae52d3edd7748be9366f4baea5b1472 with delete_type TABLET_DATA_TOMBSTONED (TS 41149896fc354b3dacb5d1b32f6788f7 not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:33238
I20250623 01:59:07.637565 18998 tablet_replica.cc:331] stopping tablet replica
I20250623 01:59:07.638345 18998 raft_consensus.cc:2241] T 8ae52d3edd7748be9366f4baea5b1472 P 41149896fc354b3dacb5d1b32f6788f7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250623 01:59:07.638442 18787 raft_consensus.cc:1273] T 62858b6460fa4faa8c4bb9a06c000863 P e3ec3497126848969a0372dcacc06c59 [term 3 FOLLOWER]: Refusing update from remote peer 419f59c561fa4aa8ab1a5a311af7a9ed: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20250623 01:59:07.638845 18998 raft_consensus.cc:2270] T 8ae52d3edd7748be9366f4baea5b1472 P 41149896fc354b3dacb5d1b32f6788f7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 01:59:07.639541 18850 consensus_queue.cc:1035] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250623 01:59:07.643473 18998 ts_tablet_manager.cc:1905] T 8ae52d3edd7748be9366f4baea5b1472 P 41149896fc354b3dacb5d1b32f6788f7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20250623 01:59:07.664589 18568 consensus_peers.cc:487] T 62858b6460fa4faa8c4bb9a06c000863 P 419f59c561fa4aa8ab1a5a311af7a9ed -> Peer 41149896fc354b3dacb5d1b32f6788f7 (127.14.43.131:45611): Couldn't send request to peer 41149896fc354b3dacb5d1b32f6788f7. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250623 01:59:07.667896 18998 ts_tablet_manager.cc:1918] T 8ae52d3edd7748be9366f4baea5b1472 P 41149896fc354b3dacb5d1b32f6788f7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.13
I20250623 01:59:07.668287 18998 log.cc:1199] T 8ae52d3edd7748be9366f4baea5b1472 P 41149896fc354b3dacb5d1b32f6788f7: Deleting WAL directory at /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal/wals/8ae52d3edd7748be9366f4baea5b1472
I20250623 01:59:07.669889 18479 catalog_manager.cc:4928] TS 41149896fc354b3dacb5d1b32f6788f7 (127.14.43.131:45611): tablet 8ae52d3edd7748be9366f4baea5b1472 (table TestTable [id=44d246c7602b478183f2b259f067a7c7]) successfully deleted
I20250623 01:59:07.679453 18873 tablet_bootstrap.cc:492] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 01:59:07.680136 18873 tablet_bootstrap.cc:492] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7: Bootstrap complete.
I20250623 01:59:07.681061 18873 ts_tablet_manager.cc:1397] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7: Time spent bootstrapping tablet: real 0.120s user 0.105s sys 0.011s
I20250623 01:59:07.682614 18873 raft_consensus.cc:357] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } } peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } attrs { promote: false } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } }
I20250623 01:59:07.683033 18873 raft_consensus.cc:738] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41149896fc354b3dacb5d1b32f6788f7, State: Initialized, Role: FOLLOWER
I20250623 01:59:07.683499 18873 consensus_queue.cc:260] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "41149896fc354b3dacb5d1b32f6788f7" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 45611 } } peers { permanent_uuid: "e3ec3497126848969a0372dcacc06c59" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44383 } attrs { promote: false } } peers { permanent_uuid: "419f59c561fa4aa8ab1a5a311af7a9ed" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44313 } attrs { promote: false } }
I20250623 01:59:07.684381 18990 heartbeater.cc:499] Master 127.14.43.190:33475 was elected leader, sending a full tablet report...
I20250623 01:59:07.684652 18873 ts_tablet_manager.cc:1428] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20250623 01:59:08.191335 18938 raft_consensus.cc:3058] T 62858b6460fa4faa8c4bb9a06c000863 P 41149896fc354b3dacb5d1b32f6788f7 [term 2 FOLLOWER]: Advancing to term 3
W20250623 01:59:08.635567 14510 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 8ae52d3edd7748be9366f4baea5b1472: tablet_id: "8ae52d3edd7748be9366f4baea5b1472" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250623 01:59:09.638904 14510 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 8ae52d3edd7748be9366f4baea5b1472: tablet_id: "8ae52d3edd7748be9366f4baea5b1472" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250623 01:59:10.643177 14510 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 8ae52d3edd7748be9366f4baea5b1472: tablet_id: "8ae52d3edd7748be9366f4baea5b1472" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250623 01:59:11.646404 14510 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 8ae52d3edd7748be9366f4baea5b1472: tablet_id: "8ae52d3edd7748be9366f4baea5b1472" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250623 01:59:12.649724 14510 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 8ae52d3edd7748be9366f4baea5b1472: tablet_id: "8ae52d3edd7748be9366f4baea5b1472" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250623 01:59:13.652959 14510 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 8ae52d3edd7748be9366f4baea5b1472: tablet_id: "8ae52d3edd7748be9366f4baea5b1472" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250623 01:59:14.657070 14510 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 8ae52d3edd7748be9366f4baea5b1472: tablet_id: "8ae52d3edd7748be9366f4baea5b1472" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250623 01:59:15.660806 14510 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 8ae52d3edd7748be9366f4baea5b1472: tablet_id: "8ae52d3edd7748be9366f4baea5b1472" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250623 01:59:16.664222 14510 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 8ae52d3edd7748be9366f4baea5b1472: tablet_id: "8ae52d3edd7748be9366f4baea5b1472" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250623 01:59:17.667822 14510 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 8ae52d3edd7748be9366f4baea5b1472: tablet_id: "8ae52d3edd7748be9366f4baea5b1472" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250623 01:59:18.671280 14510 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 8ae52d3edd7748be9366f4baea5b1472: tablet_id: "8ae52d3edd7748be9366f4baea5b1472" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250623 01:59:19.674753 14510 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 8ae52d3edd7748be9366f4baea5b1472: tablet_id: "8ae52d3edd7748be9366f4baea5b1472" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250623 01:59:20.678968 14510 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 8ae52d3edd7748be9366f4baea5b1472: tablet_id: "8ae52d3edd7748be9366f4baea5b1472" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250623 01:59:21.682339 14510 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 8ae52d3edd7748be9366f4baea5b1472: tablet_id: "8ae52d3edd7748be9366f4baea5b1472" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250623 01:59:22.685400 14510 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 8ae52d3edd7748be9366f4baea5b1472: tablet_id: "8ae52d3edd7748be9366f4baea5b1472" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250623 01:59:23.688710 14510 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 8ae52d3edd7748be9366f4baea5b1472: tablet_id: "8ae52d3edd7748be9366f4baea5b1472" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250623 01:59:24.692834 14510 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 8ae52d3edd7748be9366f4baea5b1472: tablet_id: "8ae52d3edd7748be9366f4baea5b1472" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250623 01:59:25.696662 14510 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 8ae52d3edd7748be9366f4baea5b1472: tablet_id: "8ae52d3edd7748be9366f4baea5b1472" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250623 01:59:26.699839 14510 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 8ae52d3edd7748be9366f4baea5b1472: tablet_id: "8ae52d3edd7748be9366f4baea5b1472" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/kudu-admin-test.cc:3914: Failure
Failed
Bad status: Not found: not all replicas of tablets comprising table TestTable are registered yet
I20250623 01:59:27.703780 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 18531
I20250623 01:59:27.727397 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 18687
I20250623 01:59:27.748324 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 18837
I20250623 01:59:27.770599 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 18461
2025-06-23T01:59:27Z chronyd exiting
I20250623 01:59:27.814157 14510 test_util.cc:183] -----------------------------------------------
I20250623 01:59:27.814350 14510 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750643853547882-14510-0
[ FAILED ] AdminCliTest.TestRebuildTables (54839 ms)
[----------] 5 tests from AdminCliTest (114197 ms total)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest
[ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4
I20250623 01:59:27.817814 14510 test_util.cc:276] Using random seed: -1881900994
I20250623 01:59:27.821975 14510 ts_itest-base.cc:115] Starting cluster with:
I20250623 01:59:27.822165 14510 ts_itest-base.cc:116] --------------
I20250623 01:59:27.822283 14510 ts_itest-base.cc:117] 5 tablet servers
I20250623 01:59:27.822389 14510 ts_itest-base.cc:118] 3 replicas per TS
I20250623 01:59:27.822503 14510 ts_itest-base.cc:119] --------------
2025-06-23T01:59:27Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-23T01:59:27Z Disabled control of system clock
I20250623 01:59:27.860880 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.43.190:32971
--webserver_interface=127.14.43.190
--webserver_port=0
--builtin_ntp_servers=127.14.43.148:43039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.43.190:32971
--raft_prepare_replacement_before_eviction=true with env {}
W20250623 01:59:28.148901 19025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:59:28.149456 19025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:59:28.149855 19025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:59:28.180593 19025 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250623 01:59:28.180938 19025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250623 01:59:28.181164 19025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:59:28.181373 19025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250623 01:59:28.181627 19025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250623 01:59:28.216737 19025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:43039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.43.190:32971
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.43.190:32971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.14.43.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:59:28.217988 19025 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:59:28.219638 19025 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:59:28.234515 19032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:59:28.234565 19031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:59:28.236546 19025 server_base.cc:1048] running on GCE node
W20250623 01:59:28.235435 19034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:59:29.317382 19025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:59:29.320369 19025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:59:29.321784 19025 hybrid_clock.cc:648] HybridClock initialized: now 1750643969321722 us; error 83 us; skew 500 ppm
I20250623 01:59:29.322620 19025 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:59:29.328845 19025 webserver.cc:469] Webserver started at http://127.14.43.190:33279/ using document root <none> and password file <none>
I20250623 01:59:29.329702 19025 fs_manager.cc:362] Metadata directory not provided
I20250623 01:59:29.329888 19025 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:59:29.330332 19025 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:59:29.335718 19025 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "3150da46043245c2aa8e8a9b31dee5d9"
format_stamp: "Formatted at 2025-06-23 01:59:29 on dist-test-slave-9rpg"
I20250623 01:59:29.336781 19025 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "3150da46043245c2aa8e8a9b31dee5d9"
format_stamp: "Formatted at 2025-06-23 01:59:29 on dist-test-slave-9rpg"
I20250623 01:59:29.343782 19025 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20250623 01:59:29.349188 19041 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:59:29.350200 19025 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250623 01:59:29.350507 19025 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
uuid: "3150da46043245c2aa8e8a9b31dee5d9"
format_stamp: "Formatted at 2025-06-23 01:59:29 on dist-test-slave-9rpg"
I20250623 01:59:29.350831 19025 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:59:29.394987 19025 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:59:29.396430 19025 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:59:29.396842 19025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:59:29.463218 19025 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.190:32971
I20250623 01:59:29.463284 19092 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.190:32971 every 8 connection(s)
I20250623 01:59:29.465793 19025 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250623 01:59:29.470691 19093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:59:29.473687 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 19025
I20250623 01:59:29.474103 14510 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250623 01:59:29.493235 19093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3150da46043245c2aa8e8a9b31dee5d9: Bootstrap starting.
I20250623 01:59:29.499203 19093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3150da46043245c2aa8e8a9b31dee5d9: Neither blocks nor log segments found. Creating new log.
I20250623 01:59:29.500989 19093 log.cc:826] T 00000000000000000000000000000000 P 3150da46043245c2aa8e8a9b31dee5d9: Log is configured to *not* fsync() on all Append() calls
I20250623 01:59:29.505138 19093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3150da46043245c2aa8e8a9b31dee5d9: No bootstrap required, opened a new log
I20250623 01:59:29.522863 19093 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3150da46043245c2aa8e8a9b31dee5d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3150da46043245c2aa8e8a9b31dee5d9" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 32971 } }
I20250623 01:59:29.523494 19093 raft_consensus.cc:383] T 00000000000000000000000000000000 P 3150da46043245c2aa8e8a9b31dee5d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:59:29.523733 19093 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3150da46043245c2aa8e8a9b31dee5d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3150da46043245c2aa8e8a9b31dee5d9, State: Initialized, Role: FOLLOWER
I20250623 01:59:29.524379 19093 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3150da46043245c2aa8e8a9b31dee5d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3150da46043245c2aa8e8a9b31dee5d9" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 32971 } }
I20250623 01:59:29.524859 19093 raft_consensus.cc:397] T 00000000000000000000000000000000 P 3150da46043245c2aa8e8a9b31dee5d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250623 01:59:29.525106 19093 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3150da46043245c2aa8e8a9b31dee5d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250623 01:59:29.525379 19093 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 3150da46043245c2aa8e8a9b31dee5d9 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:59:29.529315 19093 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3150da46043245c2aa8e8a9b31dee5d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3150da46043245c2aa8e8a9b31dee5d9" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 32971 } }
I20250623 01:59:29.529990 19093 leader_election.cc:304] T 00000000000000000000000000000000 P 3150da46043245c2aa8e8a9b31dee5d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3150da46043245c2aa8e8a9b31dee5d9; no voters:
I20250623 01:59:29.532112 19093 leader_election.cc:290] T 00000000000000000000000000000000 P 3150da46043245c2aa8e8a9b31dee5d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250623 01:59:29.532889 19098 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 3150da46043245c2aa8e8a9b31dee5d9 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:59:29.535038 19098 raft_consensus.cc:695] T 00000000000000000000000000000000 P 3150da46043245c2aa8e8a9b31dee5d9 [term 1 LEADER]: Becoming Leader. State: Replica: 3150da46043245c2aa8e8a9b31dee5d9, State: Running, Role: LEADER
I20250623 01:59:29.535873 19098 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3150da46043245c2aa8e8a9b31dee5d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3150da46043245c2aa8e8a9b31dee5d9" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 32971 } }
I20250623 01:59:29.536567 19093 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3150da46043245c2aa8e8a9b31dee5d9 [sys.catalog]: configured and running, proceeding with master startup.
I20250623 01:59:29.546027 19100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3150da46043245c2aa8e8a9b31dee5d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3150da46043245c2aa8e8a9b31dee5d9. Latest consensus state: current_term: 1 leader_uuid: "3150da46043245c2aa8e8a9b31dee5d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3150da46043245c2aa8e8a9b31dee5d9" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 32971 } } }
I20250623 01:59:29.546461 19099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3150da46043245c2aa8e8a9b31dee5d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3150da46043245c2aa8e8a9b31dee5d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3150da46043245c2aa8e8a9b31dee5d9" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 32971 } } }
I20250623 01:59:29.546837 19100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3150da46043245c2aa8e8a9b31dee5d9 [sys.catalog]: This master's current role is: LEADER
I20250623 01:59:29.547039 19099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3150da46043245c2aa8e8a9b31dee5d9 [sys.catalog]: This master's current role is: LEADER
I20250623 01:59:29.550647 19107 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250623 01:59:29.558419 19107 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250623 01:59:29.574914 19107 catalog_manager.cc:1349] Generated new cluster ID: 9e0eda940268457aa30d0f5d5cfd7eaa
I20250623 01:59:29.575304 19107 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250623 01:59:29.605165 19107 catalog_manager.cc:1372] Generated new certificate authority record
I20250623 01:59:29.606595 19107 catalog_manager.cc:1506] Loading token signing keys...
I20250623 01:59:29.630132 19107 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 3150da46043245c2aa8e8a9b31dee5d9: Generated new TSK 0
I20250623 01:59:29.631004 19107 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250623 01:59:29.647890 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.43.129:0
--local_ip_for_outbound_sockets=127.14.43.129
--webserver_interface=127.14.43.129
--webserver_port=0
--tserver_master_addrs=127.14.43.190:32971
--builtin_ntp_servers=127.14.43.148:43039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250623 01:59:29.942580 19117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:59:29.943056 19117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:59:29.943532 19117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:59:29.973928 19117 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250623 01:59:29.974351 19117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:59:29.975101 19117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.43.129
I20250623 01:59:30.009852 19117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:43039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.43.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.14.43.129
--webserver_port=0
--tserver_master_addrs=127.14.43.190:32971
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.43.129
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:59:30.011153 19117 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:59:30.012769 19117 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:59:30.029490 19124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:59:30.031563 19117 server_base.cc:1048] running on GCE node
W20250623 01:59:30.030647 19126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:59:30.030479 19123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:59:31.116029 19117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:59:31.118917 19117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:59:31.120434 19117 hybrid_clock.cc:648] HybridClock initialized: now 1750643971120406 us; error 37 us; skew 500 ppm
I20250623 01:59:31.121272 19117 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:59:31.127863 19117 webserver.cc:469] Webserver started at http://127.14.43.129:35483/ using document root <none> and password file <none>
I20250623 01:59:31.128793 19117 fs_manager.cc:362] Metadata directory not provided
I20250623 01:59:31.128998 19117 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:59:31.129458 19117 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:59:31.133952 19117 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "e78be0fbcab4466398aa66cf7b1dfb01"
format_stamp: "Formatted at 2025-06-23 01:59:31 on dist-test-slave-9rpg"
I20250623 01:59:31.135111 19117 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "e78be0fbcab4466398aa66cf7b1dfb01"
format_stamp: "Formatted at 2025-06-23 01:59:31 on dist-test-slave-9rpg"
I20250623 01:59:31.142537 19117 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250623 01:59:31.148401 19133 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:59:31.149525 19117 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250623 01:59:31.149849 19117 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "e78be0fbcab4466398aa66cf7b1dfb01"
format_stamp: "Formatted at 2025-06-23 01:59:31 on dist-test-slave-9rpg"
I20250623 01:59:31.150215 19117 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:59:31.209977 19117 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:59:31.211504 19117 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:59:31.211917 19117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:59:31.214743 19117 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:59:31.219022 19117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250623 01:59:31.219239 19117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250623 01:59:31.219485 19117 ts_tablet_manager.cc:610] Registered 0 tablets
I20250623 01:59:31.219635 19117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250623 01:59:31.348948 19117 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.129:44657
I20250623 01:59:31.349056 19245 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.129:44657 every 8 connection(s)
I20250623 01:59:31.351480 19117 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250623 01:59:31.357805 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 19117
I20250623 01:59:31.358312 14510 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250623 01:59:31.364053 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.43.130:0
--local_ip_for_outbound_sockets=127.14.43.130
--webserver_interface=127.14.43.130
--webserver_port=0
--tserver_master_addrs=127.14.43.190:32971
--builtin_ntp_servers=127.14.43.148:43039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250623 01:59:31.371217 19246 heartbeater.cc:344] Connected to a master server at 127.14.43.190:32971
I20250623 01:59:31.371636 19246 heartbeater.cc:461] Registering TS with master...
I20250623 01:59:31.372591 19246 heartbeater.cc:507] Master 127.14.43.190:32971 requested a full tablet report, sending...
I20250623 01:59:31.375013 19058 ts_manager.cc:194] Registered new tserver with Master: e78be0fbcab4466398aa66cf7b1dfb01 (127.14.43.129:44657)
I20250623 01:59:31.376924 19058 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.43.129:52531
W20250623 01:59:31.657233 19250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:59:31.657745 19250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:59:31.658253 19250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:59:31.689214 19250 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250623 01:59:31.689630 19250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:59:31.690438 19250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.43.130
I20250623 01:59:31.724524 19250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:43039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.43.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.14.43.130
--webserver_port=0
--tserver_master_addrs=127.14.43.190:32971
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.43.130
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:59:31.725826 19250 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:59:31.727428 19250 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:59:31.742568 19257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:59:32.380170 19246 heartbeater.cc:499] Master 127.14.43.190:32971 was elected leader, sending a full tablet report...
W20250623 01:59:31.742784 19259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:59:31.742605 19256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:59:32.804785 19258 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250623 01:59:32.804904 19250 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250623 01:59:32.808398 19250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:59:32.811093 19250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:59:32.812523 19250 hybrid_clock.cc:648] HybridClock initialized: now 1750643972812485 us; error 60 us; skew 500 ppm
I20250623 01:59:32.813315 19250 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:59:32.823422 19250 webserver.cc:469] Webserver started at http://127.14.43.130:45413/ using document root <none> and password file <none>
I20250623 01:59:32.824349 19250 fs_manager.cc:362] Metadata directory not provided
I20250623 01:59:32.824564 19250 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:59:32.824970 19250 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:59:32.829533 19250 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "854ed4ef070a4ebcbe1be48b70110323"
format_stamp: "Formatted at 2025-06-23 01:59:32 on dist-test-slave-9rpg"
I20250623 01:59:32.830657 19250 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "854ed4ef070a4ebcbe1be48b70110323"
format_stamp: "Formatted at 2025-06-23 01:59:32 on dist-test-slave-9rpg"
I20250623 01:59:32.837970 19250 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250623 01:59:32.843659 19266 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:59:32.844691 19250 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250623 01:59:32.845009 19250 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "854ed4ef070a4ebcbe1be48b70110323"
format_stamp: "Formatted at 2025-06-23 01:59:32 on dist-test-slave-9rpg"
I20250623 01:59:32.845337 19250 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:59:32.894176 19250 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:59:32.895654 19250 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:59:32.896085 19250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:59:32.898509 19250 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:59:32.902456 19250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250623 01:59:32.902664 19250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250623 01:59:32.902895 19250 ts_tablet_manager.cc:610] Registered 0 tablets
I20250623 01:59:32.903049 19250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250623 01:59:33.029839 19250 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.130:35343
I20250623 01:59:33.029940 19378 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.130:35343 every 8 connection(s)
I20250623 01:59:33.032315 19250 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250623 01:59:33.033627 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 19250
I20250623 01:59:33.034060 14510 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250623 01:59:33.041841 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.43.131:0
--local_ip_for_outbound_sockets=127.14.43.131
--webserver_interface=127.14.43.131
--webserver_port=0
--tserver_master_addrs=127.14.43.190:32971
--builtin_ntp_servers=127.14.43.148:43039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250623 01:59:33.055464 19379 heartbeater.cc:344] Connected to a master server at 127.14.43.190:32971
I20250623 01:59:33.055882 19379 heartbeater.cc:461] Registering TS with master...
I20250623 01:59:33.056838 19379 heartbeater.cc:507] Master 127.14.43.190:32971 requested a full tablet report, sending...
I20250623 01:59:33.058882 19058 ts_manager.cc:194] Registered new tserver with Master: 854ed4ef070a4ebcbe1be48b70110323 (127.14.43.130:35343)
I20250623 01:59:33.060057 19058 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.43.130:58331
W20250623 01:59:33.333154 19383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:59:33.333691 19383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:59:33.334182 19383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:59:33.365347 19383 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250623 01:59:33.365763 19383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:59:33.366647 19383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.43.131
I20250623 01:59:33.403270 19383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:43039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.43.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.14.43.131
--webserver_port=0
--tserver_master_addrs=127.14.43.190:32971
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.43.131
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:59:33.404603 19383 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:59:33.406301 19383 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:59:33.422785 19392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:59:34.063158 19379 heartbeater.cc:499] Master 127.14.43.190:32971 was elected leader, sending a full tablet report...
W20250623 01:59:33.422865 19389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:59:33.425196 19383 server_base.cc:1048] running on GCE node
W20250623 01:59:33.423014 19390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:59:34.499665 19383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:59:34.501822 19383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:59:34.503185 19383 hybrid_clock.cc:648] HybridClock initialized: now 1750643974503143 us; error 55 us; skew 500 ppm
I20250623 01:59:34.503957 19383 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:59:34.510077 19383 webserver.cc:469] Webserver started at http://127.14.43.131:33001/ using document root <none> and password file <none>
I20250623 01:59:34.510975 19383 fs_manager.cc:362] Metadata directory not provided
I20250623 01:59:34.511188 19383 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:59:34.511629 19383 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:59:34.515978 19383 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "ca43fa39173e417f91e64733e700b0af"
format_stamp: "Formatted at 2025-06-23 01:59:34 on dist-test-slave-9rpg"
I20250623 01:59:34.517037 19383 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "ca43fa39173e417f91e64733e700b0af"
format_stamp: "Formatted at 2025-06-23 01:59:34 on dist-test-slave-9rpg"
I20250623 01:59:34.524020 19383 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.001s sys 0.008s
I20250623 01:59:34.529441 19400 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:59:34.530431 19383 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250623 01:59:34.530726 19383 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ca43fa39173e417f91e64733e700b0af"
format_stamp: "Formatted at 2025-06-23 01:59:34 on dist-test-slave-9rpg"
I20250623 01:59:34.531038 19383 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:59:34.582757 19383 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:59:34.584272 19383 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:59:34.584717 19383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:59:34.587165 19383 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:59:34.591171 19383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250623 01:59:34.591390 19383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250623 01:59:34.591621 19383 ts_tablet_manager.cc:610] Registered 0 tablets
I20250623 01:59:34.591774 19383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250623 01:59:34.717753 19383 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.131:40255
I20250623 01:59:34.717866 19512 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.131:40255 every 8 connection(s)
I20250623 01:59:34.720314 19383 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250623 01:59:34.723009 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 19383
I20250623 01:59:34.723766 14510 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250623 01:59:34.731067 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.43.132:0
--local_ip_for_outbound_sockets=127.14.43.132
--webserver_interface=127.14.43.132
--webserver_port=0
--tserver_master_addrs=127.14.43.190:32971
--builtin_ntp_servers=127.14.43.148:43039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250623 01:59:34.741832 19513 heartbeater.cc:344] Connected to a master server at 127.14.43.190:32971
I20250623 01:59:34.742453 19513 heartbeater.cc:461] Registering TS with master...
I20250623 01:59:34.743714 19513 heartbeater.cc:507] Master 127.14.43.190:32971 requested a full tablet report, sending...
I20250623 01:59:34.746111 19058 ts_manager.cc:194] Registered new tserver with Master: ca43fa39173e417f91e64733e700b0af (127.14.43.131:40255)
I20250623 01:59:34.747334 19058 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.43.131:47169
W20250623 01:59:35.024478 19517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:59:35.024956 19517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:59:35.025451 19517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:59:35.062568 19517 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250623 01:59:35.062963 19517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:59:35.063728 19517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.43.132
I20250623 01:59:35.098639 19517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:43039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.43.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.14.43.132
--webserver_port=0
--tserver_master_addrs=127.14.43.190:32971
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.43.132
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:59:35.099913 19517 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:59:35.101541 19517 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:59:35.115828 19524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:59:35.750941 19513 heartbeater.cc:499] Master 127.14.43.190:32971 was elected leader, sending a full tablet report...
W20250623 01:59:35.116060 19526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:59:35.115859 19523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:59:36.194231 19525 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250623 01:59:36.194355 19517 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250623 01:59:36.198508 19517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:59:36.200966 19517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:59:36.202369 19517 hybrid_clock.cc:648] HybridClock initialized: now 1750643976202336 us; error 53 us; skew 500 ppm
I20250623 01:59:36.203109 19517 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:59:36.209278 19517 webserver.cc:469] Webserver started at http://127.14.43.132:42461/ using document root <none> and password file <none>
I20250623 01:59:36.210161 19517 fs_manager.cc:362] Metadata directory not provided
I20250623 01:59:36.210352 19517 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:59:36.210739 19517 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:59:36.214974 19517 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "2d541983e31e4a8da7b727adfcc6d628"
format_stamp: "Formatted at 2025-06-23 01:59:36 on dist-test-slave-9rpg"
I20250623 01:59:36.215976 19517 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "2d541983e31e4a8da7b727adfcc6d628"
format_stamp: "Formatted at 2025-06-23 01:59:36 on dist-test-slave-9rpg"
I20250623 01:59:36.222623 19517 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20250623 01:59:36.227968 19533 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:59:36.228945 19517 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20250623 01:59:36.229231 19517 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "2d541983e31e4a8da7b727adfcc6d628"
format_stamp: "Formatted at 2025-06-23 01:59:36 on dist-test-slave-9rpg"
I20250623 01:59:36.229522 19517 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:59:36.276865 19517 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:59:36.278321 19517 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:59:36.278707 19517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:59:36.281057 19517 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:59:36.284998 19517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250623 01:59:36.285205 19517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250623 01:59:36.285395 19517 ts_tablet_manager.cc:610] Registered 0 tablets
I20250623 01:59:36.285521 19517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250623 01:59:36.417508 19517 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.132:39941
I20250623 01:59:36.417605 19645 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.132:39941 every 8 connection(s)
I20250623 01:59:36.420009 19517 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250623 01:59:36.430160 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 19517
I20250623 01:59:36.430575 14510 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250623 01:59:36.436062 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.43.133:0
--local_ip_for_outbound_sockets=127.14.43.133
--webserver_interface=127.14.43.133
--webserver_port=0
--tserver_master_addrs=127.14.43.190:32971
--builtin_ntp_servers=127.14.43.148:43039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250623 01:59:36.440197 19646 heartbeater.cc:344] Connected to a master server at 127.14.43.190:32971
I20250623 01:59:36.440701 19646 heartbeater.cc:461] Registering TS with master...
I20250623 01:59:36.441926 19646 heartbeater.cc:507] Master 127.14.43.190:32971 requested a full tablet report, sending...
I20250623 01:59:36.444115 19058 ts_manager.cc:194] Registered new tserver with Master: 2d541983e31e4a8da7b727adfcc6d628 (127.14.43.132:39941)
I20250623 01:59:36.445283 19058 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.43.132:41273
W20250623 01:59:36.743157 19650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:59:36.743638 19650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:59:36.744112 19650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:59:36.775774 19650 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250623 01:59:36.776175 19650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:59:36.776944 19650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.43.133
I20250623 01:59:36.811097 19650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:43039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.43.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.14.43.133
--webserver_port=0
--tserver_master_addrs=127.14.43.190:32971
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.43.133
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:59:36.812383 19650 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:59:36.813934 19650 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:59:36.827921 19656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:59:37.448568 19646 heartbeater.cc:499] Master 127.14.43.190:32971 was elected leader, sending a full tablet report...
W20250623 01:59:36.828402 19657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:59:36.830423 19650 server_base.cc:1048] running on GCE node
W20250623 01:59:36.831306 19659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:59:37.906738 19650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:59:37.909216 19650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:59:37.910612 19650 hybrid_clock.cc:648] HybridClock initialized: now 1750643977910589 us; error 44 us; skew 500 ppm
I20250623 01:59:37.911373 19650 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:59:37.921195 19650 webserver.cc:469] Webserver started at http://127.14.43.133:37855/ using document root <none> and password file <none>
I20250623 01:59:37.922104 19650 fs_manager.cc:362] Metadata directory not provided
I20250623 01:59:37.922297 19650 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:59:37.922708 19650 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:59:37.927290 19650 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "bc052f7b41394146a4c7d20e1e90ad94"
format_stamp: "Formatted at 2025-06-23 01:59:37 on dist-test-slave-9rpg"
I20250623 01:59:37.928313 19650 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "bc052f7b41394146a4c7d20e1e90ad94"
format_stamp: "Formatted at 2025-06-23 01:59:37 on dist-test-slave-9rpg"
I20250623 01:59:37.935066 19650 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250623 01:59:37.940551 19666 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:59:37.941550 19650 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250623 01:59:37.941823 19650 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "bc052f7b41394146a4c7d20e1e90ad94"
format_stamp: "Formatted at 2025-06-23 01:59:37 on dist-test-slave-9rpg"
I20250623 01:59:37.942159 19650 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:59:38.006222 19650 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:59:38.007625 19650 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:59:38.008024 19650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:59:38.010460 19650 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:59:38.014386 19650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250623 01:59:38.014592 19650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250623 01:59:38.014788 19650 ts_tablet_manager.cc:610] Registered 0 tablets
I20250623 01:59:38.014917 19650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250623 01:59:38.151541 19650 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.133:38759
I20250623 01:59:38.151674 19779 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.133:38759 every 8 connection(s)
I20250623 01:59:38.154166 19650 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250623 01:59:38.155726 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 19650
I20250623 01:59:38.156208 14510 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250623 01:59:38.175688 19780 heartbeater.cc:344] Connected to a master server at 127.14.43.190:32971
I20250623 01:59:38.176195 19780 heartbeater.cc:461] Registering TS with master...
I20250623 01:59:38.177438 19780 heartbeater.cc:507] Master 127.14.43.190:32971 requested a full tablet report, sending...
I20250623 01:59:38.179818 19058 ts_manager.cc:194] Registered new tserver with Master: bc052f7b41394146a4c7d20e1e90ad94 (127.14.43.133:38759)
I20250623 01:59:38.181305 19058 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.43.133:59953
I20250623 01:59:38.193393 14510 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250623 01:59:38.228489 19058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46162:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250623 01:59:38.305373 19581 tablet_service.cc:1468] Processing CreateTablet for tablet cd2b859e74054deba2269b80565922b5 (DEFAULT_TABLE table=TestTable [id=a88940bd65394b32b4f1414fe9248596]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 01:59:38.308234 19581 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd2b859e74054deba2269b80565922b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:59:38.310388 19181 tablet_service.cc:1468] Processing CreateTablet for tablet cd2b859e74054deba2269b80565922b5 (DEFAULT_TABLE table=TestTable [id=a88940bd65394b32b4f1414fe9248596]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 01:59:38.311288 19314 tablet_service.cc:1468] Processing CreateTablet for tablet cd2b859e74054deba2269b80565922b5 (DEFAULT_TABLE table=TestTable [id=a88940bd65394b32b4f1414fe9248596]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 01:59:38.312152 19181 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd2b859e74054deba2269b80565922b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:59:38.312654 19314 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd2b859e74054deba2269b80565922b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:59:38.332970 19799 tablet_bootstrap.cc:492] T cd2b859e74054deba2269b80565922b5 P 854ed4ef070a4ebcbe1be48b70110323: Bootstrap starting.
I20250623 01:59:38.335584 19800 tablet_bootstrap.cc:492] T cd2b859e74054deba2269b80565922b5 P 2d541983e31e4a8da7b727adfcc6d628: Bootstrap starting.
I20250623 01:59:38.339438 19801 tablet_bootstrap.cc:492] T cd2b859e74054deba2269b80565922b5 P e78be0fbcab4466398aa66cf7b1dfb01: Bootstrap starting.
I20250623 01:59:38.342538 19800 tablet_bootstrap.cc:654] T cd2b859e74054deba2269b80565922b5 P 2d541983e31e4a8da7b727adfcc6d628: Neither blocks nor log segments found. Creating new log.
I20250623 01:59:38.342525 19799 tablet_bootstrap.cc:654] T cd2b859e74054deba2269b80565922b5 P 854ed4ef070a4ebcbe1be48b70110323: Neither blocks nor log segments found. Creating new log.
I20250623 01:59:38.344550 19801 tablet_bootstrap.cc:654] T cd2b859e74054deba2269b80565922b5 P e78be0fbcab4466398aa66cf7b1dfb01: Neither blocks nor log segments found. Creating new log.
I20250623 01:59:38.344789 19799 log.cc:826] T cd2b859e74054deba2269b80565922b5 P 854ed4ef070a4ebcbe1be48b70110323: Log is configured to *not* fsync() on all Append() calls
I20250623 01:59:38.344789 19800 log.cc:826] T cd2b859e74054deba2269b80565922b5 P 2d541983e31e4a8da7b727adfcc6d628: Log is configured to *not* fsync() on all Append() calls
I20250623 01:59:38.346339 19801 log.cc:826] T cd2b859e74054deba2269b80565922b5 P e78be0fbcab4466398aa66cf7b1dfb01: Log is configured to *not* fsync() on all Append() calls
I20250623 01:59:38.358655 19799 tablet_bootstrap.cc:492] T cd2b859e74054deba2269b80565922b5 P 854ed4ef070a4ebcbe1be48b70110323: No bootstrap required, opened a new log
I20250623 01:59:38.359277 19799 ts_tablet_manager.cc:1397] T cd2b859e74054deba2269b80565922b5 P 854ed4ef070a4ebcbe1be48b70110323: Time spent bootstrapping tablet: real 0.027s user 0.015s sys 0.009s
I20250623 01:59:38.359345 19800 tablet_bootstrap.cc:492] T cd2b859e74054deba2269b80565922b5 P 2d541983e31e4a8da7b727adfcc6d628: No bootstrap required, opened a new log
I20250623 01:59:38.359445 19801 tablet_bootstrap.cc:492] T cd2b859e74054deba2269b80565922b5 P e78be0fbcab4466398aa66cf7b1dfb01: No bootstrap required, opened a new log
I20250623 01:59:38.360035 19801 ts_tablet_manager.cc:1397] T cd2b859e74054deba2269b80565922b5 P e78be0fbcab4466398aa66cf7b1dfb01: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.009s
I20250623 01:59:38.359900 19800 ts_tablet_manager.cc:1397] T cd2b859e74054deba2269b80565922b5 P 2d541983e31e4a8da7b727adfcc6d628: Time spent bootstrapping tablet: real 0.025s user 0.017s sys 0.003s
I20250623 01:59:38.387176 19801 raft_consensus.cc:357] T cd2b859e74054deba2269b80565922b5 P e78be0fbcab4466398aa66cf7b1dfb01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d541983e31e4a8da7b727adfcc6d628" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 39941 } } peers { permanent_uuid: "e78be0fbcab4466398aa66cf7b1dfb01" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44657 } } peers { permanent_uuid: "854ed4ef070a4ebcbe1be48b70110323" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 35343 } }
I20250623 01:59:38.387188 19800 raft_consensus.cc:357] T cd2b859e74054deba2269b80565922b5 P 2d541983e31e4a8da7b727adfcc6d628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d541983e31e4a8da7b727adfcc6d628" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 39941 } } peers { permanent_uuid: "e78be0fbcab4466398aa66cf7b1dfb01" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44657 } } peers { permanent_uuid: "854ed4ef070a4ebcbe1be48b70110323" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 35343 } }
I20250623 01:59:38.388082 19800 raft_consensus.cc:383] T cd2b859e74054deba2269b80565922b5 P 2d541983e31e4a8da7b727adfcc6d628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:59:38.388284 19801 raft_consensus.cc:383] T cd2b859e74054deba2269b80565922b5 P e78be0fbcab4466398aa66cf7b1dfb01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:59:38.388433 19800 raft_consensus.cc:738] T cd2b859e74054deba2269b80565922b5 P 2d541983e31e4a8da7b727adfcc6d628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d541983e31e4a8da7b727adfcc6d628, State: Initialized, Role: FOLLOWER
I20250623 01:59:38.388661 19801 raft_consensus.cc:738] T cd2b859e74054deba2269b80565922b5 P e78be0fbcab4466398aa66cf7b1dfb01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e78be0fbcab4466398aa66cf7b1dfb01, State: Initialized, Role: FOLLOWER
I20250623 01:59:38.389359 19800 consensus_queue.cc:260] T cd2b859e74054deba2269b80565922b5 P 2d541983e31e4a8da7b727adfcc6d628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d541983e31e4a8da7b727adfcc6d628" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 39941 } } peers { permanent_uuid: "e78be0fbcab4466398aa66cf7b1dfb01" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44657 } } peers { permanent_uuid: "854ed4ef070a4ebcbe1be48b70110323" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 35343 } }
I20250623 01:59:38.389551 19801 consensus_queue.cc:260] T cd2b859e74054deba2269b80565922b5 P e78be0fbcab4466398aa66cf7b1dfb01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d541983e31e4a8da7b727adfcc6d628" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 39941 } } peers { permanent_uuid: "e78be0fbcab4466398aa66cf7b1dfb01" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44657 } } peers { permanent_uuid: "854ed4ef070a4ebcbe1be48b70110323" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 35343 } }
I20250623 01:59:38.391767 19799 raft_consensus.cc:357] T cd2b859e74054deba2269b80565922b5 P 854ed4ef070a4ebcbe1be48b70110323 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d541983e31e4a8da7b727adfcc6d628" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 39941 } } peers { permanent_uuid: "e78be0fbcab4466398aa66cf7b1dfb01" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44657 } } peers { permanent_uuid: "854ed4ef070a4ebcbe1be48b70110323" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 35343 } }
I20250623 01:59:38.392715 19799 raft_consensus.cc:383] T cd2b859e74054deba2269b80565922b5 P 854ed4ef070a4ebcbe1be48b70110323 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:59:38.393054 19799 raft_consensus.cc:738] T cd2b859e74054deba2269b80565922b5 P 854ed4ef070a4ebcbe1be48b70110323 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 854ed4ef070a4ebcbe1be48b70110323, State: Initialized, Role: FOLLOWER
I20250623 01:59:38.394306 19800 ts_tablet_manager.cc:1428] T cd2b859e74054deba2269b80565922b5 P 2d541983e31e4a8da7b727adfcc6d628: Time spent starting tablet: real 0.034s user 0.028s sys 0.004s
I20250623 01:59:38.394714 19801 ts_tablet_manager.cc:1428] T cd2b859e74054deba2269b80565922b5 P e78be0fbcab4466398aa66cf7b1dfb01: Time spent starting tablet: real 0.034s user 0.018s sys 0.014s
I20250623 01:59:38.395882 19799 consensus_queue.cc:260] T cd2b859e74054deba2269b80565922b5 P 854ed4ef070a4ebcbe1be48b70110323 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d541983e31e4a8da7b727adfcc6d628" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 39941 } } peers { permanent_uuid: "e78be0fbcab4466398aa66cf7b1dfb01" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44657 } } peers { permanent_uuid: "854ed4ef070a4ebcbe1be48b70110323" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 35343 } }
I20250623 01:59:38.400028 19799 ts_tablet_manager.cc:1428] T cd2b859e74054deba2269b80565922b5 P 854ed4ef070a4ebcbe1be48b70110323: Time spent starting tablet: real 0.040s user 0.028s sys 0.003s
I20250623 01:59:38.418699 19807 raft_consensus.cc:491] T cd2b859e74054deba2269b80565922b5 P 854ed4ef070a4ebcbe1be48b70110323 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:38.419291 19807 raft_consensus.cc:513] T cd2b859e74054deba2269b80565922b5 P 854ed4ef070a4ebcbe1be48b70110323 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d541983e31e4a8da7b727adfcc6d628" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 39941 } } peers { permanent_uuid: "e78be0fbcab4466398aa66cf7b1dfb01" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44657 } } peers { permanent_uuid: "854ed4ef070a4ebcbe1be48b70110323" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 35343 } }
I20250623 01:59:38.424347 19807 leader_election.cc:290] T cd2b859e74054deba2269b80565922b5 P 854ed4ef070a4ebcbe1be48b70110323 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d541983e31e4a8da7b727adfcc6d628 (127.14.43.132:39941), e78be0fbcab4466398aa66cf7b1dfb01 (127.14.43.129:44657)
W20250623 01:59:38.425968 19647 tablet.cc:2378] T cd2b859e74054deba2269b80565922b5 P 2d541983e31e4a8da7b727adfcc6d628: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250623 01:59:38.436278 19201 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd2b859e74054deba2269b80565922b5" candidate_uuid: "854ed4ef070a4ebcbe1be48b70110323" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e78be0fbcab4466398aa66cf7b1dfb01" is_pre_election: true
I20250623 01:59:38.436980 19201 raft_consensus.cc:2466] T cd2b859e74054deba2269b80565922b5 P e78be0fbcab4466398aa66cf7b1dfb01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 854ed4ef070a4ebcbe1be48b70110323 in term 0.
I20250623 01:59:38.438261 19270 leader_election.cc:304] T cd2b859e74054deba2269b80565922b5 P 854ed4ef070a4ebcbe1be48b70110323 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 854ed4ef070a4ebcbe1be48b70110323, e78be0fbcab4466398aa66cf7b1dfb01; no voters:
I20250623 01:59:38.438721 19601 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd2b859e74054deba2269b80565922b5" candidate_uuid: "854ed4ef070a4ebcbe1be48b70110323" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d541983e31e4a8da7b727adfcc6d628" is_pre_election: true
I20250623 01:59:38.439095 19807 raft_consensus.cc:2802] T cd2b859e74054deba2269b80565922b5 P 854ed4ef070a4ebcbe1be48b70110323 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:59:38.439438 19807 raft_consensus.cc:491] T cd2b859e74054deba2269b80565922b5 P 854ed4ef070a4ebcbe1be48b70110323 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:38.439682 19807 raft_consensus.cc:3058] T cd2b859e74054deba2269b80565922b5 P 854ed4ef070a4ebcbe1be48b70110323 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:59:38.439571 19601 raft_consensus.cc:2466] T cd2b859e74054deba2269b80565922b5 P 2d541983e31e4a8da7b727adfcc6d628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 854ed4ef070a4ebcbe1be48b70110323 in term 0.
I20250623 01:59:38.444015 19807 raft_consensus.cc:513] T cd2b859e74054deba2269b80565922b5 P 854ed4ef070a4ebcbe1be48b70110323 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d541983e31e4a8da7b727adfcc6d628" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 39941 } } peers { permanent_uuid: "e78be0fbcab4466398aa66cf7b1dfb01" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44657 } } peers { permanent_uuid: "854ed4ef070a4ebcbe1be48b70110323" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 35343 } }
I20250623 01:59:38.445325 19807 leader_election.cc:290] T cd2b859e74054deba2269b80565922b5 P 854ed4ef070a4ebcbe1be48b70110323 [CANDIDATE]: Term 1 election: Requested vote from peers 2d541983e31e4a8da7b727adfcc6d628 (127.14.43.132:39941), e78be0fbcab4466398aa66cf7b1dfb01 (127.14.43.129:44657)
I20250623 01:59:38.446194 19601 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd2b859e74054deba2269b80565922b5" candidate_uuid: "854ed4ef070a4ebcbe1be48b70110323" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d541983e31e4a8da7b727adfcc6d628"
I20250623 01:59:38.446378 19201 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd2b859e74054deba2269b80565922b5" candidate_uuid: "854ed4ef070a4ebcbe1be48b70110323" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e78be0fbcab4466398aa66cf7b1dfb01"
I20250623 01:59:38.446590 19601 raft_consensus.cc:3058] T cd2b859e74054deba2269b80565922b5 P 2d541983e31e4a8da7b727adfcc6d628 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:59:38.446899 19201 raft_consensus.cc:3058] T cd2b859e74054deba2269b80565922b5 P e78be0fbcab4466398aa66cf7b1dfb01 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:59:38.451036 19601 raft_consensus.cc:2466] T cd2b859e74054deba2269b80565922b5 P 2d541983e31e4a8da7b727adfcc6d628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 854ed4ef070a4ebcbe1be48b70110323 in term 1.
I20250623 01:59:38.451946 19269 leader_election.cc:304] T cd2b859e74054deba2269b80565922b5 P 854ed4ef070a4ebcbe1be48b70110323 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d541983e31e4a8da7b727adfcc6d628, 854ed4ef070a4ebcbe1be48b70110323; no voters:
I20250623 01:59:38.452565 19807 raft_consensus.cc:2802] T cd2b859e74054deba2269b80565922b5 P 854ed4ef070a4ebcbe1be48b70110323 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:59:38.453467 19201 raft_consensus.cc:2466] T cd2b859e74054deba2269b80565922b5 P e78be0fbcab4466398aa66cf7b1dfb01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 854ed4ef070a4ebcbe1be48b70110323 in term 1.
I20250623 01:59:38.454371 19807 raft_consensus.cc:695] T cd2b859e74054deba2269b80565922b5 P 854ed4ef070a4ebcbe1be48b70110323 [term 1 LEADER]: Becoming Leader. State: Replica: 854ed4ef070a4ebcbe1be48b70110323, State: Running, Role: LEADER
I20250623 01:59:38.455585 19807 consensus_queue.cc:237] T cd2b859e74054deba2269b80565922b5 P 854ed4ef070a4ebcbe1be48b70110323 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d541983e31e4a8da7b727adfcc6d628" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 39941 } } peers { permanent_uuid: "e78be0fbcab4466398aa66cf7b1dfb01" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44657 } } peers { permanent_uuid: "854ed4ef070a4ebcbe1be48b70110323" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 35343 } }
I20250623 01:59:38.468497 19057 catalog_manager.cc:5582] T cd2b859e74054deba2269b80565922b5 P 854ed4ef070a4ebcbe1be48b70110323 reported cstate change: term changed from 0 to 1, leader changed from <none> to 854ed4ef070a4ebcbe1be48b70110323 (127.14.43.130). New cstate: current_term: 1 leader_uuid: "854ed4ef070a4ebcbe1be48b70110323" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d541983e31e4a8da7b727adfcc6d628" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e78be0fbcab4466398aa66cf7b1dfb01" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 44657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "854ed4ef070a4ebcbe1be48b70110323" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 35343 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:38.506206 14510 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250623 01:59:38.510084 14510 ts_itest-base.cc:246] Waiting for 1 tablets on tserver e78be0fbcab4466398aa66cf7b1dfb01 to finish bootstrapping
I20250623 01:59:38.523033 14510 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 854ed4ef070a4ebcbe1be48b70110323 to finish bootstrapping
I20250623 01:59:38.532969 14510 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 2d541983e31e4a8da7b727adfcc6d628 to finish bootstrapping
W20250623 01:59:38.542336 19380 tablet.cc:2378] T cd2b859e74054deba2269b80565922b5 P 854ed4ef070a4ebcbe1be48b70110323: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250623 01:59:38.543412 14510 test_util.cc:276] Using random seed: -1871175395
I20250623 01:59:38.566777 14510 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20250623 01:59:38.567678 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 19117
W20250623 01:59:38.598171 19270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.43.129:44657: connect: Connection refused (error 111)
I20250623 01:59:38.598871 19601 raft_consensus.cc:1273] T cd2b859e74054deba2269b80565922b5 P 2d541983e31e4a8da7b727adfcc6d628 [term 1 FOLLOWER]: Refusing update from remote peer 854ed4ef070a4ebcbe1be48b70110323: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250623 01:59:38.600524 19807 consensus_queue.cc:1035] T cd2b859e74054deba2269b80565922b5 P 854ed4ef070a4ebcbe1be48b70110323 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d541983e31e4a8da7b727adfcc6d628" member_type: VOTER last_known_addr { host: "127.14.43.132" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 01:59:38.602360 19270 consensus_peers.cc:487] T cd2b859e74054deba2269b80565922b5 P 854ed4ef070a4ebcbe1be48b70110323 -> Peer e78be0fbcab4466398aa66cf7b1dfb01 (127.14.43.129:44657): Couldn't send request to peer e78be0fbcab4466398aa66cf7b1dfb01. Status: Network error: Client connection negotiation failed: client connection to 127.14.43.129:44657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250623 01:59:39.184489 19780 heartbeater.cc:499] Master 127.14.43.190:32971 was elected leader, sending a full tablet report...
W20250623 01:59:40.724601 19270 consensus_peers.cc:487] T cd2b859e74054deba2269b80565922b5 P 854ed4ef070a4ebcbe1be48b70110323 -> Peer e78be0fbcab4466398aa66cf7b1dfb01 (127.14.43.129:44657): Couldn't send request to peer e78be0fbcab4466398aa66cf7b1dfb01. Status: Network error: Client connection negotiation failed: client connection to 127.14.43.129:44657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250623 01:59:41.156504 19581 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250623 01:59:41.179801 19314 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250623 01:59:41.206243 19714 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250623 01:59:41.219770 19448 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250623 01:59:42.941350 19714 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250623 01:59:42.944319 19581 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250623 01:59:42.963897 19448 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250623 01:59:43.004173 19314 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
W20250623 01:59:43.163239 19270 consensus_peers.cc:487] T cd2b859e74054deba2269b80565922b5 P 854ed4ef070a4ebcbe1be48b70110323 -> Peer e78be0fbcab4466398aa66cf7b1dfb01 (127.14.43.129:44657): Couldn't send request to peer e78be0fbcab4466398aa66cf7b1dfb01. Status: Network error: Client connection negotiation failed: client connection to 127.14.43.129:44657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250623 01:59:43.616092 19270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.43.129:44657: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20250623 01:59:44.285807 19375 debug-util.cc:398] Leaking SignalData structure 0x7b08000d5520 after lost signal to thread 19251
W20250623 01:59:44.286628 19375 debug-util.cc:398] Leaking SignalData structure 0x7b08000e6880 after lost signal to thread 19378
W20250623 01:59:45.341189 19270 consensus_peers.cc:487] T cd2b859e74054deba2269b80565922b5 P 854ed4ef070a4ebcbe1be48b70110323 -> Peer e78be0fbcab4466398aa66cf7b1dfb01 (127.14.43.129:44657): Couldn't send request to peer e78be0fbcab4466398aa66cf7b1dfb01. Status: Network error: Client connection negotiation failed: client connection to 127.14.43.129:44657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250623 01:59:45.505617 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 19250
I20250623 01:59:45.539044 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 19383
I20250623 01:59:45.557476 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 19517
I20250623 01:59:45.583124 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 19650
I20250623 01:59:45.604524 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 19025
2025-06-23T01:59:45Z chronyd exiting
[ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4 (17845 ms)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest (17845 ms total)
[----------] 1 test from ListTableCliSimpleParamTest
[ RUN ] ListTableCliSimpleParamTest.TestListTables/2
I20250623 01:59:45.664156 14510 test_util.cc:276] Using random seed: -1864054655
I20250623 01:59:45.668270 14510 ts_itest-base.cc:115] Starting cluster with:
I20250623 01:59:45.668434 14510 ts_itest-base.cc:116] --------------
I20250623 01:59:45.668578 14510 ts_itest-base.cc:117] 1 tablet servers
I20250623 01:59:45.668709 14510 ts_itest-base.cc:118] 1 replicas per TS
I20250623 01:59:45.668855 14510 ts_itest-base.cc:119] --------------
2025-06-23T01:59:45Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-23T01:59:45Z Disabled control of system clock
I20250623 01:59:45.710162 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.43.190:38945
--webserver_interface=127.14.43.190
--webserver_port=0
--builtin_ntp_servers=127.14.43.148:36851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.43.190:38945 with env {}
W20250623 01:59:46.003245 19913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:59:46.003862 19913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:59:46.004329 19913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:59:46.036031 19913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250623 01:59:46.036361 19913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:59:46.036616 19913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250623 01:59:46.036842 19913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250623 01:59:46.072049 19913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:36851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.43.190:38945
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.43.190:38945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.14.43.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:59:46.073364 19913 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:59:46.075014 19913 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:59:46.089269 19919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:59:46.089926 19920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:59:46.089987 19922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:59:46.090521 19913 server_base.cc:1048] running on GCE node
I20250623 01:59:47.193779 19913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:59:47.196444 19913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:59:47.197793 19913 hybrid_clock.cc:648] HybridClock initialized: now 1750643987197753 us; error 54 us; skew 500 ppm
I20250623 01:59:47.198616 19913 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:59:47.204850 19913 webserver.cc:469] Webserver started at http://127.14.43.190:38225/ using document root <none> and password file <none>
I20250623 01:59:47.205768 19913 fs_manager.cc:362] Metadata directory not provided
I20250623 01:59:47.205965 19913 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:59:47.206463 19913 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:59:47.211024 19913 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "18c652fc8ff542beb0907f1b198a006f"
format_stamp: "Formatted at 2025-06-23 01:59:47 on dist-test-slave-9rpg"
I20250623 01:59:47.212127 19913 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "18c652fc8ff542beb0907f1b198a006f"
format_stamp: "Formatted at 2025-06-23 01:59:47 on dist-test-slave-9rpg"
I20250623 01:59:47.219406 19913 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250623 01:59:47.224853 19929 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:59:47.225872 19913 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250623 01:59:47.226215 19913 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
uuid: "18c652fc8ff542beb0907f1b198a006f"
format_stamp: "Formatted at 2025-06-23 01:59:47 on dist-test-slave-9rpg"
I20250623 01:59:47.226567 19913 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:59:47.279436 19913 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:59:47.280925 19913 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:59:47.281371 19913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:59:47.349442 19913 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.190:38945
I20250623 01:59:47.349524 19980 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.190:38945 every 8 connection(s)
I20250623 01:59:47.352094 19913 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250623 01:59:47.357287 19981 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:59:47.357699 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 19913
I20250623 01:59:47.358253 14510 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750643853547882-14510-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250623 01:59:47.379873 19981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18c652fc8ff542beb0907f1b198a006f: Bootstrap starting.
I20250623 01:59:47.385699 19981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18c652fc8ff542beb0907f1b198a006f: Neither blocks nor log segments found. Creating new log.
I20250623 01:59:47.387435 19981 log.cc:826] T 00000000000000000000000000000000 P 18c652fc8ff542beb0907f1b198a006f: Log is configured to *not* fsync() on all Append() calls
I20250623 01:59:47.391717 19981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18c652fc8ff542beb0907f1b198a006f: No bootstrap required, opened a new log
I20250623 01:59:47.408617 19981 raft_consensus.cc:357] T 00000000000000000000000000000000 P 18c652fc8ff542beb0907f1b198a006f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c652fc8ff542beb0907f1b198a006f" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 38945 } }
I20250623 01:59:47.409435 19981 raft_consensus.cc:383] T 00000000000000000000000000000000 P 18c652fc8ff542beb0907f1b198a006f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:59:47.409752 19981 raft_consensus.cc:738] T 00000000000000000000000000000000 P 18c652fc8ff542beb0907f1b198a006f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18c652fc8ff542beb0907f1b198a006f, State: Initialized, Role: FOLLOWER
I20250623 01:59:47.410530 19981 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18c652fc8ff542beb0907f1b198a006f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c652fc8ff542beb0907f1b198a006f" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 38945 } }
I20250623 01:59:47.410997 19981 raft_consensus.cc:397] T 00000000000000000000000000000000 P 18c652fc8ff542beb0907f1b198a006f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250623 01:59:47.411217 19981 raft_consensus.cc:491] T 00000000000000000000000000000000 P 18c652fc8ff542beb0907f1b198a006f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250623 01:59:47.411470 19981 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 18c652fc8ff542beb0907f1b198a006f [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:59:47.415377 19981 raft_consensus.cc:513] T 00000000000000000000000000000000 P 18c652fc8ff542beb0907f1b198a006f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c652fc8ff542beb0907f1b198a006f" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 38945 } }
I20250623 01:59:47.416056 19981 leader_election.cc:304] T 00000000000000000000000000000000 P 18c652fc8ff542beb0907f1b198a006f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18c652fc8ff542beb0907f1b198a006f; no voters:
I20250623 01:59:47.417874 19981 leader_election.cc:290] T 00000000000000000000000000000000 P 18c652fc8ff542beb0907f1b198a006f [CANDIDATE]: Term 1 election: Requested vote from peers
I20250623 01:59:47.418622 19986 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 18c652fc8ff542beb0907f1b198a006f [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:59:47.420620 19986 raft_consensus.cc:695] T 00000000000000000000000000000000 P 18c652fc8ff542beb0907f1b198a006f [term 1 LEADER]: Becoming Leader. State: Replica: 18c652fc8ff542beb0907f1b198a006f, State: Running, Role: LEADER
I20250623 01:59:47.421329 19986 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18c652fc8ff542beb0907f1b198a006f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c652fc8ff542beb0907f1b198a006f" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 38945 } }
I20250623 01:59:47.422355 19981 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18c652fc8ff542beb0907f1b198a006f [sys.catalog]: configured and running, proceeding with master startup.
I20250623 01:59:47.430593 19988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18c652fc8ff542beb0907f1b198a006f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18c652fc8ff542beb0907f1b198a006f. Latest consensus state: current_term: 1 leader_uuid: "18c652fc8ff542beb0907f1b198a006f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c652fc8ff542beb0907f1b198a006f" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 38945 } } }
I20250623 01:59:47.431259 19988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18c652fc8ff542beb0907f1b198a006f [sys.catalog]: This master's current role is: LEADER
I20250623 01:59:47.432788 19987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18c652fc8ff542beb0907f1b198a006f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18c652fc8ff542beb0907f1b198a006f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c652fc8ff542beb0907f1b198a006f" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 38945 } } }
I20250623 01:59:47.433501 19987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18c652fc8ff542beb0907f1b198a006f [sys.catalog]: This master's current role is: LEADER
I20250623 01:59:47.436935 19994 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250623 01:59:47.447122 19994 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250623 01:59:47.463186 19994 catalog_manager.cc:1349] Generated new cluster ID: 9624b89780b34352af6d4cb52926a420
I20250623 01:59:47.463428 19994 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250623 01:59:47.481299 19994 catalog_manager.cc:1372] Generated new certificate authority record
I20250623 01:59:47.483268 19994 catalog_manager.cc:1506] Loading token signing keys...
I20250623 01:59:47.505174 19994 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 18c652fc8ff542beb0907f1b198a006f: Generated new TSK 0
I20250623 01:59:47.506026 19994 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250623 01:59:47.517692 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.43.129:0
--local_ip_for_outbound_sockets=127.14.43.129
--webserver_interface=127.14.43.129
--webserver_port=0
--tserver_master_addrs=127.14.43.190:38945
--builtin_ntp_servers=127.14.43.148:36851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250623 01:59:47.817986 20005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:59:47.818725 20005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:59:47.819497 20005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:59:47.853470 20005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:59:47.854346 20005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.43.129
I20250623 01:59:47.891287 20005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:36851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.43.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.14.43.129
--webserver_port=0
--tserver_master_addrs=127.14.43.190:38945
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.43.129
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:59:47.892565 20005 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:59:47.894294 20005 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:59:47.911111 20012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:59:47.911255 20011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:59:47.913420 20005 server_base.cc:1048] running on GCE node
W20250623 01:59:47.911334 20014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:59:48.998126 20005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:59:49.000921 20005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:59:49.002416 20005 hybrid_clock.cc:648] HybridClock initialized: now 1750643989002384 us; error 43 us; skew 500 ppm
I20250623 01:59:49.003448 20005 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:59:49.013779 20005 webserver.cc:469] Webserver started at http://127.14.43.129:45521/ using document root <none> and password file <none>
I20250623 01:59:49.014997 20005 fs_manager.cc:362] Metadata directory not provided
I20250623 01:59:49.015264 20005 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:59:49.015843 20005 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:59:49.022406 20005 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "8910e0fe6a724137aa893a32150d67a1"
format_stamp: "Formatted at 2025-06-23 01:59:49 on dist-test-slave-9rpg"
I20250623 01:59:49.023862 20005 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "8910e0fe6a724137aa893a32150d67a1"
format_stamp: "Formatted at 2025-06-23 01:59:49 on dist-test-slave-9rpg"
I20250623 01:59:49.032699 20005 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.005s sys 0.004s
I20250623 01:59:49.039744 20021 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:59:49.040907 20005 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250623 01:59:49.041286 20005 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "8910e0fe6a724137aa893a32150d67a1"
format_stamp: "Formatted at 2025-06-23 01:59:49 on dist-test-slave-9rpg"
I20250623 01:59:49.041690 20005 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:59:49.119189 20005 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:59:49.120622 20005 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:59:49.121151 20005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:59:49.123828 20005 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:59:49.128129 20005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250623 01:59:49.128334 20005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250623 01:59:49.128577 20005 ts_tablet_manager.cc:610] Registered 0 tablets
I20250623 01:59:49.128721 20005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250623 01:59:49.257939 20005 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.129:45531
I20250623 01:59:49.258082 20133 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.129:45531 every 8 connection(s)
I20250623 01:59:49.260593 20005 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250623 01:59:49.266278 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 20005
I20250623 01:59:49.266808 14510 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750643853547882-14510-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250623 01:59:49.281015 20134 heartbeater.cc:344] Connected to a master server at 127.14.43.190:38945
I20250623 01:59:49.281431 20134 heartbeater.cc:461] Registering TS with master...
I20250623 01:59:49.282428 20134 heartbeater.cc:507] Master 127.14.43.190:38945 requested a full tablet report, sending...
I20250623 01:59:49.285055 19945 ts_manager.cc:194] Registered new tserver with Master: 8910e0fe6a724137aa893a32150d67a1 (127.14.43.129:45531)
I20250623 01:59:49.286427 14510 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250623 01:59:49.287454 19945 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.43.129:45637
I20250623 01:59:49.316076 19945 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49198:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250623 01:59:49.375265 20069 tablet_service.cc:1468] Processing CreateTablet for tablet e1955d215a114954b12a2db515a8cc65 (DEFAULT_TABLE table=TestTable [id=5bd170a54faa44298efadba29f806f2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 01:59:49.377140 20069 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1955d215a114954b12a2db515a8cc65. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:59:49.403699 20149 tablet_bootstrap.cc:492] T e1955d215a114954b12a2db515a8cc65 P 8910e0fe6a724137aa893a32150d67a1: Bootstrap starting.
I20250623 01:59:49.409165 20149 tablet_bootstrap.cc:654] T e1955d215a114954b12a2db515a8cc65 P 8910e0fe6a724137aa893a32150d67a1: Neither blocks nor log segments found. Creating new log.
I20250623 01:59:49.410912 20149 log.cc:826] T e1955d215a114954b12a2db515a8cc65 P 8910e0fe6a724137aa893a32150d67a1: Log is configured to *not* fsync() on all Append() calls
I20250623 01:59:49.415223 20149 tablet_bootstrap.cc:492] T e1955d215a114954b12a2db515a8cc65 P 8910e0fe6a724137aa893a32150d67a1: No bootstrap required, opened a new log
I20250623 01:59:49.415599 20149 ts_tablet_manager.cc:1397] T e1955d215a114954b12a2db515a8cc65 P 8910e0fe6a724137aa893a32150d67a1: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.006s
I20250623 01:59:49.432451 20149 raft_consensus.cc:357] T e1955d215a114954b12a2db515a8cc65 P 8910e0fe6a724137aa893a32150d67a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8910e0fe6a724137aa893a32150d67a1" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45531 } }
I20250623 01:59:49.433018 20149 raft_consensus.cc:383] T e1955d215a114954b12a2db515a8cc65 P 8910e0fe6a724137aa893a32150d67a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:59:49.433254 20149 raft_consensus.cc:738] T e1955d215a114954b12a2db515a8cc65 P 8910e0fe6a724137aa893a32150d67a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8910e0fe6a724137aa893a32150d67a1, State: Initialized, Role: FOLLOWER
I20250623 01:59:49.433916 20149 consensus_queue.cc:260] T e1955d215a114954b12a2db515a8cc65 P 8910e0fe6a724137aa893a32150d67a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8910e0fe6a724137aa893a32150d67a1" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45531 } }
I20250623 01:59:49.434441 20149 raft_consensus.cc:397] T e1955d215a114954b12a2db515a8cc65 P 8910e0fe6a724137aa893a32150d67a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250623 01:59:49.434696 20149 raft_consensus.cc:491] T e1955d215a114954b12a2db515a8cc65 P 8910e0fe6a724137aa893a32150d67a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250623 01:59:49.434978 20149 raft_consensus.cc:3058] T e1955d215a114954b12a2db515a8cc65 P 8910e0fe6a724137aa893a32150d67a1 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:59:49.438947 20149 raft_consensus.cc:513] T e1955d215a114954b12a2db515a8cc65 P 8910e0fe6a724137aa893a32150d67a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8910e0fe6a724137aa893a32150d67a1" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45531 } }
I20250623 01:59:49.439596 20149 leader_election.cc:304] T e1955d215a114954b12a2db515a8cc65 P 8910e0fe6a724137aa893a32150d67a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8910e0fe6a724137aa893a32150d67a1; no voters:
I20250623 01:59:49.441197 20149 leader_election.cc:290] T e1955d215a114954b12a2db515a8cc65 P 8910e0fe6a724137aa893a32150d67a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250623 01:59:49.441568 20151 raft_consensus.cc:2802] T e1955d215a114954b12a2db515a8cc65 P 8910e0fe6a724137aa893a32150d67a1 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:59:49.444120 20151 raft_consensus.cc:695] T e1955d215a114954b12a2db515a8cc65 P 8910e0fe6a724137aa893a32150d67a1 [term 1 LEADER]: Becoming Leader. State: Replica: 8910e0fe6a724137aa893a32150d67a1, State: Running, Role: LEADER
I20250623 01:59:49.444554 20134 heartbeater.cc:499] Master 127.14.43.190:38945 was elected leader, sending a full tablet report...
I20250623 01:59:49.444867 20151 consensus_queue.cc:237] T e1955d215a114954b12a2db515a8cc65 P 8910e0fe6a724137aa893a32150d67a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8910e0fe6a724137aa893a32150d67a1" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45531 } }
I20250623 01:59:49.446358 20149 ts_tablet_manager.cc:1428] T e1955d215a114954b12a2db515a8cc65 P 8910e0fe6a724137aa893a32150d67a1: Time spent starting tablet: real 0.031s user 0.024s sys 0.007s
I20250623 01:59:49.456635 19945 catalog_manager.cc:5582] T e1955d215a114954b12a2db515a8cc65 P 8910e0fe6a724137aa893a32150d67a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8910e0fe6a724137aa893a32150d67a1 (127.14.43.129). New cstate: current_term: 1 leader_uuid: "8910e0fe6a724137aa893a32150d67a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8910e0fe6a724137aa893a32150d67a1" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45531 } health_report { overall_health: HEALTHY } } }
I20250623 01:59:49.473239 14510 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250623 01:59:49.476150 14510 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 8910e0fe6a724137aa893a32150d67a1 to finish bootstrapping
I20250623 01:59:52.035567 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 20005
I20250623 01:59:52.055730 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 19913
2025-06-23T01:59:52Z chronyd exiting
[ OK ] ListTableCliSimpleParamTest.TestListTables/2 (6441 ms)
[----------] 1 test from ListTableCliSimpleParamTest (6441 ms total)
[----------] 1 test from ListTableCliParamTest
[ RUN ] ListTableCliParamTest.ListTabletWithPartitionInfo/4
I20250623 01:59:52.106216 14510 test_util.cc:276] Using random seed: -1857612594
[ OK ] ListTableCliParamTest.ListTabletWithPartitionInfo/4 (11 ms)
[----------] 1 test from ListTableCliParamTest (12 ms total)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest
[ RUN ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0
2025-06-23T01:59:52Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-23T01:59:52Z Disabled control of system clock
I20250623 01:59:52.155151 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.43.190:40161
--webserver_interface=127.14.43.190
--webserver_port=0
--builtin_ntp_servers=127.14.43.148:33689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.43.190:40161 with env {}
W20250623 01:59:52.445091 20177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:59:52.445713 20177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:59:52.446188 20177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:59:52.476920 20177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250623 01:59:52.477221 20177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:59:52.477433 20177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250623 01:59:52.477624 20177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250623 01:59:52.512360 20177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:33689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.43.190:40161
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.43.190:40161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.43.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:59:52.513594 20177 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:59:52.515355 20177 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:59:52.530656 20184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:59:52.531157 20183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:59:52.532176 20177 server_base.cc:1048] running on GCE node
W20250623 01:59:52.531435 20186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:59:53.614835 20177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:59:53.617681 20177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:59:53.619084 20177 hybrid_clock.cc:648] HybridClock initialized: now 1750643993619052 us; error 52 us; skew 500 ppm
I20250623 01:59:53.619863 20177 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:59:53.626168 20177 webserver.cc:469] Webserver started at http://127.14.43.190:32893/ using document root <none> and password file <none>
I20250623 01:59:53.627069 20177 fs_manager.cc:362] Metadata directory not provided
I20250623 01:59:53.627252 20177 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:59:53.627660 20177 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:59:53.632122 20177 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/master-0/data/instance:
uuid: "9248be0caf234ab9bf2873a3317dd8ae"
format_stamp: "Formatted at 2025-06-23 01:59:53 on dist-test-slave-9rpg"
I20250623 01:59:53.633158 20177 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/master-0/wal/instance:
uuid: "9248be0caf234ab9bf2873a3317dd8ae"
format_stamp: "Formatted at 2025-06-23 01:59:53 on dist-test-slave-9rpg"
I20250623 01:59:53.639976 20177 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20250623 01:59:53.645340 20193 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:59:53.646350 20177 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250623 01:59:53.646669 20177 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/master-0/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/master-0/wal
uuid: "9248be0caf234ab9bf2873a3317dd8ae"
format_stamp: "Formatted at 2025-06-23 01:59:53 on dist-test-slave-9rpg"
I20250623 01:59:53.646961 20177 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:59:53.698823 20177 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:59:53.700318 20177 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:59:53.700733 20177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:59:53.767359 20177 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.190:40161
I20250623 01:59:53.767464 20244 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.190:40161 every 8 connection(s)
I20250623 01:59:53.769915 20177 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/master-0/data/info.pb
I20250623 01:59:53.772172 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 20177
I20250623 01:59:53.772718 14510 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/master-0/wal/instance
I20250623 01:59:53.775974 20245 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:59:53.799106 20245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9248be0caf234ab9bf2873a3317dd8ae: Bootstrap starting.
I20250623 01:59:53.804348 20245 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9248be0caf234ab9bf2873a3317dd8ae: Neither blocks nor log segments found. Creating new log.
I20250623 01:59:53.805950 20245 log.cc:826] T 00000000000000000000000000000000 P 9248be0caf234ab9bf2873a3317dd8ae: Log is configured to *not* fsync() on all Append() calls
I20250623 01:59:53.810237 20245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9248be0caf234ab9bf2873a3317dd8ae: No bootstrap required, opened a new log
I20250623 01:59:53.826948 20245 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9248be0caf234ab9bf2873a3317dd8ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9248be0caf234ab9bf2873a3317dd8ae" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 40161 } }
I20250623 01:59:53.827625 20245 raft_consensus.cc:383] T 00000000000000000000000000000000 P 9248be0caf234ab9bf2873a3317dd8ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:59:53.827879 20245 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9248be0caf234ab9bf2873a3317dd8ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9248be0caf234ab9bf2873a3317dd8ae, State: Initialized, Role: FOLLOWER
I20250623 01:59:53.828616 20245 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9248be0caf234ab9bf2873a3317dd8ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9248be0caf234ab9bf2873a3317dd8ae" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 40161 } }
I20250623 01:59:53.829207 20245 raft_consensus.cc:397] T 00000000000000000000000000000000 P 9248be0caf234ab9bf2873a3317dd8ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250623 01:59:53.829517 20245 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9248be0caf234ab9bf2873a3317dd8ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250623 01:59:53.829797 20245 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9248be0caf234ab9bf2873a3317dd8ae [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:59:53.833750 20245 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9248be0caf234ab9bf2873a3317dd8ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9248be0caf234ab9bf2873a3317dd8ae" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 40161 } }
I20250623 01:59:53.834442 20245 leader_election.cc:304] T 00000000000000000000000000000000 P 9248be0caf234ab9bf2873a3317dd8ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9248be0caf234ab9bf2873a3317dd8ae; no voters:
I20250623 01:59:53.835995 20245 leader_election.cc:290] T 00000000000000000000000000000000 P 9248be0caf234ab9bf2873a3317dd8ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20250623 01:59:53.836638 20250 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9248be0caf234ab9bf2873a3317dd8ae [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:59:53.839272 20250 raft_consensus.cc:695] T 00000000000000000000000000000000 P 9248be0caf234ab9bf2873a3317dd8ae [term 1 LEADER]: Becoming Leader. State: Replica: 9248be0caf234ab9bf2873a3317dd8ae, State: Running, Role: LEADER
I20250623 01:59:53.839910 20250 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9248be0caf234ab9bf2873a3317dd8ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9248be0caf234ab9bf2873a3317dd8ae" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 40161 } }
I20250623 01:59:53.840503 20245 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9248be0caf234ab9bf2873a3317dd8ae [sys.catalog]: configured and running, proceeding with master startup.
I20250623 01:59:53.850270 20251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9248be0caf234ab9bf2873a3317dd8ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9248be0caf234ab9bf2873a3317dd8ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9248be0caf234ab9bf2873a3317dd8ae" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 40161 } } }
I20250623 01:59:53.851002 20251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9248be0caf234ab9bf2873a3317dd8ae [sys.catalog]: This master's current role is: LEADER
I20250623 01:59:53.851173 20252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9248be0caf234ab9bf2873a3317dd8ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9248be0caf234ab9bf2873a3317dd8ae. Latest consensus state: current_term: 1 leader_uuid: "9248be0caf234ab9bf2873a3317dd8ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9248be0caf234ab9bf2873a3317dd8ae" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 40161 } } }
I20250623 01:59:53.851765 20252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9248be0caf234ab9bf2873a3317dd8ae [sys.catalog]: This master's current role is: LEADER
I20250623 01:59:53.854638 20258 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250623 01:59:53.863821 20258 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250623 01:59:53.879447 20258 catalog_manager.cc:1349] Generated new cluster ID: 55eb3c11f13e45cfa5a9a4822c7c64b8
I20250623 01:59:53.879745 20258 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250623 01:59:53.907996 20258 catalog_manager.cc:1372] Generated new certificate authority record
I20250623 01:59:53.909934 20258 catalog_manager.cc:1506] Loading token signing keys...
I20250623 01:59:53.924897 20258 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 9248be0caf234ab9bf2873a3317dd8ae: Generated new TSK 0
I20250623 01:59:53.925961 20258 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250623 01:59:53.939378 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.43.129:0
--local_ip_for_outbound_sockets=127.14.43.129
--webserver_interface=127.14.43.129
--webserver_port=0
--tserver_master_addrs=127.14.43.190:40161
--builtin_ntp_servers=127.14.43.148:33689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20250623 01:59:54.228909 20270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:59:54.229535 20270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:59:54.230223 20270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:59:54.261197 20270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:59:54.262059 20270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.43.129
I20250623 01:59:54.296904 20270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:33689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.43.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.43.129
--webserver_port=0
--tserver_master_addrs=127.14.43.190:40161
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.43.129
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:59:54.298234 20270 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:59:54.299854 20270 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:59:54.316103 20277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:59:54.318481 20276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:59:54.316404 20279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:59:55.392097 20278 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1077 milliseconds
I20250623 01:59:55.392673 20270 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250623 01:59:55.396018 20270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:59:55.398944 20270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:59:55.400498 20270 hybrid_clock.cc:648] HybridClock initialized: now 1750643995400435 us; error 55 us; skew 500 ppm
I20250623 01:59:55.401556 20270 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:59:55.411882 20270 webserver.cc:469] Webserver started at http://127.14.43.129:46049/ using document root <none> and password file <none>
I20250623 01:59:55.412789 20270 fs_manager.cc:362] Metadata directory not provided
I20250623 01:59:55.412993 20270 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:59:55.413437 20270 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:59:55.417749 20270 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-0/data/instance:
uuid: "9f26b3913c2845988766eae8e7bd2a0b"
format_stamp: "Formatted at 2025-06-23 01:59:55 on dist-test-slave-9rpg"
I20250623 01:59:55.418815 20270 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-0/wal/instance:
uuid: "9f26b3913c2845988766eae8e7bd2a0b"
format_stamp: "Formatted at 2025-06-23 01:59:55 on dist-test-slave-9rpg"
I20250623 01:59:55.425849 20270 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250623 01:59:55.431619 20286 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:59:55.432824 20270 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.005s
I20250623 01:59:55.433130 20270 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-0/wal
uuid: "9f26b3913c2845988766eae8e7bd2a0b"
format_stamp: "Formatted at 2025-06-23 01:59:55 on dist-test-slave-9rpg"
I20250623 01:59:55.433458 20270 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:59:55.491634 20270 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:59:55.493077 20270 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:59:55.493506 20270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:59:55.496371 20270 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:59:55.500458 20270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250623 01:59:55.500665 20270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250623 01:59:55.500905 20270 ts_tablet_manager.cc:610] Registered 0 tablets
I20250623 01:59:55.501055 20270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250623 01:59:55.637497 20270 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.129:45459
I20250623 01:59:55.637578 20398 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.129:45459 every 8 connection(s)
I20250623 01:59:55.639990 20270 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-0/data/info.pb
I20250623 01:59:55.645387 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 20270
I20250623 01:59:55.645851 14510 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-0/wal/instance
I20250623 01:59:55.651921 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.43.130:0
--local_ip_for_outbound_sockets=127.14.43.130
--webserver_interface=127.14.43.130
--webserver_port=0
--tserver_master_addrs=127.14.43.190:40161
--builtin_ntp_servers=127.14.43.148:33689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250623 01:59:55.660948 20399 heartbeater.cc:344] Connected to a master server at 127.14.43.190:40161
I20250623 01:59:55.661476 20399 heartbeater.cc:461] Registering TS with master...
I20250623 01:59:55.662740 20399 heartbeater.cc:507] Master 127.14.43.190:40161 requested a full tablet report, sending...
I20250623 01:59:55.665406 20210 ts_manager.cc:194] Registered new tserver with Master: 9f26b3913c2845988766eae8e7bd2a0b (127.14.43.129:45459)
I20250623 01:59:55.667300 20210 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.43.129:36301
W20250623 01:59:55.946467 20403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:59:55.947068 20403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:59:55.947824 20403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:59:55.979347 20403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:59:55.980197 20403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.43.130
I20250623 01:59:56.014752 20403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:33689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.43.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.43.130
--webserver_port=0
--tserver_master_addrs=127.14.43.190:40161
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.43.130
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:59:56.016032 20403 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:59:56.017692 20403 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:59:56.032323 20409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:59:56.671125 20399 heartbeater.cc:499] Master 127.14.43.190:40161 was elected leader, sending a full tablet report...
W20250623 01:59:56.032330 20410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:59:56.033010 20412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:59:57.112634 20411 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1078 milliseconds
I20250623 01:59:57.112741 20403 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250623 01:59:57.113883 20403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:59:57.116434 20403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:59:57.117844 20403 hybrid_clock.cc:648] HybridClock initialized: now 1750643997117817 us; error 44 us; skew 500 ppm
I20250623 01:59:57.118664 20403 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:59:57.125108 20403 webserver.cc:469] Webserver started at http://127.14.43.130:42133/ using document root <none> and password file <none>
I20250623 01:59:57.125983 20403 fs_manager.cc:362] Metadata directory not provided
I20250623 01:59:57.126232 20403 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:59:57.126644 20403 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:59:57.130954 20403 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-1/data/instance:
uuid: "da8eeec4d1184c43942f5579da019e8e"
format_stamp: "Formatted at 2025-06-23 01:59:57 on dist-test-slave-9rpg"
I20250623 01:59:57.131975 20403 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-1/wal/instance:
uuid: "da8eeec4d1184c43942f5579da019e8e"
format_stamp: "Formatted at 2025-06-23 01:59:57 on dist-test-slave-9rpg"
I20250623 01:59:57.138917 20403 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20250623 01:59:57.144482 20419 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:59:57.145670 20403 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250623 01:59:57.145987 20403 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-1/wal
uuid: "da8eeec4d1184c43942f5579da019e8e"
format_stamp: "Formatted at 2025-06-23 01:59:57 on dist-test-slave-9rpg"
I20250623 01:59:57.146377 20403 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:59:57.196956 20403 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:59:57.198465 20403 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:59:57.198889 20403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:59:57.201372 20403 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:59:57.205330 20403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250623 01:59:57.205545 20403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250623 01:59:57.205775 20403 ts_tablet_manager.cc:610] Registered 0 tablets
I20250623 01:59:57.205930 20403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250623 01:59:57.333009 20403 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.130:44559
I20250623 01:59:57.333113 20531 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.130:44559 every 8 connection(s)
I20250623 01:59:57.335578 20403 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-1/data/info.pb
I20250623 01:59:57.339551 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 20403
I20250623 01:59:57.340054 14510 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-1/wal/instance
I20250623 01:59:57.347667 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.43.131:0
--local_ip_for_outbound_sockets=127.14.43.131
--webserver_interface=127.14.43.131
--webserver_port=0
--tserver_master_addrs=127.14.43.190:40161
--builtin_ntp_servers=127.14.43.148:33689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250623 01:59:57.358422 20532 heartbeater.cc:344] Connected to a master server at 127.14.43.190:40161
I20250623 01:59:57.358845 20532 heartbeater.cc:461] Registering TS with master...
I20250623 01:59:57.359830 20532 heartbeater.cc:507] Master 127.14.43.190:40161 requested a full tablet report, sending...
I20250623 01:59:57.361889 20210 ts_manager.cc:194] Registered new tserver with Master: da8eeec4d1184c43942f5579da019e8e (127.14.43.130:44559)
I20250623 01:59:57.363144 20210 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.43.130:42649
W20250623 01:59:57.638231 20536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 01:59:57.638880 20536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 01:59:57.639624 20536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 01:59:57.675640 20536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 01:59:57.676709 20536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.43.131
I20250623 01:59:57.711468 20536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:33689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.43.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.43.131
--webserver_port=0
--tserver_master_addrs=127.14.43.190:40161
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.43.131
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 01:59:57.712776 20536 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 01:59:57.714555 20536 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 01:59:57.730680 20543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:59:58.366281 20532 heartbeater.cc:499] Master 127.14.43.190:40161 was elected leader, sending a full tablet report...
W20250623 01:59:57.730669 20542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 01:59:57.731606 20545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 01:59:57.733398 20536 server_base.cc:1048] running on GCE node
I20250623 01:59:58.813066 20536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 01:59:58.815618 20536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 01:59:58.817090 20536 hybrid_clock.cc:648] HybridClock initialized: now 1750643998817055 us; error 56 us; skew 500 ppm
I20250623 01:59:58.817881 20536 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 01:59:58.824236 20536 webserver.cc:469] Webserver started at http://127.14.43.131:41295/ using document root <none> and password file <none>
I20250623 01:59:58.825084 20536 fs_manager.cc:362] Metadata directory not provided
I20250623 01:59:58.825277 20536 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 01:59:58.825683 20536 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 01:59:58.829963 20536 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-2/data/instance:
uuid: "7395562be1584b47bea13989e88c8b52"
format_stamp: "Formatted at 2025-06-23 01:59:58 on dist-test-slave-9rpg"
I20250623 01:59:58.830983 20536 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-2/wal/instance:
uuid: "7395562be1584b47bea13989e88c8b52"
format_stamp: "Formatted at 2025-06-23 01:59:58 on dist-test-slave-9rpg"
I20250623 01:59:58.837606 20536 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250623 01:59:58.843108 20552 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 01:59:58.844048 20536 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250623 01:59:58.844362 20536 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-2/wal
uuid: "7395562be1584b47bea13989e88c8b52"
format_stamp: "Formatted at 2025-06-23 01:59:58 on dist-test-slave-9rpg"
I20250623 01:59:58.844671 20536 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 01:59:58.897152 20536 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 01:59:58.898605 20536 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 01:59:58.899017 20536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 01:59:58.901408 20536 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 01:59:58.905323 20536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250623 01:59:58.905529 20536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250623 01:59:58.905756 20536 ts_tablet_manager.cc:610] Registered 0 tablets
I20250623 01:59:58.905915 20536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250623 01:59:59.031579 20536 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.131:46727
I20250623 01:59:59.031685 20664 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.131:46727 every 8 connection(s)
I20250623 01:59:59.034006 20536 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-2/data/info.pb
I20250623 01:59:59.036109 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 20536
I20250623 01:59:59.036635 14510 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-2/wal/instance
I20250623 01:59:59.053606 20665 heartbeater.cc:344] Connected to a master server at 127.14.43.190:40161
I20250623 01:59:59.053999 20665 heartbeater.cc:461] Registering TS with master...
I20250623 01:59:59.054958 20665 heartbeater.cc:507] Master 127.14.43.190:40161 requested a full tablet report, sending...
I20250623 01:59:59.057034 20210 ts_manager.cc:194] Registered new tserver with Master: 7395562be1584b47bea13989e88c8b52 (127.14.43.131:46727)
I20250623 01:59:59.058238 20210 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.43.131:56899
I20250623 01:59:59.070266 14510 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250623 01:59:59.098415 14510 test_util.cc:276] Using random seed: -1850620392
I20250623 01:59:59.136240 20210 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56190:
name: "pre_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 01:59:59.138554 20210 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table pre_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 01:59:59.184072 20467 tablet_service.cc:1468] Processing CreateTablet for tablet 9ffd15b775094bbb920b3cb00b356a30 (DEFAULT_TABLE table=pre_rebuild [id=e6971bdb673e42329743f831c9d2e24a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 01:59:59.185868 20467 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ffd15b775094bbb920b3cb00b356a30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:59:59.187464 20600 tablet_service.cc:1468] Processing CreateTablet for tablet 9ffd15b775094bbb920b3cb00b356a30 (DEFAULT_TABLE table=pre_rebuild [id=e6971bdb673e42329743f831c9d2e24a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 01:59:59.189224 20600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ffd15b775094bbb920b3cb00b356a30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:59:59.189127 20334 tablet_service.cc:1468] Processing CreateTablet for tablet 9ffd15b775094bbb920b3cb00b356a30 (DEFAULT_TABLE table=pre_rebuild [id=e6971bdb673e42329743f831c9d2e24a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 01:59:59.190840 20334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ffd15b775094bbb920b3cb00b356a30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 01:59:59.210177 20689 tablet_bootstrap.cc:492] T 9ffd15b775094bbb920b3cb00b356a30 P 7395562be1584b47bea13989e88c8b52: Bootstrap starting.
I20250623 01:59:59.212237 20690 tablet_bootstrap.cc:492] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e: Bootstrap starting.
I20250623 01:59:59.216774 20691 tablet_bootstrap.cc:492] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b: Bootstrap starting.
I20250623 01:59:59.217568 20689 tablet_bootstrap.cc:654] T 9ffd15b775094bbb920b3cb00b356a30 P 7395562be1584b47bea13989e88c8b52: Neither blocks nor log segments found. Creating new log.
I20250623 01:59:59.219295 20690 tablet_bootstrap.cc:654] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e: Neither blocks nor log segments found. Creating new log.
I20250623 01:59:59.219830 20689 log.cc:826] T 9ffd15b775094bbb920b3cb00b356a30 P 7395562be1584b47bea13989e88c8b52: Log is configured to *not* fsync() on all Append() calls
I20250623 01:59:59.221537 20690 log.cc:826] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e: Log is configured to *not* fsync() on all Append() calls
I20250623 01:59:59.223205 20691 tablet_bootstrap.cc:654] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b: Neither blocks nor log segments found. Creating new log.
I20250623 01:59:59.225004 20691 log.cc:826] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b: Log is configured to *not* fsync() on all Append() calls
I20250623 01:59:59.226279 20689 tablet_bootstrap.cc:492] T 9ffd15b775094bbb920b3cb00b356a30 P 7395562be1584b47bea13989e88c8b52: No bootstrap required, opened a new log
I20250623 01:59:59.226788 20689 ts_tablet_manager.cc:1397] T 9ffd15b775094bbb920b3cb00b356a30 P 7395562be1584b47bea13989e88c8b52: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.009s
I20250623 01:59:59.229212 20690 tablet_bootstrap.cc:492] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e: No bootstrap required, opened a new log
I20250623 01:59:59.229693 20690 ts_tablet_manager.cc:1397] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.004s
I20250623 01:59:59.229784 20691 tablet_bootstrap.cc:492] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b: No bootstrap required, opened a new log
I20250623 01:59:59.230258 20691 ts_tablet_manager.cc:1397] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.011s
I20250623 01:59:59.244738 20689 raft_consensus.cc:357] T 9ffd15b775094bbb920b3cb00b356a30 P 7395562be1584b47bea13989e88c8b52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8eeec4d1184c43942f5579da019e8e" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44559 } } peers { permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 } } peers { permanent_uuid: "7395562be1584b47bea13989e88c8b52" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 46727 } }
I20250623 01:59:59.245390 20689 raft_consensus.cc:383] T 9ffd15b775094bbb920b3cb00b356a30 P 7395562be1584b47bea13989e88c8b52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:59:59.245632 20689 raft_consensus.cc:738] T 9ffd15b775094bbb920b3cb00b356a30 P 7395562be1584b47bea13989e88c8b52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7395562be1584b47bea13989e88c8b52, State: Initialized, Role: FOLLOWER
I20250623 01:59:59.246393 20689 consensus_queue.cc:260] T 9ffd15b775094bbb920b3cb00b356a30 P 7395562be1584b47bea13989e88c8b52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8eeec4d1184c43942f5579da019e8e" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44559 } } peers { permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 } } peers { permanent_uuid: "7395562be1584b47bea13989e88c8b52" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 46727 } }
I20250623 01:59:59.249197 20665 heartbeater.cc:499] Master 127.14.43.190:40161 was elected leader, sending a full tablet report...
I20250623 01:59:59.250586 20689 ts_tablet_manager.cc:1428] T 9ffd15b775094bbb920b3cb00b356a30 P 7395562be1584b47bea13989e88c8b52: Time spent starting tablet: real 0.024s user 0.020s sys 0.004s
I20250623 01:59:59.255431 20691 raft_consensus.cc:357] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8eeec4d1184c43942f5579da019e8e" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44559 } } peers { permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 } } peers { permanent_uuid: "7395562be1584b47bea13989e88c8b52" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 46727 } }
I20250623 01:59:59.256317 20691 raft_consensus.cc:383] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:59:59.256608 20691 raft_consensus.cc:738] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f26b3913c2845988766eae8e7bd2a0b, State: Initialized, Role: FOLLOWER
I20250623 01:59:59.257390 20690 raft_consensus.cc:357] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8eeec4d1184c43942f5579da019e8e" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44559 } } peers { permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 } } peers { permanent_uuid: "7395562be1584b47bea13989e88c8b52" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 46727 } }
I20250623 01:59:59.257550 20691 consensus_queue.cc:260] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8eeec4d1184c43942f5579da019e8e" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44559 } } peers { permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 } } peers { permanent_uuid: "7395562be1584b47bea13989e88c8b52" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 46727 } }
I20250623 01:59:59.258240 20690 raft_consensus.cc:383] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 01:59:59.258534 20690 raft_consensus.cc:738] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da8eeec4d1184c43942f5579da019e8e, State: Initialized, Role: FOLLOWER
I20250623 01:59:59.259495 20690 consensus_queue.cc:260] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8eeec4d1184c43942f5579da019e8e" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44559 } } peers { permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 } } peers { permanent_uuid: "7395562be1584b47bea13989e88c8b52" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 46727 } }
I20250623 01:59:59.261672 20691 ts_tablet_manager.cc:1428] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b: Time spent starting tablet: real 0.031s user 0.029s sys 0.001s
I20250623 01:59:59.264328 20690 ts_tablet_manager.cc:1428] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e: Time spent starting tablet: real 0.034s user 0.028s sys 0.004s
W20250623 01:59:59.288427 20666 tablet.cc:2378] T 9ffd15b775094bbb920b3cb00b356a30 P 7395562be1584b47bea13989e88c8b52: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250623 01:59:59.341717 20533 tablet.cc:2378] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250623 01:59:59.398073 20400 tablet.cc:2378] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250623 01:59:59.525417 20697 raft_consensus.cc:491] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:59.525904 20697 raft_consensus.cc:513] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8eeec4d1184c43942f5579da019e8e" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44559 } } peers { permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 } } peers { permanent_uuid: "7395562be1584b47bea13989e88c8b52" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 46727 } }
I20250623 01:59:59.528414 20697 leader_election.cc:290] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f26b3913c2845988766eae8e7bd2a0b (127.14.43.129:45459), 7395562be1584b47bea13989e88c8b52 (127.14.43.131:46727)
I20250623 01:59:59.533739 20696 raft_consensus.cc:491] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:59.534262 20696 raft_consensus.cc:513] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8eeec4d1184c43942f5579da019e8e" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44559 } } peers { permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 } } peers { permanent_uuid: "7395562be1584b47bea13989e88c8b52" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 46727 } }
I20250623 01:59:59.536531 20695 raft_consensus.cc:491] T 9ffd15b775094bbb920b3cb00b356a30 P 7395562be1584b47bea13989e88c8b52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 01:59:59.537071 20695 raft_consensus.cc:513] T 9ffd15b775094bbb920b3cb00b356a30 P 7395562be1584b47bea13989e88c8b52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8eeec4d1184c43942f5579da019e8e" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44559 } } peers { permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 } } peers { permanent_uuid: "7395562be1584b47bea13989e88c8b52" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 46727 } }
I20250623 01:59:59.539992 20696 leader_election.cc:290] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da8eeec4d1184c43942f5579da019e8e (127.14.43.130:44559), 7395562be1584b47bea13989e88c8b52 (127.14.43.131:46727)
I20250623 01:59:59.544255 20620 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ffd15b775094bbb920b3cb00b356a30" candidate_uuid: "da8eeec4d1184c43942f5579da019e8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7395562be1584b47bea13989e88c8b52" is_pre_election: true
I20250623 01:59:59.544545 20354 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ffd15b775094bbb920b3cb00b356a30" candidate_uuid: "da8eeec4d1184c43942f5579da019e8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f26b3913c2845988766eae8e7bd2a0b" is_pre_election: true
I20250623 01:59:59.545029 20620 raft_consensus.cc:2466] T 9ffd15b775094bbb920b3cb00b356a30 P 7395562be1584b47bea13989e88c8b52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da8eeec4d1184c43942f5579da019e8e in term 0.
I20250623 01:59:59.545282 20354 raft_consensus.cc:2466] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da8eeec4d1184c43942f5579da019e8e in term 0.
I20250623 01:59:59.546450 20422 leader_election.cc:304] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7395562be1584b47bea13989e88c8b52, da8eeec4d1184c43942f5579da019e8e; no voters:
I20250623 01:59:59.547329 20697 raft_consensus.cc:2802] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 01:59:59.547725 20697 raft_consensus.cc:491] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 01:59:59.548094 20697 raft_consensus.cc:3058] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:59:59.555310 20695 leader_election.cc:290] T 9ffd15b775094bbb920b3cb00b356a30 P 7395562be1584b47bea13989e88c8b52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da8eeec4d1184c43942f5579da019e8e (127.14.43.130:44559), 9f26b3913c2845988766eae8e7bd2a0b (127.14.43.129:45459)
I20250623 01:59:59.558037 20487 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ffd15b775094bbb920b3cb00b356a30" candidate_uuid: "9f26b3913c2845988766eae8e7bd2a0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da8eeec4d1184c43942f5579da019e8e" is_pre_election: true
I20250623 01:59:59.559185 20697 raft_consensus.cc:513] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8eeec4d1184c43942f5579da019e8e" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44559 } } peers { permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 } } peers { permanent_uuid: "7395562be1584b47bea13989e88c8b52" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 46727 } }
I20250623 01:59:59.560359 20487 raft_consensus.cc:2391] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f26b3913c2845988766eae8e7bd2a0b in current term 1: Already voted for candidate da8eeec4d1184c43942f5579da019e8e in this term.
I20250623 01:59:59.564764 20487 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ffd15b775094bbb920b3cb00b356a30" candidate_uuid: "7395562be1584b47bea13989e88c8b52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da8eeec4d1184c43942f5579da019e8e" is_pre_election: true
I20250623 01:59:59.565151 20697 leader_election.cc:290] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e [CANDIDATE]: Term 1 election: Requested vote from peers 9f26b3913c2845988766eae8e7bd2a0b (127.14.43.129:45459), 7395562be1584b47bea13989e88c8b52 (127.14.43.131:46727)
I20250623 01:59:59.565447 20487 raft_consensus.cc:2391] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7395562be1584b47bea13989e88c8b52 in current term 1: Already voted for candidate da8eeec4d1184c43942f5579da019e8e in this term.
I20250623 01:59:59.566550 20354 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ffd15b775094bbb920b3cb00b356a30" candidate_uuid: "da8eeec4d1184c43942f5579da019e8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f26b3913c2845988766eae8e7bd2a0b"
I20250623 01:59:59.567049 20354 raft_consensus.cc:3058] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:59:59.567521 20620 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ffd15b775094bbb920b3cb00b356a30" candidate_uuid: "da8eeec4d1184c43942f5579da019e8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7395562be1584b47bea13989e88c8b52"
I20250623 01:59:59.568053 20620 raft_consensus.cc:3058] T 9ffd15b775094bbb920b3cb00b356a30 P 7395562be1584b47bea13989e88c8b52 [term 0 FOLLOWER]: Advancing to term 1
I20250623 01:59:59.569664 20353 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ffd15b775094bbb920b3cb00b356a30" candidate_uuid: "7395562be1584b47bea13989e88c8b52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f26b3913c2845988766eae8e7bd2a0b" is_pre_election: true
I20250623 01:59:59.569929 20619 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ffd15b775094bbb920b3cb00b356a30" candidate_uuid: "9f26b3913c2845988766eae8e7bd2a0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7395562be1584b47bea13989e88c8b52" is_pre_election: true
I20250623 01:59:59.571401 20553 leader_election.cc:304] T 9ffd15b775094bbb920b3cb00b356a30 P 7395562be1584b47bea13989e88c8b52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7395562be1584b47bea13989e88c8b52; no voters: 9f26b3913c2845988766eae8e7bd2a0b, da8eeec4d1184c43942f5579da019e8e
I20250623 01:59:59.571728 20289 leader_election.cc:304] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f26b3913c2845988766eae8e7bd2a0b; no voters: 7395562be1584b47bea13989e88c8b52, da8eeec4d1184c43942f5579da019e8e
I20250623 01:59:59.573926 20620 raft_consensus.cc:2466] T 9ffd15b775094bbb920b3cb00b356a30 P 7395562be1584b47bea13989e88c8b52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da8eeec4d1184c43942f5579da019e8e in term 1.
I20250623 01:59:59.573985 20354 raft_consensus.cc:2466] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da8eeec4d1184c43942f5579da019e8e in term 1.
I20250623 01:59:59.574373 20696 raft_consensus.cc:2747] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:59:59.574945 20695 raft_consensus.cc:2747] T 9ffd15b775094bbb920b3cb00b356a30 P 7395562be1584b47bea13989e88c8b52 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 01:59:59.574909 20422 leader_election.cc:304] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7395562be1584b47bea13989e88c8b52, da8eeec4d1184c43942f5579da019e8e; no voters:
I20250623 01:59:59.575639 20697 raft_consensus.cc:2802] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e [term 1 FOLLOWER]: Leader election won for term 1
I20250623 01:59:59.577664 20697 raft_consensus.cc:695] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e [term 1 LEADER]: Becoming Leader. State: Replica: da8eeec4d1184c43942f5579da019e8e, State: Running, Role: LEADER
I20250623 01:59:59.578697 20697 consensus_queue.cc:237] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8eeec4d1184c43942f5579da019e8e" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44559 } } peers { permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 } } peers { permanent_uuid: "7395562be1584b47bea13989e88c8b52" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 46727 } }
I20250623 01:59:59.587700 20210 catalog_manager.cc:5582] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e reported cstate change: term changed from 0 to 1, leader changed from <none> to da8eeec4d1184c43942f5579da019e8e (127.14.43.130). New cstate: current_term: 1 leader_uuid: "da8eeec4d1184c43942f5579da019e8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8eeec4d1184c43942f5579da019e8e" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7395562be1584b47bea13989e88c8b52" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 46727 } health_report { overall_health: UNKNOWN } } }
I20250623 01:59:59.746920 20354 raft_consensus.cc:1273] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [term 1 FOLLOWER]: Refusing update from remote peer da8eeec4d1184c43942f5579da019e8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250623 01:59:59.746891 20620 raft_consensus.cc:1273] T 9ffd15b775094bbb920b3cb00b356a30 P 7395562be1584b47bea13989e88c8b52 [term 1 FOLLOWER]: Refusing update from remote peer da8eeec4d1184c43942f5579da019e8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250623 01:59:59.748322 20704 consensus_queue.cc:1035] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:59:59.748945 20697 consensus_queue.cc:1035] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7395562be1584b47bea13989e88c8b52" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 46727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 01:59:59.776512 20712 mvcc.cc:204] Tried to move back new op lower bound from 7170637822947635200 to 7170637822282342400. Current Snapshot: MvccSnapshot[applied={T|T < 7170637822947635200}]
I20250623 02:00:04.661316 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 20177
W20250623 02:00:04.849538 20399 heartbeater.cc:646] Failed to heartbeat to 127.14.43.190:40161 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.43.190:40161: connect: Connection refused (error 111)
W20250623 02:00:04.889660 20532 heartbeater.cc:646] Failed to heartbeat to 127.14.43.190:40161 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.43.190:40161: connect: Connection refused (error 111)
W20250623 02:00:04.899955 20665 heartbeater.cc:646] Failed to heartbeat to 127.14.43.190:40161 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.43.190:40161: connect: Connection refused (error 111)
W20250623 02:00:05.007927 20742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 02:00:05.008538 20742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 02:00:05.046394 20742 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250623 02:00:06.455176 20742 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.355s user 0.538s sys 0.802s
W20250623 02:00:06.455474 20742 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.355s user 0.538s sys 0.802s
I20250623 02:00:06.542729 20742 minidump.cc:252] Setting minidump size limit to 20M
I20250623 02:00:06.545040 20742 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 02:00:06.546341 20742 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 02:00:06.556289 20778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 02:00:06.557106 20779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 02:00:06.692262 20742 server_base.cc:1048] running on GCE node
W20250623 02:00:06.692276 20781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 02:00:06.693439 20742 hybrid_clock.cc:584] initializing the hybrid clock with 'system' time source
I20250623 02:00:06.693876 20742 hybrid_clock.cc:648] HybridClock initialized: now 1750644006693851 us; error 44239 us; skew 500 ppm
I20250623 02:00:06.694581 20742 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 02:00:06.699087 20742 webserver.cc:469] Webserver started at http://0.0.0.0:40649/ using document root <none> and password file <none>
I20250623 02:00:06.699894 20742 fs_manager.cc:362] Metadata directory not provided
I20250623 02:00:06.700124 20742 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 02:00:06.700551 20742 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 02:00:06.704735 20742 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/master-0/data/instance:
uuid: "5d090a643c8f494fac466bb333e2527b"
format_stamp: "Formatted at 2025-06-23 02:00:06 on dist-test-slave-9rpg"
I20250623 02:00:06.705806 20742 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/master-0/wal/instance:
uuid: "5d090a643c8f494fac466bb333e2527b"
format_stamp: "Formatted at 2025-06-23 02:00:06 on dist-test-slave-9rpg"
I20250623 02:00:06.711547 20742 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250623 02:00:06.716148 20786 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 02:00:06.717044 20742 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20250623 02:00:06.717355 20742 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/master-0/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/master-0/wal
uuid: "5d090a643c8f494fac466bb333e2527b"
format_stamp: "Formatted at 2025-06-23 02:00:06 on dist-test-slave-9rpg"
I20250623 02:00:06.717674 20742 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 02:00:06.837515 20742 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 02:00:06.838914 20742 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 02:00:06.839341 20742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 02:00:06.843716 20742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 02:00:06.857921 20742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b: Bootstrap starting.
I20250623 02:00:06.862825 20742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b: Neither blocks nor log segments found. Creating new log.
I20250623 02:00:06.864714 20742 log.cc:826] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b: Log is configured to *not* fsync() on all Append() calls
I20250623 02:00:06.868605 20742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b: No bootstrap required, opened a new log
I20250623 02:00:06.884414 20742 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d090a643c8f494fac466bb333e2527b" member_type: VOTER }
I20250623 02:00:06.884883 20742 raft_consensus.cc:383] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 02:00:06.885121 20742 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d090a643c8f494fac466bb333e2527b, State: Initialized, Role: FOLLOWER
I20250623 02:00:06.885792 20742 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d090a643c8f494fac466bb333e2527b" member_type: VOTER }
I20250623 02:00:06.886291 20742 raft_consensus.cc:397] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250623 02:00:06.886541 20742 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250623 02:00:06.886809 20742 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b [term 0 FOLLOWER]: Advancing to term 1
I20250623 02:00:06.890599 20742 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d090a643c8f494fac466bb333e2527b" member_type: VOTER }
I20250623 02:00:06.891235 20742 leader_election.cc:304] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d090a643c8f494fac466bb333e2527b; no voters:
I20250623 02:00:06.892794 20742 leader_election.cc:290] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250623 02:00:06.893025 20794 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b [term 1 FOLLOWER]: Leader election won for term 1
I20250623 02:00:06.894946 20794 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b [term 1 LEADER]: Becoming Leader. State: Replica: 5d090a643c8f494fac466bb333e2527b, State: Running, Role: LEADER
I20250623 02:00:06.895812 20794 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d090a643c8f494fac466bb333e2527b" member_type: VOTER }
I20250623 02:00:06.903129 20796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d090a643c8f494fac466bb333e2527b. Latest consensus state: current_term: 1 leader_uuid: "5d090a643c8f494fac466bb333e2527b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d090a643c8f494fac466bb333e2527b" member_type: VOTER } }
I20250623 02:00:06.903651 20796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b [sys.catalog]: This master's current role is: LEADER
I20250623 02:00:06.904034 20795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d090a643c8f494fac466bb333e2527b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d090a643c8f494fac466bb333e2527b" member_type: VOTER } }
I20250623 02:00:06.904558 20795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b [sys.catalog]: This master's current role is: LEADER
I20250623 02:00:06.915359 20742 tablet_replica.cc:331] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b: stopping tablet replica
I20250623 02:00:06.915866 20742 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b [term 1 LEADER]: Raft consensus shutting down.
I20250623 02:00:06.916276 20742 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 02:00:06.918304 20742 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250623 02:00:06.918704 20742 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250623 02:00:06.943125 20742 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
I20250623 02:00:07.970598 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 20270
I20250623 02:00:08.000160 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 20403
I20250623 02:00:08.035693 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 20536
I20250623 02:00:08.064692 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.43.190:40161
--webserver_interface=127.14.43.190
--webserver_port=32893
--builtin_ntp_servers=127.14.43.148:33689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.43.190:40161 with env {}
W20250623 02:00:08.364368 20806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 02:00:08.365013 20806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 02:00:08.365540 20806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 02:00:08.398206 20806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250623 02:00:08.398523 20806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 02:00:08.398768 20806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250623 02:00:08.398991 20806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250623 02:00:08.434789 20806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:33689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.43.190:40161
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.43.190:40161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.43.190
--webserver_port=32893
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 02:00:08.436094 20806 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 02:00:08.437760 20806 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 02:00:08.451560 20812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 02:00:08.451869 20815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 02:00:08.452685 20813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 02:00:08.453776 20806 server_base.cc:1048] running on GCE node
I20250623 02:00:09.544900 20806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 02:00:09.547853 20806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 02:00:09.549273 20806 hybrid_clock.cc:648] HybridClock initialized: now 1750644009549236 us; error 56 us; skew 500 ppm
I20250623 02:00:09.550088 20806 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 02:00:09.556464 20806 webserver.cc:469] Webserver started at http://127.14.43.190:32893/ using document root <none> and password file <none>
I20250623 02:00:09.557363 20806 fs_manager.cc:362] Metadata directory not provided
I20250623 02:00:09.557595 20806 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 02:00:09.565294 20806 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20250623 02:00:09.569622 20822 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 02:00:09.570649 20806 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20250623 02:00:09.570946 20806 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/master-0/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/master-0/wal
uuid: "5d090a643c8f494fac466bb333e2527b"
format_stamp: "Formatted at 2025-06-23 02:00:06 on dist-test-slave-9rpg"
I20250623 02:00:09.572793 20806 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 02:00:09.615535 20806 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 02:00:09.616962 20806 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 02:00:09.617403 20806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 02:00:09.683863 20806 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.190:40161
I20250623 02:00:09.683945 20873 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.190:40161 every 8 connection(s)
I20250623 02:00:09.686614 20806 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/master-0/data/info.pb
I20250623 02:00:09.692607 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 20806
I20250623 02:00:09.694165 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.43.129:45459
--local_ip_for_outbound_sockets=127.14.43.129
--tserver_master_addrs=127.14.43.190:40161
--webserver_port=46049
--webserver_interface=127.14.43.129
--builtin_ntp_servers=127.14.43.148:33689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250623 02:00:09.696317 20874 sys_catalog.cc:263] Verifying existing consensus state
I20250623 02:00:09.710506 20874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b: Bootstrap starting.
I20250623 02:00:09.723938 20874 log.cc:826] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b: Log is configured to *not* fsync() on all Append() calls
I20250623 02:00:09.738767 20874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 02:00:09.739554 20874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b: Bootstrap complete.
I20250623 02:00:09.759860 20874 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d090a643c8f494fac466bb333e2527b" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 40161 } }
I20250623 02:00:09.760528 20874 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d090a643c8f494fac466bb333e2527b, State: Initialized, Role: FOLLOWER
I20250623 02:00:09.761299 20874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d090a643c8f494fac466bb333e2527b" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 40161 } }
I20250623 02:00:09.761819 20874 raft_consensus.cc:397] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250623 02:00:09.762099 20874 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250623 02:00:09.762398 20874 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b [term 1 FOLLOWER]: Advancing to term 2
I20250623 02:00:09.766484 20874 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d090a643c8f494fac466bb333e2527b" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 40161 } }
I20250623 02:00:09.767136 20874 leader_election.cc:304] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d090a643c8f494fac466bb333e2527b; no voters:
I20250623 02:00:09.768850 20874 leader_election.cc:290] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b [CANDIDATE]: Term 2 election: Requested vote from peers
I20250623 02:00:09.769230 20878 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b [term 2 FOLLOWER]: Leader election won for term 2
I20250623 02:00:09.772255 20878 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b [term 2 LEADER]: Becoming Leader. State: Replica: 5d090a643c8f494fac466bb333e2527b, State: Running, Role: LEADER
I20250623 02:00:09.773243 20874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b [sys.catalog]: configured and running, proceeding with master startup.
I20250623 02:00:09.773124 20878 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d090a643c8f494fac466bb333e2527b" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 40161 } }
I20250623 02:00:09.785281 20880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d090a643c8f494fac466bb333e2527b. Latest consensus state: current_term: 2 leader_uuid: "5d090a643c8f494fac466bb333e2527b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d090a643c8f494fac466bb333e2527b" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 40161 } } }
I20250623 02:00:09.785343 20879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5d090a643c8f494fac466bb333e2527b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d090a643c8f494fac466bb333e2527b" member_type: VOTER last_known_addr { host: "127.14.43.190" port: 40161 } } }
I20250623 02:00:09.786078 20879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b [sys.catalog]: This master's current role is: LEADER
I20250623 02:00:09.787170 20880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b [sys.catalog]: This master's current role is: LEADER
I20250623 02:00:09.800104 20888 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250623 02:00:09.811273 20888 catalog_manager.cc:671] Loaded metadata for table pre_rebuild [id=b7b1f75c820f46d19522a564bdd9e05b]
I20250623 02:00:09.817699 20888 tablet_loader.cc:96] loaded metadata for tablet 9ffd15b775094bbb920b3cb00b356a30 (table pre_rebuild [id=b7b1f75c820f46d19522a564bdd9e05b])
I20250623 02:00:09.818974 20888 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250623 02:00:09.827515 20895 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250623 02:00:09.834800 20888 catalog_manager.cc:1349] Generated new cluster ID: 1959b7f3e9ef423f9a6c68a62d6e7b8e
I20250623 02:00:09.835143 20888 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250623 02:00:09.872426 20888 catalog_manager.cc:1372] Generated new certificate authority record
I20250623 02:00:09.874439 20888 catalog_manager.cc:1506] Loading token signing keys...
I20250623 02:00:09.897531 20888 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b: Generated new TSK 0
I20250623 02:00:09.898667 20888 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250623 02:00:10.025748 20876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 02:00:10.026284 20876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 02:00:10.026778 20876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 02:00:10.058626 20876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 02:00:10.059468 20876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.43.129
I20250623 02:00:10.094425 20876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:33689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.43.129:45459
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.43.129
--webserver_port=46049
--tserver_master_addrs=127.14.43.190:40161
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.43.129
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 02:00:10.095702 20876 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 02:00:10.097270 20876 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 02:00:10.113848 20903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 02:00:10.113994 20902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 02:00:10.113865 20905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 02:00:11.198259 20904 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250623 02:00:11.198439 20876 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250623 02:00:11.202769 20876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 02:00:11.205607 20876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 02:00:11.207149 20876 hybrid_clock.cc:648] HybridClock initialized: now 1750644011207101 us; error 51 us; skew 500 ppm
I20250623 02:00:11.208232 20876 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 02:00:11.215793 20876 webserver.cc:469] Webserver started at http://127.14.43.129:46049/ using document root <none> and password file <none>
I20250623 02:00:11.217047 20876 fs_manager.cc:362] Metadata directory not provided
I20250623 02:00:11.217360 20876 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 02:00:11.228109 20876 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.009s sys 0.000s
I20250623 02:00:11.234076 20912 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 02:00:11.235285 20876 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250623 02:00:11.235673 20876 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-0/wal
uuid: "9f26b3913c2845988766eae8e7bd2a0b"
format_stamp: "Formatted at 2025-06-23 01:59:55 on dist-test-slave-9rpg"
I20250623 02:00:11.238466 20876 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 02:00:11.299650 20876 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 02:00:11.301100 20876 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 02:00:11.301534 20876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 02:00:11.304477 20876 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 02:00:11.310540 20919 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250623 02:00:11.321511 20876 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250623 02:00:11.321861 20876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.000s sys 0.002s
I20250623 02:00:11.322175 20876 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250623 02:00:11.326956 20876 ts_tablet_manager.cc:610] Registered 1 tablets
I20250623 02:00:11.327168 20876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.004s sys 0.000s
I20250623 02:00:11.327674 20919 tablet_bootstrap.cc:492] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b: Bootstrap starting.
I20250623 02:00:11.496042 20876 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.129:45459
I20250623 02:00:11.496148 21025 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.129:45459 every 8 connection(s)
I20250623 02:00:11.498836 20876 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-0/data/info.pb
I20250623 02:00:11.501534 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 20876
I20250623 02:00:11.503417 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.43.130:44559
--local_ip_for_outbound_sockets=127.14.43.130
--tserver_master_addrs=127.14.43.190:40161
--webserver_port=42133
--webserver_interface=127.14.43.130
--builtin_ntp_servers=127.14.43.148:33689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250623 02:00:11.546670 21026 heartbeater.cc:344] Connected to a master server at 127.14.43.190:40161
I20250623 02:00:11.547188 21026 heartbeater.cc:461] Registering TS with master...
I20250623 02:00:11.548434 21026 heartbeater.cc:507] Master 127.14.43.190:40161 requested a full tablet report, sending...
I20250623 02:00:11.553172 20839 ts_manager.cc:194] Registered new tserver with Master: 9f26b3913c2845988766eae8e7bd2a0b (127.14.43.129:45459)
I20250623 02:00:11.560657 20839 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.43.129:39433
I20250623 02:00:11.642926 20919 log.cc:826] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b: Log is configured to *not* fsync() on all Append() calls
W20250623 02:00:11.834308 21030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 02:00:11.834790 21030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 02:00:11.835278 21030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 02:00:11.865978 21030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 02:00:11.866850 21030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.43.130
I20250623 02:00:11.901628 21030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:33689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.43.130:44559
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.43.130
--webserver_port=42133
--tserver_master_addrs=127.14.43.190:40161
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.43.130
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 02:00:11.902932 21030 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 02:00:11.904610 21030 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 02:00:11.920984 21038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 02:00:12.564711 21026 heartbeater.cc:499] Master 127.14.43.190:40161 was elected leader, sending a full tablet report...
W20250623 02:00:11.922431 21037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 02:00:13.247864 21040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 02:00:13.248852 21039 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1326 milliseconds
W20250623 02:00:13.250923 21030 thread.cc:640] OpenStack (cloud detector) Time spent creating pthread: real 1.333s user 0.493s sys 0.787s
W20250623 02:00:13.251294 21030 thread.cc:606] OpenStack (cloud detector) Time spent starting thread: real 1.334s user 0.493s sys 0.787s
I20250623 02:00:13.251534 21030 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250623 02:00:13.252563 21030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 02:00:13.255242 21030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 02:00:13.256637 21030 hybrid_clock.cc:648] HybridClock initialized: now 1750644013256582 us; error 73 us; skew 500 ppm
I20250623 02:00:13.257431 21030 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 02:00:13.265249 21030 webserver.cc:469] Webserver started at http://127.14.43.130:42133/ using document root <none> and password file <none>
I20250623 02:00:13.266412 21030 fs_manager.cc:362] Metadata directory not provided
I20250623 02:00:13.266695 21030 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 02:00:13.275130 21030 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.002s
I20250623 02:00:13.280160 21047 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 02:00:13.281266 21030 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250623 02:00:13.281570 21030 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-1/wal
uuid: "da8eeec4d1184c43942f5579da019e8e"
format_stamp: "Formatted at 2025-06-23 01:59:57 on dist-test-slave-9rpg"
I20250623 02:00:13.283584 21030 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 02:00:13.353644 21030 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 02:00:13.355156 21030 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 02:00:13.355597 21030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 02:00:13.358502 21030 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 02:00:13.364867 21054 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250623 02:00:13.372486 21030 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250623 02:00:13.372720 21030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.001s sys 0.001s
I20250623 02:00:13.372992 21030 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250623 02:00:13.377794 21030 ts_tablet_manager.cc:610] Registered 1 tablets
I20250623 02:00:13.378069 21030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.006s sys 0.000s
I20250623 02:00:13.378600 21054 tablet_bootstrap.cc:492] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e: Bootstrap starting.
I20250623 02:00:13.586440 21030 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.130:44559
I20250623 02:00:13.586547 21160 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.130:44559 every 8 connection(s)
I20250623 02:00:13.589143 21030 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-1/data/info.pb
I20250623 02:00:13.591562 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 21030
I20250623 02:00:13.593384 14510 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
/tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.43.131:46727
--local_ip_for_outbound_sockets=127.14.43.131
--tserver_master_addrs=127.14.43.190:40161
--webserver_port=41295
--webserver_interface=127.14.43.131
--builtin_ntp_servers=127.14.43.148:33689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250623 02:00:13.624624 21161 heartbeater.cc:344] Connected to a master server at 127.14.43.190:40161
I20250623 02:00:13.625077 21161 heartbeater.cc:461] Registering TS with master...
I20250623 02:00:13.626039 21161 heartbeater.cc:507] Master 127.14.43.190:40161 requested a full tablet report, sending...
I20250623 02:00:13.629137 20839 ts_manager.cc:194] Registered new tserver with Master: da8eeec4d1184c43942f5579da019e8e (127.14.43.130:44559)
I20250623 02:00:13.631915 20839 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.43.130:38013
I20250623 02:00:13.697798 21054 log.cc:826] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e: Log is configured to *not* fsync() on all Append() calls
W20250623 02:00:13.922293 21165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 02:00:13.922793 21165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 02:00:13.923333 21165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 02:00:13.955484 21165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 02:00:13.956391 21165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.43.131
I20250623 02:00:13.994091 21165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.43.148:33689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.43.131:46727
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.43.131
--webserver_port=41295
--tserver_master_addrs=127.14.43.190:40161
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.43.131
--log_dir=/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 01:43:23 UTC on 5fd53c4cbb9d
build id 6725
TSAN enabled
I20250623 02:00:13.995415 21165 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 02:00:13.997037 21165 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 02:00:14.014257 21172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 02:00:14.600929 20919 tablet_bootstrap.cc:492] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 02:00:14.602133 20919 tablet_bootstrap.cc:492] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b: Bootstrap complete.
I20250623 02:00:14.604139 20919 ts_tablet_manager.cc:1397] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b: Time spent bootstrapping tablet: real 3.277s user 3.207s sys 0.048s
I20250623 02:00:14.622134 20919 raft_consensus.cc:357] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8eeec4d1184c43942f5579da019e8e" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44559 } } peers { permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 } } peers { permanent_uuid: "7395562be1584b47bea13989e88c8b52" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 46727 } }
I20250623 02:00:14.625268 20919 raft_consensus.cc:738] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f26b3913c2845988766eae8e7bd2a0b, State: Initialized, Role: FOLLOWER
I20250623 02:00:14.626412 20919 consensus_queue.cc:260] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8eeec4d1184c43942f5579da019e8e" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44559 } } peers { permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 } } peers { permanent_uuid: "7395562be1584b47bea13989e88c8b52" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 46727 } }
I20250623 02:00:14.635895 20919 ts_tablet_manager.cc:1428] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b: Time spent starting tablet: real 0.031s user 0.029s sys 0.000s
I20250623 02:00:14.636147 21161 heartbeater.cc:499] Master 127.14.43.190:40161 was elected leader, sending a full tablet report...
W20250623 02:00:14.017271 21173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 02:00:15.646996 21165 thread.cc:640] OpenStack (cloud detector) Time spent creating pthread: real 1.630s user 0.593s sys 0.958s
W20250623 02:00:15.648039 21165 thread.cc:606] OpenStack (cloud detector) Time spent starting thread: real 1.631s user 0.594s sys 0.958s
W20250623 02:00:15.411345 21171 debug-util.cc:398] Leaking SignalData structure 0x7b0800000a80 after lost signal to thread 21165
W20250623 02:00:15.654824 21174 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1632 milliseconds
I20250623 02:00:15.655009 21165 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250623 02:00:15.655082 21175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 02:00:15.656390 21165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 02:00:15.658522 21165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 02:00:15.659863 21165 hybrid_clock.cc:648] HybridClock initialized: now 1750644015659833 us; error 33 us; skew 500 ppm
I20250623 02:00:15.660650 21165 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 02:00:15.666872 21165 webserver.cc:469] Webserver started at http://127.14.43.131:41295/ using document root <none> and password file <none>
I20250623 02:00:15.667744 21165 fs_manager.cc:362] Metadata directory not provided
I20250623 02:00:15.667973 21165 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 02:00:15.675978 21165 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.001s sys 0.003s
I20250623 02:00:15.680753 21184 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 02:00:15.681991 21165 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250623 02:00:15.682387 21165 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-2/wal
uuid: "7395562be1584b47bea13989e88c8b52"
format_stamp: "Formatted at 2025-06-23 01:59:58 on dist-test-slave-9rpg"
I20250623 02:00:15.684955 21165 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 02:00:15.743080 21165 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 02:00:15.744568 21165 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 02:00:15.744983 21165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 02:00:15.747897 21165 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 02:00:15.754070 21191 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250623 02:00:15.765067 21165 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250623 02:00:15.765321 21165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.001s sys 0.001s
I20250623 02:00:15.765578 21165 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250623 02:00:15.770520 21165 ts_tablet_manager.cc:610] Registered 1 tablets
I20250623 02:00:15.770713 21165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.005s sys 0.000s
I20250623 02:00:15.771205 21191 tablet_bootstrap.cc:492] T 9ffd15b775094bbb920b3cb00b356a30 P 7395562be1584b47bea13989e88c8b52: Bootstrap starting.
I20250623 02:00:15.951429 21290 raft_consensus.cc:491] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 02:00:15.952263 21290 raft_consensus.cc:513] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8eeec4d1184c43942f5579da019e8e" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44559 } } peers { permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 } } peers { permanent_uuid: "7395562be1584b47bea13989e88c8b52" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 46727 } }
I20250623 02:00:15.966841 21290 leader_election.cc:290] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers da8eeec4d1184c43942f5579da019e8e (127.14.43.130:44559), 7395562be1584b47bea13989e88c8b52 (127.14.43.131:46727)
W20250623 02:00:15.969211 20915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.43.131:46727: connect: Connection refused (error 111)
W20250623 02:00:15.984480 20915 leader_election.cc:336] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7395562be1584b47bea13989e88c8b52 (127.14.43.131:46727): Network error: Client connection negotiation failed: client connection to 127.14.43.131:46727: connect: Connection refused (error 111)
I20250623 02:00:15.987598 21116 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ffd15b775094bbb920b3cb00b356a30" candidate_uuid: "9f26b3913c2845988766eae8e7bd2a0b" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "da8eeec4d1184c43942f5579da019e8e" is_pre_election: true
W20250623 02:00:16.004779 20913 leader_election.cc:343] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer da8eeec4d1184c43942f5579da019e8e (127.14.43.130:44559): Illegal state: must be running to vote when last-logged opid is not known
I20250623 02:00:16.005298 20913 leader_election.cc:304] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f26b3913c2845988766eae8e7bd2a0b; no voters: 7395562be1584b47bea13989e88c8b52, da8eeec4d1184c43942f5579da019e8e
I20250623 02:00:16.006898 21290 raft_consensus.cc:2747] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 02:00:16.010061 21165 rpc_server.cc:307] RPC server started. Bound to: 127.14.43.131:46727
I20250623 02:00:16.010195 21302 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.43.131:46727 every 8 connection(s)
I20250623 02:00:16.013453 21165 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-2/data/info.pb
I20250623 02:00:16.019210 14510 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu as pid 21165
I20250623 02:00:16.064386 21303 heartbeater.cc:344] Connected to a master server at 127.14.43.190:40161
I20250623 02:00:16.064935 21303 heartbeater.cc:461] Registering TS with master...
I20250623 02:00:16.066249 21303 heartbeater.cc:507] Master 127.14.43.190:40161 requested a full tablet report, sending...
I20250623 02:00:16.070415 20838 ts_manager.cc:194] Registered new tserver with Master: 7395562be1584b47bea13989e88c8b52 (127.14.43.131:46727)
I20250623 02:00:16.073563 20838 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.43.131:38177
I20250623 02:00:16.078887 14510 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250623 02:00:16.081167 21191 log.cc:826] T 9ffd15b775094bbb920b3cb00b356a30 P 7395562be1584b47bea13989e88c8b52: Log is configured to *not* fsync() on all Append() calls
I20250623 02:00:16.671949 21054 tablet_bootstrap.cc:492] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 02:00:16.672750 21054 tablet_bootstrap.cc:492] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e: Bootstrap complete.
I20250623 02:00:16.673880 21054 ts_tablet_manager.cc:1397] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e: Time spent bootstrapping tablet: real 3.296s user 2.884s sys 0.076s
I20250623 02:00:16.679139 21054 raft_consensus.cc:357] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8eeec4d1184c43942f5579da019e8e" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44559 } } peers { permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 } } peers { permanent_uuid: "7395562be1584b47bea13989e88c8b52" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 46727 } }
I20250623 02:00:16.681237 21054 raft_consensus.cc:738] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: da8eeec4d1184c43942f5579da019e8e, State: Initialized, Role: FOLLOWER
I20250623 02:00:16.681978 21054 consensus_queue.cc:260] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8eeec4d1184c43942f5579da019e8e" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44559 } } peers { permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 } } peers { permanent_uuid: "7395562be1584b47bea13989e88c8b52" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 46727 } }
I20250623 02:00:16.685253 21054 ts_tablet_manager.cc:1428] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20250623 02:00:17.077772 21303 heartbeater.cc:499] Master 127.14.43.190:40161 was elected leader, sending a full tablet report...
I20250623 02:00:17.935849 21316 raft_consensus.cc:491] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 02:00:17.936237 21316 raft_consensus.cc:513] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8eeec4d1184c43942f5579da019e8e" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44559 } } peers { permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 } } peers { permanent_uuid: "7395562be1584b47bea13989e88c8b52" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 46727 } }
I20250623 02:00:17.937594 21316 leader_election.cc:290] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers da8eeec4d1184c43942f5579da019e8e (127.14.43.130:44559), 7395562be1584b47bea13989e88c8b52 (127.14.43.131:46727)
I20250623 02:00:17.938709 21116 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ffd15b775094bbb920b3cb00b356a30" candidate_uuid: "9f26b3913c2845988766eae8e7bd2a0b" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "da8eeec4d1184c43942f5579da019e8e" is_pre_election: true
I20250623 02:00:17.939498 21116 raft_consensus.cc:2466] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f26b3913c2845988766eae8e7bd2a0b in term 1.
I20250623 02:00:17.940917 20913 leader_election.cc:304] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f26b3913c2845988766eae8e7bd2a0b, da8eeec4d1184c43942f5579da019e8e; no voters:
I20250623 02:00:17.941896 21316 raft_consensus.cc:2802] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 02:00:17.942317 21316 raft_consensus.cc:491] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 02:00:17.942711 21316 raft_consensus.cc:3058] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [term 1 FOLLOWER]: Advancing to term 2
I20250623 02:00:17.952200 21316 raft_consensus.cc:513] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8eeec4d1184c43942f5579da019e8e" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44559 } } peers { permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 } } peers { permanent_uuid: "7395562be1584b47bea13989e88c8b52" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 46727 } }
I20250623 02:00:17.954352 21316 leader_election.cc:290] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [CANDIDATE]: Term 2 election: Requested vote from peers da8eeec4d1184c43942f5579da019e8e (127.14.43.130:44559), 7395562be1584b47bea13989e88c8b52 (127.14.43.131:46727)
I20250623 02:00:17.954973 21116 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ffd15b775094bbb920b3cb00b356a30" candidate_uuid: "9f26b3913c2845988766eae8e7bd2a0b" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "da8eeec4d1184c43942f5579da019e8e"
I20250623 02:00:17.955521 21116 raft_consensus.cc:3058] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e [term 1 FOLLOWER]: Advancing to term 2
I20250623 02:00:17.965598 21116 raft_consensus.cc:2466] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f26b3913c2845988766eae8e7bd2a0b in term 2.
I20250623 02:00:17.966711 20913 leader_election.cc:304] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f26b3913c2845988766eae8e7bd2a0b, da8eeec4d1184c43942f5579da019e8e; no voters:
I20250623 02:00:17.967473 21316 raft_consensus.cc:2802] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [term 2 FOLLOWER]: Leader election won for term 2
I20250623 02:00:17.969951 21316 raft_consensus.cc:695] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [term 2 LEADER]: Becoming Leader. State: Replica: 9f26b3913c2845988766eae8e7bd2a0b, State: Running, Role: LEADER
I20250623 02:00:17.962832 21252 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ffd15b775094bbb920b3cb00b356a30" candidate_uuid: "9f26b3913c2845988766eae8e7bd2a0b" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "7395562be1584b47bea13989e88c8b52"
I20250623 02:00:17.962430 21253 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ffd15b775094bbb920b3cb00b356a30" candidate_uuid: "9f26b3913c2845988766eae8e7bd2a0b" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "7395562be1584b47bea13989e88c8b52" is_pre_election: true
I20250623 02:00:17.970964 21316 consensus_queue.cc:237] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8eeec4d1184c43942f5579da019e8e" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44559 } } peers { permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 } } peers { permanent_uuid: "7395562be1584b47bea13989e88c8b52" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 46727 } }
W20250623 02:00:17.972580 20915 leader_election.cc:343] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 7395562be1584b47bea13989e88c8b52 (127.14.43.131:46727): Illegal state: must be running to vote when last-logged opid is not known
I20250623 02:00:17.982749 20838 catalog_manager.cc:5582] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b reported cstate change: term changed from 0 to 2, leader changed from <none> to 9f26b3913c2845988766eae8e7bd2a0b (127.14.43.129), VOTER 7395562be1584b47bea13989e88c8b52 (127.14.43.131) added, VOTER 9f26b3913c2845988766eae8e7bd2a0b (127.14.43.129) added, VOTER da8eeec4d1184c43942f5579da019e8e (127.14.43.130) added. New cstate: current_term: 2 leader_uuid: "9f26b3913c2845988766eae8e7bd2a0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8eeec4d1184c43942f5579da019e8e" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7395562be1584b47bea13989e88c8b52" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 46727 } health_report { overall_health: UNKNOWN } } }
I20250623 02:00:18.001688 21191 tablet_bootstrap.cc:492] T 9ffd15b775094bbb920b3cb00b356a30 P 7395562be1584b47bea13989e88c8b52: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250623 02:00:18.002746 21191 tablet_bootstrap.cc:492] T 9ffd15b775094bbb920b3cb00b356a30 P 7395562be1584b47bea13989e88c8b52: Bootstrap complete.
I20250623 02:00:18.004421 21191 ts_tablet_manager.cc:1397] T 9ffd15b775094bbb920b3cb00b356a30 P 7395562be1584b47bea13989e88c8b52: Time spent bootstrapping tablet: real 2.234s user 2.127s sys 0.088s
I20250623 02:00:18.011977 21191 raft_consensus.cc:357] T 9ffd15b775094bbb920b3cb00b356a30 P 7395562be1584b47bea13989e88c8b52 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8eeec4d1184c43942f5579da019e8e" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44559 } } peers { permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 } } peers { permanent_uuid: "7395562be1584b47bea13989e88c8b52" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 46727 } }
I20250623 02:00:18.014982 21191 raft_consensus.cc:738] T 9ffd15b775094bbb920b3cb00b356a30 P 7395562be1584b47bea13989e88c8b52 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7395562be1584b47bea13989e88c8b52, State: Initialized, Role: FOLLOWER
I20250623 02:00:18.016119 21191 consensus_queue.cc:260] T 9ffd15b775094bbb920b3cb00b356a30 P 7395562be1584b47bea13989e88c8b52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8eeec4d1184c43942f5579da019e8e" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44559 } } peers { permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 } } peers { permanent_uuid: "7395562be1584b47bea13989e88c8b52" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 46727 } }
I20250623 02:00:18.019379 21191 ts_tablet_manager.cc:1428] T 9ffd15b775094bbb920b3cb00b356a30 P 7395562be1584b47bea13989e88c8b52: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
W20250623 02:00:18.466593 14510 scanner-internal.cc:458] Time spent opening tablet: real 2.348s user 0.007s sys 0.000s
I20250623 02:00:18.539870 21116 raft_consensus.cc:1273] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e [term 2 FOLLOWER]: Refusing update from remote peer 9f26b3913c2845988766eae8e7bd2a0b: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250623 02:00:18.541692 21331 consensus_queue.cc:1035] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "da8eeec4d1184c43942f5579da019e8e" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
I20250623 02:00:18.555366 21253 raft_consensus.cc:3058] T 9ffd15b775094bbb920b3cb00b356a30 P 7395562be1584b47bea13989e88c8b52 [term 1 FOLLOWER]: Advancing to term 2
I20250623 02:00:18.565335 21253 raft_consensus.cc:1273] T 9ffd15b775094bbb920b3cb00b356a30 P 7395562be1584b47bea13989e88c8b52 [term 2 FOLLOWER]: Refusing update from remote peer 9f26b3913c2845988766eae8e7bd2a0b: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250623 02:00:18.586196 21331 consensus_queue.cc:1035] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7395562be1584b47bea13989e88c8b52" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 46727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.001s
I20250623 02:00:18.645362 20981 consensus_queue.cc:237] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 2.206, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "da8eeec4d1184c43942f5579da019e8e" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44559 } } peers { permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 } }
I20250623 02:00:18.651124 21116 raft_consensus.cc:1273] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e [term 2 FOLLOWER]: Refusing update from remote peer 9f26b3913c2845988766eae8e7bd2a0b: Log matching property violated. Preceding OpId in replica: term: 2 index: 206. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
I20250623 02:00:18.652824 21335 consensus_queue.cc:1035] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "da8eeec4d1184c43942f5579da019e8e" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.001s
I20250623 02:00:18.659273 21334 raft_consensus.cc:2953] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [term 2 LEADER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 7395562be1584b47bea13989e88c8b52 (127.14.43.131) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "da8eeec4d1184c43942f5579da019e8e" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44559 } } peers { permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 } } }
I20250623 02:00:18.660809 21116 raft_consensus.cc:2953] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e [term 2 FOLLOWER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 7395562be1584b47bea13989e88c8b52 (127.14.43.131) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "da8eeec4d1184c43942f5579da019e8e" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44559 } } peers { permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 } } }
I20250623 02:00:18.669582 20823 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 9ffd15b775094bbb920b3cb00b356a30 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250623 02:00:18.673261 20839 catalog_manager.cc:5582] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b reported cstate change: config changed from index -1 to 207, VOTER 7395562be1584b47bea13989e88c8b52 (127.14.43.131) evicted. New cstate: current_term: 2 leader_uuid: "9f26b3913c2845988766eae8e7bd2a0b" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "da8eeec4d1184c43942f5579da019e8e" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 } health_report { overall_health: HEALTHY } } }
I20250623 02:00:18.709954 20981 consensus_queue.cc:237] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 2.207, Last appended by leader: 205, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 } }
I20250623 02:00:18.712610 21331 raft_consensus.cc:2953] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [term 2 LEADER]: Committing config change with OpId 2.208: config changed from index 207 to 208, VOTER da8eeec4d1184c43942f5579da019e8e (127.14.43.130) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 } } }
I20250623 02:00:18.722029 20823 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 9ffd15b775094bbb920b3cb00b356a30 with cas_config_opid_index 207: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250623 02:00:18.726878 20838 catalog_manager.cc:5582] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b reported cstate change: config changed from index 207 to 208, VOTER da8eeec4d1184c43942f5579da019e8e (127.14.43.130) evicted. New cstate: current_term: 2 leader_uuid: "9f26b3913c2845988766eae8e7bd2a0b" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 } health_report { overall_health: HEALTHY } } }
I20250623 02:00:18.733333 21233 tablet_service.cc:1515] Processing DeleteTablet for tablet 9ffd15b775094bbb920b3cb00b356a30 with delete_type TABLET_DATA_TOMBSTONED (TS 7395562be1584b47bea13989e88c8b52 not found in new config with opid_index 207) from {username='slave'} at 127.0.0.1:42896
I20250623 02:00:18.743196 21346 tablet_replica.cc:331] stopping tablet replica
I20250623 02:00:18.744129 21346 raft_consensus.cc:2241] T 9ffd15b775094bbb920b3cb00b356a30 P 7395562be1584b47bea13989e88c8b52 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250623 02:00:18.744827 21346 raft_consensus.cc:2270] T 9ffd15b775094bbb920b3cb00b356a30 P 7395562be1584b47bea13989e88c8b52 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 02:00:18.765875 21096 tablet_service.cc:1515] Processing DeleteTablet for tablet 9ffd15b775094bbb920b3cb00b356a30 with delete_type TABLET_DATA_TOMBSTONED (TS da8eeec4d1184c43942f5579da019e8e not found in new config with opid_index 208) from {username='slave'} at 127.0.0.1:41602
I20250623 02:00:18.775655 21348 tablet_replica.cc:331] stopping tablet replica
I20250623 02:00:18.776566 21348 raft_consensus.cc:2241] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e [term 2 FOLLOWER]: Raft consensus shutting down.
I20250623 02:00:18.778357 21346 ts_tablet_manager.cc:1905] T 9ffd15b775094bbb920b3cb00b356a30 P 7395562be1584b47bea13989e88c8b52: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250623 02:00:18.782495 21348 raft_consensus.cc:2270] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 02:00:18.794943 21346 ts_tablet_manager.cc:1918] T 9ffd15b775094bbb920b3cb00b356a30 P 7395562be1584b47bea13989e88c8b52: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.206
I20250623 02:00:18.795481 21346 log.cc:1199] T 9ffd15b775094bbb920b3cb00b356a30 P 7395562be1584b47bea13989e88c8b52: Deleting WAL directory at /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-2/wal/wals/9ffd15b775094bbb920b3cb00b356a30
I20250623 02:00:18.797469 20825 catalog_manager.cc:4928] TS 7395562be1584b47bea13989e88c8b52 (127.14.43.131:46727): tablet 9ffd15b775094bbb920b3cb00b356a30 (table pre_rebuild [id=b7b1f75c820f46d19522a564bdd9e05b]) successfully deleted
I20250623 02:00:18.813200 21348 ts_tablet_manager.cc:1905] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250623 02:00:18.828529 21348 ts_tablet_manager.cc:1918] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.207
I20250623 02:00:18.828946 21348 log.cc:1199] T 9ffd15b775094bbb920b3cb00b356a30 P da8eeec4d1184c43942f5579da019e8e: Deleting WAL directory at /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-1/wal/wals/9ffd15b775094bbb920b3cb00b356a30
I20250623 02:00:18.830721 20823 catalog_manager.cc:4928] TS da8eeec4d1184c43942f5579da019e8e (127.14.43.130:44559): tablet 9ffd15b775094bbb920b3cb00b356a30 (table pre_rebuild [id=b7b1f75c820f46d19522a564bdd9e05b]) successfully deleted
I20250623 02:00:19.165122 21233 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250623 02:00:19.172751 21096 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250623 02:00:19.179268 20961 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
5d090a643c8f494fac466bb333e2527b | 127.14.43.190:40161 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.14.43.148:33689 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
7395562be1584b47bea13989e88c8b52 | 127.14.43.131:46727 | HEALTHY | <none> | 0 | 0
9f26b3913c2845988766eae8e7bd2a0b | 127.14.43.129:45459 | HEALTHY | <none> | 1 | 0
da8eeec4d1184c43942f5579da019e8e | 127.14.43.130:44559 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.14.43.129 | experimental | 127.14.43.129:45459
local_ip_for_outbound_sockets | 127.14.43.130 | experimental | 127.14.43.130:44559
local_ip_for_outbound_sockets | 127.14.43.131 | experimental | 127.14.43.131:46727
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-0/data/info.pb | hidden | 127.14.43.129:45459
server_dump_info_path | /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-1/data/info.pb | hidden | 127.14.43.130:44559
server_dump_info_path | /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-2/data/info.pb | hidden | 127.14.43.131:46727
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.14.43.148:33689 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------+----+---------+---------------+---------+------------+------------------+-------------
pre_rebuild | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 0
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 1
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250623 02:00:19.436460 14510 log_verifier.cc:126] Checking tablet 9ffd15b775094bbb920b3cb00b356a30
I20250623 02:00:19.733821 14510 log_verifier.cc:177] Verified matching terms for 208 ops in tablet 9ffd15b775094bbb920b3cb00b356a30
I20250623 02:00:19.736624 20839 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42340:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250623 02:00:19.737275 20839 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42340:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250623 02:00:19.755280 20839 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 5d090a643c8f494fac466bb333e2527b: Sending DeleteTablet for 1 replicas of tablet 9ffd15b775094bbb920b3cb00b356a30
I20250623 02:00:19.757599 14510 test_util.cc:276] Using random seed: -1829961210
I20250623 02:00:19.757691 20961 tablet_service.cc:1515] Processing DeleteTablet for tablet 9ffd15b775094bbb920b3cb00b356a30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 02:00:19 UTC) from {username='slave'} at 127.0.0.1:60350
I20250623 02:00:19.759387 21378 tablet_replica.cc:331] stopping tablet replica
I20250623 02:00:19.759971 21378 raft_consensus.cc:2241] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [term 2 LEADER]: Raft consensus shutting down.
I20250623 02:00:19.760432 21378 raft_consensus.cc:2270] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 02:00:19.789750 21378 ts_tablet_manager.cc:1905] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 02:00:19.798533 20839 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42394:
name: "post_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 02:00:19.801945 20839 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table post_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 02:00:19.803768 21378 ts_tablet_manager.cc:1918] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.208
I20250623 02:00:19.804282 21378 log.cc:1199] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b: Deleting WAL directory at /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-0/wal/wals/9ffd15b775094bbb920b3cb00b356a30
I20250623 02:00:19.805270 21378 ts_tablet_manager.cc:1939] T 9ffd15b775094bbb920b3cb00b356a30 P 9f26b3913c2845988766eae8e7bd2a0b: Deleting consensus metadata
I20250623 02:00:19.808012 20823 catalog_manager.cc:4928] TS 9f26b3913c2845988766eae8e7bd2a0b (127.14.43.129:45459): tablet 9ffd15b775094bbb920b3cb00b356a30 (table pre_rebuild [id=b7b1f75c820f46d19522a564bdd9e05b]) successfully deleted
I20250623 02:00:19.825783 21096 tablet_service.cc:1468] Processing CreateTablet for tablet 9fe242dd38404d08986ef30ff614c1f1 (DEFAULT_TABLE table=post_rebuild [id=a9699097145b4bf6981d902bec7df3e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 02:00:19.826265 21233 tablet_service.cc:1468] Processing CreateTablet for tablet 9fe242dd38404d08986ef30ff614c1f1 (DEFAULT_TABLE table=post_rebuild [id=a9699097145b4bf6981d902bec7df3e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 02:00:19.826656 20961 tablet_service.cc:1468] Processing CreateTablet for tablet 9fe242dd38404d08986ef30ff614c1f1 (DEFAULT_TABLE table=post_rebuild [id=a9699097145b4bf6981d902bec7df3e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 02:00:19.827183 21096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fe242dd38404d08986ef30ff614c1f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 02:00:19.827466 21233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fe242dd38404d08986ef30ff614c1f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 02:00:19.827698 20961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fe242dd38404d08986ef30ff614c1f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 02:00:19.845284 21386 tablet_bootstrap.cc:492] T 9fe242dd38404d08986ef30ff614c1f1 P da8eeec4d1184c43942f5579da019e8e: Bootstrap starting.
I20250623 02:00:19.847966 21385 tablet_bootstrap.cc:492] T 9fe242dd38404d08986ef30ff614c1f1 P 9f26b3913c2845988766eae8e7bd2a0b: Bootstrap starting.
I20250623 02:00:19.854409 21386 tablet_bootstrap.cc:654] T 9fe242dd38404d08986ef30ff614c1f1 P da8eeec4d1184c43942f5579da019e8e: Neither blocks nor log segments found. Creating new log.
I20250623 02:00:19.861356 21385 tablet_bootstrap.cc:654] T 9fe242dd38404d08986ef30ff614c1f1 P 9f26b3913c2845988766eae8e7bd2a0b: Neither blocks nor log segments found. Creating new log.
I20250623 02:00:19.863022 21387 tablet_bootstrap.cc:492] T 9fe242dd38404d08986ef30ff614c1f1 P 7395562be1584b47bea13989e88c8b52: Bootstrap starting.
I20250623 02:00:19.865756 21386 tablet_bootstrap.cc:492] T 9fe242dd38404d08986ef30ff614c1f1 P da8eeec4d1184c43942f5579da019e8e: No bootstrap required, opened a new log
I20250623 02:00:19.866281 21386 ts_tablet_manager.cc:1397] T 9fe242dd38404d08986ef30ff614c1f1 P da8eeec4d1184c43942f5579da019e8e: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.010s
I20250623 02:00:19.868197 21386 raft_consensus.cc:357] T 9fe242dd38404d08986ef30ff614c1f1 P da8eeec4d1184c43942f5579da019e8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8eeec4d1184c43942f5579da019e8e" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44559 } } peers { permanent_uuid: "7395562be1584b47bea13989e88c8b52" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 46727 } } peers { permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 } }
I20250623 02:00:19.869050 21387 tablet_bootstrap.cc:654] T 9fe242dd38404d08986ef30ff614c1f1 P 7395562be1584b47bea13989e88c8b52: Neither blocks nor log segments found. Creating new log.
I20250623 02:00:19.869051 21386 raft_consensus.cc:383] T 9fe242dd38404d08986ef30ff614c1f1 P da8eeec4d1184c43942f5579da019e8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 02:00:19.869444 21386 raft_consensus.cc:738] T 9fe242dd38404d08986ef30ff614c1f1 P da8eeec4d1184c43942f5579da019e8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da8eeec4d1184c43942f5579da019e8e, State: Initialized, Role: FOLLOWER
I20250623 02:00:19.870219 21386 consensus_queue.cc:260] T 9fe242dd38404d08986ef30ff614c1f1 P da8eeec4d1184c43942f5579da019e8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8eeec4d1184c43942f5579da019e8e" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44559 } } peers { permanent_uuid: "7395562be1584b47bea13989e88c8b52" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 46727 } } peers { permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 } }
I20250623 02:00:19.876303 21386 ts_tablet_manager.cc:1428] T 9fe242dd38404d08986ef30ff614c1f1 P da8eeec4d1184c43942f5579da019e8e: Time spent starting tablet: real 0.010s user 0.003s sys 0.005s
I20250623 02:00:19.891188 21387 tablet_bootstrap.cc:492] T 9fe242dd38404d08986ef30ff614c1f1 P 7395562be1584b47bea13989e88c8b52: No bootstrap required, opened a new log
I20250623 02:00:19.891212 21385 tablet_bootstrap.cc:492] T 9fe242dd38404d08986ef30ff614c1f1 P 9f26b3913c2845988766eae8e7bd2a0b: No bootstrap required, opened a new log
I20250623 02:00:19.891541 21387 ts_tablet_manager.cc:1397] T 9fe242dd38404d08986ef30ff614c1f1 P 7395562be1584b47bea13989e88c8b52: Time spent bootstrapping tablet: real 0.029s user 0.013s sys 0.013s
I20250623 02:00:19.891604 21385 ts_tablet_manager.cc:1397] T 9fe242dd38404d08986ef30ff614c1f1 P 9f26b3913c2845988766eae8e7bd2a0b: Time spent bootstrapping tablet: real 0.044s user 0.011s sys 0.011s
I20250623 02:00:19.893476 21387 raft_consensus.cc:357] T 9fe242dd38404d08986ef30ff614c1f1 P 7395562be1584b47bea13989e88c8b52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8eeec4d1184c43942f5579da019e8e" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44559 } } peers { permanent_uuid: "7395562be1584b47bea13989e88c8b52" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 46727 } } peers { permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 } }
I20250623 02:00:19.894055 21387 raft_consensus.cc:383] T 9fe242dd38404d08986ef30ff614c1f1 P 7395562be1584b47bea13989e88c8b52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 02:00:19.894292 21387 raft_consensus.cc:738] T 9fe242dd38404d08986ef30ff614c1f1 P 7395562be1584b47bea13989e88c8b52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7395562be1584b47bea13989e88c8b52, State: Initialized, Role: FOLLOWER
I20250623 02:00:19.894161 21385 raft_consensus.cc:357] T 9fe242dd38404d08986ef30ff614c1f1 P 9f26b3913c2845988766eae8e7bd2a0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8eeec4d1184c43942f5579da019e8e" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44559 } } peers { permanent_uuid: "7395562be1584b47bea13989e88c8b52" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 46727 } } peers { permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 } }
I20250623 02:00:19.894867 21385 raft_consensus.cc:383] T 9fe242dd38404d08986ef30ff614c1f1 P 9f26b3913c2845988766eae8e7bd2a0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 02:00:19.894853 21387 consensus_queue.cc:260] T 9fe242dd38404d08986ef30ff614c1f1 P 7395562be1584b47bea13989e88c8b52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8eeec4d1184c43942f5579da019e8e" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44559 } } peers { permanent_uuid: "7395562be1584b47bea13989e88c8b52" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 46727 } } peers { permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 } }
I20250623 02:00:19.895172 21385 raft_consensus.cc:738] T 9fe242dd38404d08986ef30ff614c1f1 P 9f26b3913c2845988766eae8e7bd2a0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f26b3913c2845988766eae8e7bd2a0b, State: Initialized, Role: FOLLOWER
I20250623 02:00:19.895800 21385 consensus_queue.cc:260] T 9fe242dd38404d08986ef30ff614c1f1 P 9f26b3913c2845988766eae8e7bd2a0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8eeec4d1184c43942f5579da019e8e" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44559 } } peers { permanent_uuid: "7395562be1584b47bea13989e88c8b52" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 46727 } } peers { permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 } }
I20250623 02:00:19.901835 21385 ts_tablet_manager.cc:1428] T 9fe242dd38404d08986ef30ff614c1f1 P 9f26b3913c2845988766eae8e7bd2a0b: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20250623 02:00:19.904083 21387 ts_tablet_manager.cc:1428] T 9fe242dd38404d08986ef30ff614c1f1 P 7395562be1584b47bea13989e88c8b52: Time spent starting tablet: real 0.012s user 0.006s sys 0.006s
W20250623 02:00:20.033191 21027 tablet.cc:2378] T 9fe242dd38404d08986ef30ff614c1f1 P 9f26b3913c2845988766eae8e7bd2a0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250623 02:00:20.050382 21304 tablet.cc:2378] T 9fe242dd38404d08986ef30ff614c1f1 P 7395562be1584b47bea13989e88c8b52: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250623 02:00:20.121603 21162 tablet.cc:2378] T 9fe242dd38404d08986ef30ff614c1f1 P da8eeec4d1184c43942f5579da019e8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250623 02:00:20.226114 21393 raft_consensus.cc:491] T 9fe242dd38404d08986ef30ff614c1f1 P 9f26b3913c2845988766eae8e7bd2a0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 02:00:20.226985 21393 raft_consensus.cc:513] T 9fe242dd38404d08986ef30ff614c1f1 P 9f26b3913c2845988766eae8e7bd2a0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8eeec4d1184c43942f5579da019e8e" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44559 } } peers { permanent_uuid: "7395562be1584b47bea13989e88c8b52" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 46727 } } peers { permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 } }
I20250623 02:00:20.229307 21393 leader_election.cc:290] T 9fe242dd38404d08986ef30ff614c1f1 P 9f26b3913c2845988766eae8e7bd2a0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da8eeec4d1184c43942f5579da019e8e (127.14.43.130:44559), 7395562be1584b47bea13989e88c8b52 (127.14.43.131:46727)
I20250623 02:00:20.230434 21116 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9fe242dd38404d08986ef30ff614c1f1" candidate_uuid: "9f26b3913c2845988766eae8e7bd2a0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da8eeec4d1184c43942f5579da019e8e" is_pre_election: true
I20250623 02:00:20.230628 21253 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9fe242dd38404d08986ef30ff614c1f1" candidate_uuid: "9f26b3913c2845988766eae8e7bd2a0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7395562be1584b47bea13989e88c8b52" is_pre_election: true
I20250623 02:00:20.230912 21116 raft_consensus.cc:2466] T 9fe242dd38404d08986ef30ff614c1f1 P da8eeec4d1184c43942f5579da019e8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f26b3913c2845988766eae8e7bd2a0b in term 0.
I20250623 02:00:20.231333 21253 raft_consensus.cc:2466] T 9fe242dd38404d08986ef30ff614c1f1 P 7395562be1584b47bea13989e88c8b52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f26b3913c2845988766eae8e7bd2a0b in term 0.
I20250623 02:00:20.231946 20913 leader_election.cc:304] T 9fe242dd38404d08986ef30ff614c1f1 P 9f26b3913c2845988766eae8e7bd2a0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f26b3913c2845988766eae8e7bd2a0b, da8eeec4d1184c43942f5579da019e8e; no voters:
I20250623 02:00:20.232750 21393 raft_consensus.cc:2802] T 9fe242dd38404d08986ef30ff614c1f1 P 9f26b3913c2845988766eae8e7bd2a0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 02:00:20.233134 21393 raft_consensus.cc:491] T 9fe242dd38404d08986ef30ff614c1f1 P 9f26b3913c2845988766eae8e7bd2a0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 02:00:20.233426 21393 raft_consensus.cc:3058] T 9fe242dd38404d08986ef30ff614c1f1 P 9f26b3913c2845988766eae8e7bd2a0b [term 0 FOLLOWER]: Advancing to term 1
I20250623 02:00:20.238902 21393 raft_consensus.cc:513] T 9fe242dd38404d08986ef30ff614c1f1 P 9f26b3913c2845988766eae8e7bd2a0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8eeec4d1184c43942f5579da019e8e" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44559 } } peers { permanent_uuid: "7395562be1584b47bea13989e88c8b52" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 46727 } } peers { permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 } }
I20250623 02:00:20.240515 21393 leader_election.cc:290] T 9fe242dd38404d08986ef30ff614c1f1 P 9f26b3913c2845988766eae8e7bd2a0b [CANDIDATE]: Term 1 election: Requested vote from peers da8eeec4d1184c43942f5579da019e8e (127.14.43.130:44559), 7395562be1584b47bea13989e88c8b52 (127.14.43.131:46727)
I20250623 02:00:20.241343 21116 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9fe242dd38404d08986ef30ff614c1f1" candidate_uuid: "9f26b3913c2845988766eae8e7bd2a0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da8eeec4d1184c43942f5579da019e8e"
I20250623 02:00:20.241851 21116 raft_consensus.cc:3058] T 9fe242dd38404d08986ef30ff614c1f1 P da8eeec4d1184c43942f5579da019e8e [term 0 FOLLOWER]: Advancing to term 1
I20250623 02:00:20.241815 21253 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9fe242dd38404d08986ef30ff614c1f1" candidate_uuid: "9f26b3913c2845988766eae8e7bd2a0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7395562be1584b47bea13989e88c8b52"
I20250623 02:00:20.242389 21253 raft_consensus.cc:3058] T 9fe242dd38404d08986ef30ff614c1f1 P 7395562be1584b47bea13989e88c8b52 [term 0 FOLLOWER]: Advancing to term 1
I20250623 02:00:20.248451 21116 raft_consensus.cc:2466] T 9fe242dd38404d08986ef30ff614c1f1 P da8eeec4d1184c43942f5579da019e8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f26b3913c2845988766eae8e7bd2a0b in term 1.
I20250623 02:00:20.248668 21253 raft_consensus.cc:2466] T 9fe242dd38404d08986ef30ff614c1f1 P 7395562be1584b47bea13989e88c8b52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f26b3913c2845988766eae8e7bd2a0b in term 1.
I20250623 02:00:20.249665 20915 leader_election.cc:304] T 9fe242dd38404d08986ef30ff614c1f1 P 9f26b3913c2845988766eae8e7bd2a0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7395562be1584b47bea13989e88c8b52, 9f26b3913c2845988766eae8e7bd2a0b; no voters:
I20250623 02:00:20.250349 21393 raft_consensus.cc:2802] T 9fe242dd38404d08986ef30ff614c1f1 P 9f26b3913c2845988766eae8e7bd2a0b [term 1 FOLLOWER]: Leader election won for term 1
I20250623 02:00:20.251598 21393 raft_consensus.cc:695] T 9fe242dd38404d08986ef30ff614c1f1 P 9f26b3913c2845988766eae8e7bd2a0b [term 1 LEADER]: Becoming Leader. State: Replica: 9f26b3913c2845988766eae8e7bd2a0b, State: Running, Role: LEADER
I20250623 02:00:20.252434 21393 consensus_queue.cc:237] T 9fe242dd38404d08986ef30ff614c1f1 P 9f26b3913c2845988766eae8e7bd2a0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8eeec4d1184c43942f5579da019e8e" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44559 } } peers { permanent_uuid: "7395562be1584b47bea13989e88c8b52" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 46727 } } peers { permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 } }
I20250623 02:00:20.260174 20839 catalog_manager.cc:5582] T 9fe242dd38404d08986ef30ff614c1f1 P 9f26b3913c2845988766eae8e7bd2a0b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f26b3913c2845988766eae8e7bd2a0b (127.14.43.129). New cstate: current_term: 1 leader_uuid: "9f26b3913c2845988766eae8e7bd2a0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8eeec4d1184c43942f5579da019e8e" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7395562be1584b47bea13989e88c8b52" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 46727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f26b3913c2845988766eae8e7bd2a0b" member_type: VOTER last_known_addr { host: "127.14.43.129" port: 45459 } health_report { overall_health: HEALTHY } } }
I20250623 02:00:20.506841 21253 raft_consensus.cc:1273] T 9fe242dd38404d08986ef30ff614c1f1 P 7395562be1584b47bea13989e88c8b52 [term 1 FOLLOWER]: Refusing update from remote peer 9f26b3913c2845988766eae8e7bd2a0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250623 02:00:20.507386 21116 raft_consensus.cc:1273] T 9fe242dd38404d08986ef30ff614c1f1 P da8eeec4d1184c43942f5579da019e8e [term 1 FOLLOWER]: Refusing update from remote peer 9f26b3913c2845988766eae8e7bd2a0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250623 02:00:20.508592 21393 consensus_queue.cc:1035] T 9fe242dd38404d08986ef30ff614c1f1 P 9f26b3913c2845988766eae8e7bd2a0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7395562be1584b47bea13989e88c8b52" member_type: VOTER last_known_addr { host: "127.14.43.131" port: 46727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 02:00:20.509443 21395 consensus_queue.cc:1035] T 9fe242dd38404d08986ef30ff614c1f1 P 9f26b3913c2845988766eae8e7bd2a0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "da8eeec4d1184c43942f5579da019e8e" member_type: VOTER last_known_addr { host: "127.14.43.130" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 02:00:20.582296 21409 mvcc.cc:204] Tried to move back new op lower bound from 7170637907977871360 to 7170637906960539648. Current Snapshot: MvccSnapshot[applied={T|T < 7170637907977871360}]
I20250623 02:00:20.625360 21404 mvcc.cc:204] Tried to move back new op lower bound from 7170637907977871360 to 7170637906960539648. Current Snapshot: MvccSnapshot[applied={T|T < 7170637907977871360}]
W20250623 02:00:24.208447 21379 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.14.43.129:45459, user_credentials={real_user=slave}} blocked reactor thread for 44112.3us
I20250623 02:00:25.280674 21233 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250623 02:00:25.287308 20961 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250623 02:00:25.287521 21096 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
5d090a643c8f494fac466bb333e2527b | 127.14.43.190:40161 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.14.43.148:33689 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
7395562be1584b47bea13989e88c8b52 | 127.14.43.131:46727 | HEALTHY | <none> | 0 | 0
9f26b3913c2845988766eae8e7bd2a0b | 127.14.43.129:45459 | HEALTHY | <none> | 1 | 0
da8eeec4d1184c43942f5579da019e8e | 127.14.43.130:44559 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.14.43.129 | experimental | 127.14.43.129:45459
local_ip_for_outbound_sockets | 127.14.43.130 | experimental | 127.14.43.130:44559
local_ip_for_outbound_sockets | 127.14.43.131 | experimental | 127.14.43.131:46727
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-0/data/info.pb | hidden | 127.14.43.129:45459
server_dump_info_path | /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-1/data/info.pb | hidden | 127.14.43.130:44559
server_dump_info_path | /tmp/dist-test-taskZdkTaH/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750643853547882-14510-0/minicluster-data/ts-2/data/info.pb | hidden | 127.14.43.131:46727
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.14.43.148:33689 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
--------------+----+---------+---------------+---------+------------+------------------+-------------
post_rebuild | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250623 02:00:25.496115 14510 log_verifier.cc:126] Checking tablet 9fe242dd38404d08986ef30ff614c1f1
I20250623 02:00:26.257536 14510 log_verifier.cc:177] Verified matching terms for 205 ops in tablet 9fe242dd38404d08986ef30ff614c1f1
I20250623 02:00:26.258531 14510 log_verifier.cc:126] Checking tablet 9ffd15b775094bbb920b3cb00b356a30
I20250623 02:00:26.258796 14510 log_verifier.cc:177] Verified matching terms for 0 ops in tablet 9ffd15b775094bbb920b3cb00b356a30
I20250623 02:00:26.283377 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 20876
I20250623 02:00:26.321151 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 21030
I20250623 02:00:26.351531 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 21165
I20250623 02:00:26.379392 14510 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskZdkTaH/build/tsan/bin/kudu with pid 20806
2025-06-23T02:00:26Z chronyd exiting
[ OK ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0 (34319 ms)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest (34319 ms total)
[----------] Global test environment tear-down
[==========] 9 tests from 5 test suites ran. (172818 ms total)
[ PASSED ] 8 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AdminCliTest.TestRebuildTables
1 FAILED TEST