Note: This is test shard 6 of 8.
[==========] Running 9 tests from 5 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from AdminCliTest
[ RUN ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250626 01:57:37.779156 16463 test_util.cc:276] Using random seed: -489977551
W20250626 01:57:38.994908 16463 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.170s user 0.474s sys 0.694s
W20250626 01:57:38.995361 16463 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.170s user 0.474s sys 0.694s
I20250626 01:57:38.997635 16463 ts_itest-base.cc:115] Starting cluster with:
I20250626 01:57:38.997893 16463 ts_itest-base.cc:116] --------------
I20250626 01:57:38.998095 16463 ts_itest-base.cc:117] 4 tablet servers
I20250626 01:57:38.998322 16463 ts_itest-base.cc:118] 3 replicas per TS
I20250626 01:57:38.998536 16463 ts_itest-base.cc:119] --------------
2025-06-26T01:57:39Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-26T01:57:39Z Disabled control of system clock
I20250626 01:57:39.037503 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.19.254:43727
--webserver_interface=127.16.19.254
--webserver_port=0
--builtin_ntp_servers=127.16.19.212:40729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.19.254:43727 with env {}
W20250626 01:57:39.328907 16477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:57:39.329493 16477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 01:57:39.329954 16477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:57:39.361362 16477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 01:57:39.361696 16477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 01:57:39.361945 16477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 01:57:39.362176 16477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 01:57:39.396940 16477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:40729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.19.254:43727
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.19.254:43727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.19.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 01:57:39.398238 16477 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 01:57:39.399878 16477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:57:39.413647 16484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:57:39.413662 16483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:57:39.416064 16486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:57:40.594719 16485 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 01:57:40.594856 16477 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 01:57:40.598989 16477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 01:57:40.602164 16477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 01:57:40.603621 16477 hybrid_clock.cc:648] HybridClock initialized: now 1750903060603580 us; error 56 us; skew 500 ppm
I20250626 01:57:40.604439 16477 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:57:40.615435 16477 webserver.cc:469] Webserver started at http://127.16.19.254:43057/ using document root <none> and password file <none>
I20250626 01:57:40.616429 16477 fs_manager.cc:362] Metadata directory not provided
I20250626 01:57:40.616703 16477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:57:40.617179 16477 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:57:40.621778 16477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "904ef2bacbb14a33b522e0bf8c3b3c0d"
format_stamp: "Formatted at 2025-06-26 01:57:40 on dist-test-slave-h4rq"
I20250626 01:57:40.622884 16477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "904ef2bacbb14a33b522e0bf8c3b3c0d"
format_stamp: "Formatted at 2025-06-26 01:57:40 on dist-test-slave-h4rq"
I20250626 01:57:40.630102 16477 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20250626 01:57:40.635584 16493 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:57:40.636520 16477 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20250626 01:57:40.636886 16477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
uuid: "904ef2bacbb14a33b522e0bf8c3b3c0d"
format_stamp: "Formatted at 2025-06-26 01:57:40 on dist-test-slave-h4rq"
I20250626 01:57:40.637217 16477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:57:40.692421 16477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:57:40.693950 16477 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 01:57:40.694377 16477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:57:40.765451 16477 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.254:43727
I20250626 01:57:40.765527 16544 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.254:43727 every 8 connection(s)
I20250626 01:57:40.768069 16477 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250626 01:57:40.773167 16545 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:57:40.774861 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 16477
I20250626 01:57:40.775420 16463 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250626 01:57:40.794083 16545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d: Bootstrap starting.
I20250626 01:57:40.799448 16545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d: Neither blocks nor log segments found. Creating new log.
I20250626 01:57:40.801831 16545 log.cc:826] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d: Log is configured to *not* fsync() on all Append() calls
I20250626 01:57:40.807097 16545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d: No bootstrap required, opened a new log
I20250626 01:57:40.826895 16545 raft_consensus.cc:357] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "904ef2bacbb14a33b522e0bf8c3b3c0d" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 43727 } }
I20250626 01:57:40.827574 16545 raft_consensus.cc:383] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:57:40.827791 16545 raft_consensus.cc:738] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 904ef2bacbb14a33b522e0bf8c3b3c0d, State: Initialized, Role: FOLLOWER
I20250626 01:57:40.828409 16545 consensus_queue.cc:260] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "904ef2bacbb14a33b522e0bf8c3b3c0d" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 43727 } }
I20250626 01:57:40.828980 16545 raft_consensus.cc:397] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 01:57:40.829246 16545 raft_consensus.cc:491] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 01:57:40.829560 16545 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:57:40.833647 16545 raft_consensus.cc:513] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "904ef2bacbb14a33b522e0bf8c3b3c0d" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 43727 } }
I20250626 01:57:40.834414 16545 leader_election.cc:304] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 904ef2bacbb14a33b522e0bf8c3b3c0d; no voters:
I20250626 01:57:40.836330 16545 leader_election.cc:290] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 01:57:40.837181 16550 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:57:40.839310 16550 raft_consensus.cc:695] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d [term 1 LEADER]: Becoming Leader. State: Replica: 904ef2bacbb14a33b522e0bf8c3b3c0d, State: Running, Role: LEADER
I20250626 01:57:40.840101 16550 consensus_queue.cc:237] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "904ef2bacbb14a33b522e0bf8c3b3c0d" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 43727 } }
I20250626 01:57:40.841454 16545 sys_catalog.cc:564] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d [sys.catalog]: configured and running, proceeding with master startup.
I20250626 01:57:40.851699 16552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "904ef2bacbb14a33b522e0bf8c3b3c0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "904ef2bacbb14a33b522e0bf8c3b3c0d" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 43727 } } }
I20250626 01:57:40.851857 16551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 904ef2bacbb14a33b522e0bf8c3b3c0d. Latest consensus state: current_term: 1 leader_uuid: "904ef2bacbb14a33b522e0bf8c3b3c0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "904ef2bacbb14a33b522e0bf8c3b3c0d" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 43727 } } }
I20250626 01:57:40.852749 16551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d [sys.catalog]: This master's current role is: LEADER
I20250626 01:57:40.852650 16552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d [sys.catalog]: This master's current role is: LEADER
I20250626 01:57:40.857268 16560 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 01:57:40.869434 16560 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 01:57:40.885401 16560 catalog_manager.cc:1349] Generated new cluster ID: 26edc70c6efe4ccb950f893589f214e4
I20250626 01:57:40.885660 16560 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 01:57:40.908730 16560 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 01:57:40.910131 16560 catalog_manager.cc:1506] Loading token signing keys...
I20250626 01:57:40.924434 16560 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d: Generated new TSK 0
I20250626 01:57:40.925367 16560 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 01:57:40.942437 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.19.193:0
--local_ip_for_outbound_sockets=127.16.19.193
--webserver_interface=127.16.19.193
--webserver_port=0
--tserver_master_addrs=127.16.19.254:43727
--builtin_ntp_servers=127.16.19.212:40729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250626 01:57:41.263728 16569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:57:41.264281 16569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 01:57:41.264900 16569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:57:41.297498 16569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 01:57:41.298399 16569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.19.193
I20250626 01:57:41.333792 16569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:40729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.19.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.19.193
--webserver_port=0
--tserver_master_addrs=127.16.19.254:43727
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.19.193
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 01:57:41.335122 16569 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 01:57:41.336894 16569 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:57:41.355346 16576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:57:41.356441 16575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:57:41.358096 16578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:57:42.542205 16577 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 01:57:42.542687 16569 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 01:57:42.546634 16569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 01:57:42.557639 16569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 01:57:42.559137 16569 hybrid_clock.cc:648] HybridClock initialized: now 1750903062559076 us; error 88 us; skew 500 ppm
I20250626 01:57:42.560017 16569 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:57:42.567451 16569 webserver.cc:469] Webserver started at http://127.16.19.193:43849/ using document root <none> and password file <none>
I20250626 01:57:42.568406 16569 fs_manager.cc:362] Metadata directory not provided
I20250626 01:57:42.568642 16569 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:57:42.569136 16569 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:57:42.573737 16569 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "8711c0db646c4f09930414c4e99ade2c"
format_stamp: "Formatted at 2025-06-26 01:57:42 on dist-test-slave-h4rq"
I20250626 01:57:42.574862 16569 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "8711c0db646c4f09930414c4e99ade2c"
format_stamp: "Formatted at 2025-06-26 01:57:42 on dist-test-slave-h4rq"
I20250626 01:57:42.582494 16569 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20250626 01:57:42.588599 16585 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:57:42.589730 16569 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250626 01:57:42.590037 16569 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "8711c0db646c4f09930414c4e99ade2c"
format_stamp: "Formatted at 2025-06-26 01:57:42 on dist-test-slave-h4rq"
I20250626 01:57:42.590349 16569 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:57:42.646963 16569 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:57:42.648415 16569 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 01:57:42.648877 16569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:57:42.651307 16569 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:57:42.655534 16569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:57:42.655740 16569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:57:42.656033 16569 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:57:42.656203 16569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:57:42.791289 16569 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.193:33947
I20250626 01:57:42.791383 16697 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.193:33947 every 8 connection(s)
I20250626 01:57:42.793944 16569 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250626 01:57:42.796402 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 16569
I20250626 01:57:42.797006 16463 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250626 01:57:42.804201 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.19.194:0
--local_ip_for_outbound_sockets=127.16.19.194
--webserver_interface=127.16.19.194
--webserver_port=0
--tserver_master_addrs=127.16.19.254:43727
--builtin_ntp_servers=127.16.19.212:40729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 01:57:42.815447 16698 heartbeater.cc:344] Connected to a master server at 127.16.19.254:43727
I20250626 01:57:42.815933 16698 heartbeater.cc:461] Registering TS with master...
I20250626 01:57:42.817003 16698 heartbeater.cc:507] Master 127.16.19.254:43727 requested a full tablet report, sending...
I20250626 01:57:42.819484 16510 ts_manager.cc:194] Registered new tserver with Master: 8711c0db646c4f09930414c4e99ade2c (127.16.19.193:33947)
I20250626 01:57:42.821411 16510 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.19.193:53029
W20250626 01:57:43.104081 16702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:57:43.104637 16702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 01:57:43.105209 16702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:57:43.137393 16702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 01:57:43.138218 16702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.19.194
I20250626 01:57:43.173146 16702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:40729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.19.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.16.19.194
--webserver_port=0
--tserver_master_addrs=127.16.19.254:43727
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.19.194
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 01:57:43.174376 16702 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 01:57:43.175937 16702 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:57:43.190832 16709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:57:43.825755 16698 heartbeater.cc:499] Master 127.16.19.254:43727 was elected leader, sending a full tablet report...
W20250626 01:57:43.191028 16708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:57:43.193094 16711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:57:44.411168 16710 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1218 milliseconds
I20250626 01:57:44.411299 16702 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 01:57:44.412503 16702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 01:57:44.414706 16702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 01:57:44.416069 16702 hybrid_clock.cc:648] HybridClock initialized: now 1750903064416034 us; error 55 us; skew 500 ppm
I20250626 01:57:44.416924 16702 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:57:44.423429 16702 webserver.cc:469] Webserver started at http://127.16.19.194:41631/ using document root <none> and password file <none>
I20250626 01:57:44.424369 16702 fs_manager.cc:362] Metadata directory not provided
I20250626 01:57:44.424571 16702 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:57:44.425096 16702 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:57:44.429507 16702 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "564ffe4de0df4f54bfe8b725c5fac685"
format_stamp: "Formatted at 2025-06-26 01:57:44 on dist-test-slave-h4rq"
I20250626 01:57:44.430624 16702 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "564ffe4de0df4f54bfe8b725c5fac685"
format_stamp: "Formatted at 2025-06-26 01:57:44 on dist-test-slave-h4rq"
I20250626 01:57:44.437614 16702 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20250626 01:57:44.442955 16718 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:57:44.443955 16702 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250626 01:57:44.444252 16702 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "564ffe4de0df4f54bfe8b725c5fac685"
format_stamp: "Formatted at 2025-06-26 01:57:44 on dist-test-slave-h4rq"
I20250626 01:57:44.444557 16702 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:57:44.504906 16702 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:57:44.506358 16702 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 01:57:44.506805 16702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:57:44.509295 16702 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:57:44.513413 16702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:57:44.513628 16702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:57:44.513903 16702 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:57:44.514058 16702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:57:44.651851 16702 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.194:44795
I20250626 01:57:44.651952 16830 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.194:44795 every 8 connection(s)
I20250626 01:57:44.654402 16702 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250626 01:57:44.657168 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 16702
I20250626 01:57:44.657672 16463 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250626 01:57:44.664234 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.19.195:0
--local_ip_for_outbound_sockets=127.16.19.195
--webserver_interface=127.16.19.195
--webserver_port=0
--tserver_master_addrs=127.16.19.254:43727
--builtin_ntp_servers=127.16.19.212:40729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 01:57:44.675352 16831 heartbeater.cc:344] Connected to a master server at 127.16.19.254:43727
I20250626 01:57:44.675930 16831 heartbeater.cc:461] Registering TS with master...
I20250626 01:57:44.677309 16831 heartbeater.cc:507] Master 127.16.19.254:43727 requested a full tablet report, sending...
I20250626 01:57:44.679704 16510 ts_manager.cc:194] Registered new tserver with Master: 564ffe4de0df4f54bfe8b725c5fac685 (127.16.19.194:44795)
I20250626 01:57:44.681676 16510 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.19.194:39093
W20250626 01:57:44.975395 16835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:57:44.975878 16835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 01:57:44.976325 16835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:57:45.007078 16835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 01:57:45.007875 16835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.19.195
I20250626 01:57:45.042129 16835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:40729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.19.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.16.19.195
--webserver_port=0
--tserver_master_addrs=127.16.19.254:43727
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.19.195
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 01:57:45.043419 16835 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 01:57:45.045048 16835 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:57:45.062240 16844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:57:45.684966 16831 heartbeater.cc:499] Master 127.16.19.254:43727 was elected leader, sending a full tablet report...
W20250626 01:57:45.062433 16841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:57:45.062423 16842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:57:46.228320 16843 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 01:57:46.228466 16835 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 01:57:46.233047 16835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 01:57:46.235770 16835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 01:57:46.237203 16835 hybrid_clock.cc:648] HybridClock initialized: now 1750903066237171 us; error 56 us; skew 500 ppm
I20250626 01:57:46.238009 16835 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:57:46.250495 16835 webserver.cc:469] Webserver started at http://127.16.19.195:37089/ using document root <none> and password file <none>
I20250626 01:57:46.251466 16835 fs_manager.cc:362] Metadata directory not provided
I20250626 01:57:46.251674 16835 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:57:46.252144 16835 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:57:46.256601 16835 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "95051b8c263847058d44f0e678adbee5"
format_stamp: "Formatted at 2025-06-26 01:57:46 on dist-test-slave-h4rq"
I20250626 01:57:46.257841 16835 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "95051b8c263847058d44f0e678adbee5"
format_stamp: "Formatted at 2025-06-26 01:57:46 on dist-test-slave-h4rq"
I20250626 01:57:46.265113 16835 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250626 01:57:46.270821 16851 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:57:46.272061 16835 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250626 01:57:46.272387 16835 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "95051b8c263847058d44f0e678adbee5"
format_stamp: "Formatted at 2025-06-26 01:57:46 on dist-test-slave-h4rq"
I20250626 01:57:46.272768 16835 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:57:46.320300 16835 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:57:46.321775 16835 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 01:57:46.322203 16835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:57:46.324767 16835 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:57:46.328794 16835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:57:46.328984 16835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:57:46.329255 16835 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:57:46.329409 16835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:57:46.466559 16835 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.195:37143
I20250626 01:57:46.466657 16963 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.195:37143 every 8 connection(s)
I20250626 01:57:46.469306 16835 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250626 01:57:46.476416 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 16835
I20250626 01:57:46.476903 16463 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250626 01:57:46.482756 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.19.196:0
--local_ip_for_outbound_sockets=127.16.19.196
--webserver_interface=127.16.19.196
--webserver_port=0
--tserver_master_addrs=127.16.19.254:43727
--builtin_ntp_servers=127.16.19.212:40729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 01:57:46.491104 16964 heartbeater.cc:344] Connected to a master server at 127.16.19.254:43727
I20250626 01:57:46.491532 16964 heartbeater.cc:461] Registering TS with master...
I20250626 01:57:46.492563 16964 heartbeater.cc:507] Master 127.16.19.254:43727 requested a full tablet report, sending...
I20250626 01:57:46.494789 16510 ts_manager.cc:194] Registered new tserver with Master: 95051b8c263847058d44f0e678adbee5 (127.16.19.195:37143)
I20250626 01:57:46.495998 16510 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.19.195:49977
W20250626 01:57:46.783597 16968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:57:46.784091 16968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 01:57:46.784595 16968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:57:46.823813 16968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 01:57:46.824716 16968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.19.196
I20250626 01:57:46.860116 16968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:40729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.19.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.16.19.196
--webserver_port=0
--tserver_master_addrs=127.16.19.254:43727
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.19.196
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 01:57:46.861470 16968 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 01:57:46.863368 16968 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:57:46.880937 16975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:57:47.499092 16964 heartbeater.cc:499] Master 127.16.19.254:43727 was elected leader, sending a full tablet report...
W20250626 01:57:46.880928 16977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:57:46.882606 16974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:57:48.037654 16968 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250626 01:57:48.038209 16976 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 01:57:48.041654 16968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 01:57:48.043920 16968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 01:57:48.045248 16968 hybrid_clock.cc:648] HybridClock initialized: now 1750903068045213 us; error 55 us; skew 500 ppm
I20250626 01:57:48.046038 16968 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:57:48.052958 16968 webserver.cc:469] Webserver started at http://127.16.19.196:40541/ using document root <none> and password file <none>
I20250626 01:57:48.053857 16968 fs_manager.cc:362] Metadata directory not provided
I20250626 01:57:48.054073 16968 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:57:48.054520 16968 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:57:48.060441 16968 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "d6154bdb58394449bbca4ce76fb2356e"
format_stamp: "Formatted at 2025-06-26 01:57:48 on dist-test-slave-h4rq"
I20250626 01:57:48.061578 16968 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "d6154bdb58394449bbca4ce76fb2356e"
format_stamp: "Formatted at 2025-06-26 01:57:48 on dist-test-slave-h4rq"
I20250626 01:57:48.068533 16968 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250626 01:57:48.074069 16985 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:57:48.075040 16968 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20250626 01:57:48.075353 16968 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "d6154bdb58394449bbca4ce76fb2356e"
format_stamp: "Formatted at 2025-06-26 01:57:48 on dist-test-slave-h4rq"
I20250626 01:57:48.075683 16968 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:57:48.132431 16968 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:57:48.133910 16968 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 01:57:48.134342 16968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:57:48.136904 16968 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:57:48.141032 16968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:57:48.141284 16968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:57:48.141531 16968 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:57:48.141691 16968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:57:48.274950 16968 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.196:36263
I20250626 01:57:48.275063 17097 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.196:36263 every 8 connection(s)
I20250626 01:57:48.277597 16968 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250626 01:57:48.283665 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 16968
I20250626 01:57:48.284439 16463 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250626 01:57:48.298450 17098 heartbeater.cc:344] Connected to a master server at 127.16.19.254:43727
I20250626 01:57:48.298885 17098 heartbeater.cc:461] Registering TS with master...
I20250626 01:57:48.299856 17098 heartbeater.cc:507] Master 127.16.19.254:43727 requested a full tablet report, sending...
I20250626 01:57:48.302006 16510 ts_manager.cc:194] Registered new tserver with Master: d6154bdb58394449bbca4ce76fb2356e (127.16.19.196:36263)
I20250626 01:57:48.303397 16510 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.19.196:47727
I20250626 01:57:48.305639 16463 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250626 01:57:48.344456 16510 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58692:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250626 01:57:48.412242 17033 tablet_service.cc:1468] Processing CreateTablet for tablet 75c7fafa68214e0c82edbb7d3479774d (DEFAULT_TABLE table=TestTable [id=39738b9209294ab6a5f7102b84047fa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 01:57:48.414247 17033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75c7fafa68214e0c82edbb7d3479774d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:57:48.413686 16766 tablet_service.cc:1468] Processing CreateTablet for tablet 75c7fafa68214e0c82edbb7d3479774d (DEFAULT_TABLE table=TestTable [id=39738b9209294ab6a5f7102b84047fa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 01:57:48.415484 16766 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75c7fafa68214e0c82edbb7d3479774d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:57:48.425137 16633 tablet_service.cc:1468] Processing CreateTablet for tablet 75c7fafa68214e0c82edbb7d3479774d (DEFAULT_TABLE table=TestTable [id=39738b9209294ab6a5f7102b84047fa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 01:57:48.427224 16633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75c7fafa68214e0c82edbb7d3479774d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:57:48.438295 17117 tablet_bootstrap.cc:492] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e: Bootstrap starting.
I20250626 01:57:48.448597 17118 tablet_bootstrap.cc:492] T 75c7fafa68214e0c82edbb7d3479774d P 564ffe4de0df4f54bfe8b725c5fac685: Bootstrap starting.
I20250626 01:57:48.451189 17117 tablet_bootstrap.cc:654] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e: Neither blocks nor log segments found. Creating new log.
I20250626 01:57:48.455703 17117 log.cc:826] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e: Log is configured to *not* fsync() on all Append() calls
I20250626 01:57:48.456959 17119 tablet_bootstrap.cc:492] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c: Bootstrap starting.
I20250626 01:57:48.457607 17118 tablet_bootstrap.cc:654] T 75c7fafa68214e0c82edbb7d3479774d P 564ffe4de0df4f54bfe8b725c5fac685: Neither blocks nor log segments found. Creating new log.
I20250626 01:57:48.460088 17118 log.cc:826] T 75c7fafa68214e0c82edbb7d3479774d P 564ffe4de0df4f54bfe8b725c5fac685: Log is configured to *not* fsync() on all Append() calls
I20250626 01:57:48.462347 17117 tablet_bootstrap.cc:492] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e: No bootstrap required, opened a new log
I20250626 01:57:48.462945 17117 ts_tablet_manager.cc:1397] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e: Time spent bootstrapping tablet: real 0.025s user 0.017s sys 0.000s
I20250626 01:57:48.465572 17118 tablet_bootstrap.cc:492] T 75c7fafa68214e0c82edbb7d3479774d P 564ffe4de0df4f54bfe8b725c5fac685: No bootstrap required, opened a new log
I20250626 01:57:48.465910 17119 tablet_bootstrap.cc:654] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c: Neither blocks nor log segments found. Creating new log.
I20250626 01:57:48.466116 17118 ts_tablet_manager.cc:1397] T 75c7fafa68214e0c82edbb7d3479774d P 564ffe4de0df4f54bfe8b725c5fac685: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.003s
I20250626 01:57:48.468395 17119 log.cc:826] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c: Log is configured to *not* fsync() on all Append() calls
I20250626 01:57:48.482642 17119 tablet_bootstrap.cc:492] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c: No bootstrap required, opened a new log
I20250626 01:57:48.483222 17119 ts_tablet_manager.cc:1397] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.007s
I20250626 01:57:48.489805 17118 raft_consensus.cc:357] T 75c7fafa68214e0c82edbb7d3479774d P 564ffe4de0df4f54bfe8b725c5fac685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" member_type: VOTER last_known_addr { host: "127.16.19.196" port: 36263 } } peers { permanent_uuid: "564ffe4de0df4f54bfe8b725c5fac685" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 44795 } } peers { permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 } }
I20250626 01:57:48.490502 17118 raft_consensus.cc:383] T 75c7fafa68214e0c82edbb7d3479774d P 564ffe4de0df4f54bfe8b725c5fac685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:57:48.490787 17118 raft_consensus.cc:738] T 75c7fafa68214e0c82edbb7d3479774d P 564ffe4de0df4f54bfe8b725c5fac685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 564ffe4de0df4f54bfe8b725c5fac685, State: Initialized, Role: FOLLOWER
I20250626 01:57:48.491619 17118 consensus_queue.cc:260] T 75c7fafa68214e0c82edbb7d3479774d P 564ffe4de0df4f54bfe8b725c5fac685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" member_type: VOTER last_known_addr { host: "127.16.19.196" port: 36263 } } peers { permanent_uuid: "564ffe4de0df4f54bfe8b725c5fac685" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 44795 } } peers { permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 } }
I20250626 01:57:48.495684 17118 ts_tablet_manager.cc:1428] T 75c7fafa68214e0c82edbb7d3479774d P 564ffe4de0df4f54bfe8b725c5fac685: Time spent starting tablet: real 0.029s user 0.024s sys 0.003s
I20250626 01:57:48.496750 17117 raft_consensus.cc:357] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" member_type: VOTER last_known_addr { host: "127.16.19.196" port: 36263 } } peers { permanent_uuid: "564ffe4de0df4f54bfe8b725c5fac685" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 44795 } } peers { permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 } }
I20250626 01:57:48.497792 17117 raft_consensus.cc:383] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:57:48.498140 17117 raft_consensus.cc:738] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6154bdb58394449bbca4ce76fb2356e, State: Initialized, Role: FOLLOWER
I20250626 01:57:48.499091 17117 consensus_queue.cc:260] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" member_type: VOTER last_known_addr { host: "127.16.19.196" port: 36263 } } peers { permanent_uuid: "564ffe4de0df4f54bfe8b725c5fac685" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 44795 } } peers { permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 } }
I20250626 01:57:48.502378 17098 heartbeater.cc:499] Master 127.16.19.254:43727 was elected leader, sending a full tablet report...
I20250626 01:57:48.503823 17117 ts_tablet_manager.cc:1428] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e: Time spent starting tablet: real 0.041s user 0.030s sys 0.003s
I20250626 01:57:48.511073 17119 raft_consensus.cc:357] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" member_type: VOTER last_known_addr { host: "127.16.19.196" port: 36263 } } peers { permanent_uuid: "564ffe4de0df4f54bfe8b725c5fac685" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 44795 } } peers { permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 } }
I20250626 01:57:48.512058 17119 raft_consensus.cc:383] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:57:48.512315 17119 raft_consensus.cc:738] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8711c0db646c4f09930414c4e99ade2c, State: Initialized, Role: FOLLOWER
I20250626 01:57:48.513231 17119 consensus_queue.cc:260] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" member_type: VOTER last_known_addr { host: "127.16.19.196" port: 36263 } } peers { permanent_uuid: "564ffe4de0df4f54bfe8b725c5fac685" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 44795 } } peers { permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 } }
I20250626 01:57:48.516475 17119 ts_tablet_manager.cc:1428] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c: Time spent starting tablet: real 0.033s user 0.024s sys 0.008s
I20250626 01:57:48.529081 17123 raft_consensus.cc:491] T 75c7fafa68214e0c82edbb7d3479774d P 564ffe4de0df4f54bfe8b725c5fac685 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:57:48.529595 17123 raft_consensus.cc:513] T 75c7fafa68214e0c82edbb7d3479774d P 564ffe4de0df4f54bfe8b725c5fac685 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" member_type: VOTER last_known_addr { host: "127.16.19.196" port: 36263 } } peers { permanent_uuid: "564ffe4de0df4f54bfe8b725c5fac685" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 44795 } } peers { permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 } }
I20250626 01:57:48.532228 17123 leader_election.cc:290] T 75c7fafa68214e0c82edbb7d3479774d P 564ffe4de0df4f54bfe8b725c5fac685 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6154bdb58394449bbca4ce76fb2356e (127.16.19.196:36263), 8711c0db646c4f09930414c4e99ade2c (127.16.19.193:33947)
W20250626 01:57:48.532397 17099 tablet.cc:2378] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 01:57:48.544999 17053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75c7fafa68214e0c82edbb7d3479774d" candidate_uuid: "564ffe4de0df4f54bfe8b725c5fac685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6154bdb58394449bbca4ce76fb2356e" is_pre_election: true
I20250626 01:57:48.545320 16653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75c7fafa68214e0c82edbb7d3479774d" candidate_uuid: "564ffe4de0df4f54bfe8b725c5fac685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8711c0db646c4f09930414c4e99ade2c" is_pre_election: true
I20250626 01:57:48.545728 17053 raft_consensus.cc:2466] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 564ffe4de0df4f54bfe8b725c5fac685 in term 0.
I20250626 01:57:48.546064 16653 raft_consensus.cc:2466] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 564ffe4de0df4f54bfe8b725c5fac685 in term 0.
I20250626 01:57:48.547030 16719 leader_election.cc:304] T 75c7fafa68214e0c82edbb7d3479774d P 564ffe4de0df4f54bfe8b725c5fac685 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 564ffe4de0df4f54bfe8b725c5fac685, d6154bdb58394449bbca4ce76fb2356e; no voters:
I20250626 01:57:48.547910 17123 raft_consensus.cc:2802] T 75c7fafa68214e0c82edbb7d3479774d P 564ffe4de0df4f54bfe8b725c5fac685 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:57:48.548164 17123 raft_consensus.cc:491] T 75c7fafa68214e0c82edbb7d3479774d P 564ffe4de0df4f54bfe8b725c5fac685 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:57:48.548425 17123 raft_consensus.cc:3058] T 75c7fafa68214e0c82edbb7d3479774d P 564ffe4de0df4f54bfe8b725c5fac685 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:57:48.552785 17123 raft_consensus.cc:513] T 75c7fafa68214e0c82edbb7d3479774d P 564ffe4de0df4f54bfe8b725c5fac685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" member_type: VOTER last_known_addr { host: "127.16.19.196" port: 36263 } } peers { permanent_uuid: "564ffe4de0df4f54bfe8b725c5fac685" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 44795 } } peers { permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 } }
I20250626 01:57:48.554159 17123 leader_election.cc:290] T 75c7fafa68214e0c82edbb7d3479774d P 564ffe4de0df4f54bfe8b725c5fac685 [CANDIDATE]: Term 1 election: Requested vote from peers d6154bdb58394449bbca4ce76fb2356e (127.16.19.196:36263), 8711c0db646c4f09930414c4e99ade2c (127.16.19.193:33947)
W20250626 01:57:48.554586 16699 tablet.cc:2378] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 01:57:48.555087 17053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75c7fafa68214e0c82edbb7d3479774d" candidate_uuid: "564ffe4de0df4f54bfe8b725c5fac685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6154bdb58394449bbca4ce76fb2356e"
I20250626 01:57:48.555182 16653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75c7fafa68214e0c82edbb7d3479774d" candidate_uuid: "564ffe4de0df4f54bfe8b725c5fac685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8711c0db646c4f09930414c4e99ade2c"
I20250626 01:57:48.555610 16653 raft_consensus.cc:3058] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:57:48.555608 17053 raft_consensus.cc:3058] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:57:48.560060 16653 raft_consensus.cc:2466] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 564ffe4de0df4f54bfe8b725c5fac685 in term 1.
I20250626 01:57:48.560070 17053 raft_consensus.cc:2466] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 564ffe4de0df4f54bfe8b725c5fac685 in term 1.
I20250626 01:57:48.561045 16719 leader_election.cc:304] T 75c7fafa68214e0c82edbb7d3479774d P 564ffe4de0df4f54bfe8b725c5fac685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 564ffe4de0df4f54bfe8b725c5fac685, 8711c0db646c4f09930414c4e99ade2c, d6154bdb58394449bbca4ce76fb2356e; no voters:
I20250626 01:57:48.561738 17123 raft_consensus.cc:2802] T 75c7fafa68214e0c82edbb7d3479774d P 564ffe4de0df4f54bfe8b725c5fac685 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:57:48.563300 17123 raft_consensus.cc:695] T 75c7fafa68214e0c82edbb7d3479774d P 564ffe4de0df4f54bfe8b725c5fac685 [term 1 LEADER]: Becoming Leader. State: Replica: 564ffe4de0df4f54bfe8b725c5fac685, State: Running, Role: LEADER
I20250626 01:57:48.564067 17123 consensus_queue.cc:237] T 75c7fafa68214e0c82edbb7d3479774d P 564ffe4de0df4f54bfe8b725c5fac685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" member_type: VOTER last_known_addr { host: "127.16.19.196" port: 36263 } } peers { permanent_uuid: "564ffe4de0df4f54bfe8b725c5fac685" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 44795 } } peers { permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 } }
I20250626 01:57:48.576071 16508 catalog_manager.cc:5582] T 75c7fafa68214e0c82edbb7d3479774d P 564ffe4de0df4f54bfe8b725c5fac685 reported cstate change: term changed from 0 to 1, leader changed from <none> to 564ffe4de0df4f54bfe8b725c5fac685 (127.16.19.194). New cstate: current_term: 1 leader_uuid: "564ffe4de0df4f54bfe8b725c5fac685" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" member_type: VOTER last_known_addr { host: "127.16.19.196" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "564ffe4de0df4f54bfe8b725c5fac685" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 44795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 } health_report { overall_health: UNKNOWN } } }
I20250626 01:57:48.631645 16463 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250626 01:57:48.636468 16463 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 8711c0db646c4f09930414c4e99ade2c to finish bootstrapping
I20250626 01:57:48.658051 16463 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 564ffe4de0df4f54bfe8b725c5fac685 to finish bootstrapping
W20250626 01:57:48.663444 16832 tablet.cc:2378] T 75c7fafa68214e0c82edbb7d3479774d P 564ffe4de0df4f54bfe8b725c5fac685: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 01:57:48.670817 16463 ts_itest-base.cc:246] Waiting for 1 tablets on tserver d6154bdb58394449bbca4ce76fb2356e to finish bootstrapping
I20250626 01:57:48.683123 16463 kudu-admin-test.cc:709] Waiting for Master to see the current replicas...
I20250626 01:57:48.686465 16463 kudu-admin-test.cc:716] Tablet locations:
tablet_locations {
tablet_id: "75c7fafa68214e0c82edbb7d3479774d"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: LEADER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e"
rpc_addresses {
host: "127.16.19.196"
port: 36263
}
}
ts_infos {
permanent_uuid: "564ffe4de0df4f54bfe8b725c5fac685"
rpc_addresses {
host: "127.16.19.194"
port: 44795
}
}
ts_infos {
permanent_uuid: "8711c0db646c4f09930414c4e99ade2c"
rpc_addresses {
host: "127.16.19.193"
port: 33947
}
}
I20250626 01:57:49.123212 17132 consensus_queue.cc:1035] T 75c7fafa68214e0c82edbb7d3479774d P 564ffe4de0df4f54bfe8b725c5fac685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:57:49.141729 17132 consensus_queue.cc:1035] T 75c7fafa68214e0c82edbb7d3479774d P 564ffe4de0df4f54bfe8b725c5fac685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" member_type: VOTER last_known_addr { host: "127.16.19.196" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:57:49.145911 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 16702
I20250626 01:57:49.171159 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 16477
I20250626 01:57:49.201479 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.19.254:43727
--webserver_interface=127.16.19.254
--webserver_port=43057
--builtin_ntp_servers=127.16.19.212:40729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.19.254:43727 with env {}
W20250626 01:57:49.501437 17139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:57:49.501991 17139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 01:57:49.502403 17139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:57:49.510512 16964 heartbeater.cc:646] Failed to heartbeat to 127.16.19.254:43727 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.19.254:43727: connect: Connection refused (error 111)
W20250626 01:57:49.517771 17098 heartbeater.cc:646] Failed to heartbeat to 127.16.19.254:43727 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.19.254:43727: connect: Connection refused (error 111)
W20250626 01:57:49.539086 17139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 01:57:49.539425 17139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 01:57:49.539678 17139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 01:57:49.539923 17139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 01:57:49.575625 17139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:40729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.19.254:43727
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.19.254:43727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.19.254
--webserver_port=43057
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 01:57:49.577006 17139 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 01:57:49.578751 17139 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:57:49.594892 17147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:57:49.654587 16960 debug-util.cc:398] Leaking SignalData structure 0x7b08000ac0a0 after lost signal to thread 16836
W20250626 01:57:49.655476 16960 debug-util.cc:398] Leaking SignalData structure 0x7b08000accc0 after lost signal to thread 16963
W20250626 01:57:50.152493 16698 heartbeater.cc:646] Failed to heartbeat to 127.16.19.254:43727 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.19.254:43727: connect: Connection refused (error 111)
I20250626 01:57:50.638329 17156 raft_consensus.cc:491] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 564ffe4de0df4f54bfe8b725c5fac685)
I20250626 01:57:50.638764 17156 raft_consensus.cc:513] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" member_type: VOTER last_known_addr { host: "127.16.19.196" port: 36263 } } peers { permanent_uuid: "564ffe4de0df4f54bfe8b725c5fac685" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 44795 } } peers { permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 } }
I20250626 01:57:50.641040 17156 leader_election.cc:290] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6154bdb58394449bbca4ce76fb2356e (127.16.19.196:36263), 564ffe4de0df4f54bfe8b725c5fac685 (127.16.19.194:44795)
W20250626 01:57:50.658041 16588 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.19.194:44795: connect: Connection refused (error 111)
W20250626 01:57:50.668588 16588 leader_election.cc:336] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 564ffe4de0df4f54bfe8b725c5fac685 (127.16.19.194:44795): Network error: Client connection negotiation failed: client connection to 127.16.19.194:44795: connect: Connection refused (error 111)
I20250626 01:57:50.668982 17053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75c7fafa68214e0c82edbb7d3479774d" candidate_uuid: "8711c0db646c4f09930414c4e99ade2c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d6154bdb58394449bbca4ce76fb2356e" is_pre_election: true
I20250626 01:57:50.669627 17053 raft_consensus.cc:2466] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8711c0db646c4f09930414c4e99ade2c in term 1.
I20250626 01:57:50.671073 16586 leader_election.cc:304] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8711c0db646c4f09930414c4e99ade2c, d6154bdb58394449bbca4ce76fb2356e; no voters: 564ffe4de0df4f54bfe8b725c5fac685
I20250626 01:57:50.672005 17156 raft_consensus.cc:2802] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 01:57:50.672358 17156 raft_consensus.cc:491] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 564ffe4de0df4f54bfe8b725c5fac685)
I20250626 01:57:50.672719 17156 raft_consensus.cc:3058] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:57:50.680066 17156 raft_consensus.cc:513] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" member_type: VOTER last_known_addr { host: "127.16.19.196" port: 36263 } } peers { permanent_uuid: "564ffe4de0df4f54bfe8b725c5fac685" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 44795 } } peers { permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 } }
I20250626 01:57:50.683678 17053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75c7fafa68214e0c82edbb7d3479774d" candidate_uuid: "8711c0db646c4f09930414c4e99ade2c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d6154bdb58394449bbca4ce76fb2356e"
I20250626 01:57:50.684279 17053 raft_consensus.cc:3058] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:57:50.682150 17156 leader_election.cc:290] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [CANDIDATE]: Term 2 election: Requested vote from peers d6154bdb58394449bbca4ce76fb2356e (127.16.19.196:36263), 564ffe4de0df4f54bfe8b725c5fac685 (127.16.19.194:44795)
W20250626 01:57:50.687767 16588 leader_election.cc:336] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 564ffe4de0df4f54bfe8b725c5fac685 (127.16.19.194:44795): Network error: Client connection negotiation failed: client connection to 127.16.19.194:44795: connect: Connection refused (error 111)
I20250626 01:57:50.691951 17053 raft_consensus.cc:2466] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8711c0db646c4f09930414c4e99ade2c in term 2.
I20250626 01:57:50.694123 16586 leader_election.cc:304] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8711c0db646c4f09930414c4e99ade2c, d6154bdb58394449bbca4ce76fb2356e; no voters: 564ffe4de0df4f54bfe8b725c5fac685
I20250626 01:57:50.694967 17156 raft_consensus.cc:2802] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [term 2 FOLLOWER]: Leader election won for term 2
I20250626 01:57:50.697067 17156 raft_consensus.cc:695] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [term 2 LEADER]: Becoming Leader. State: Replica: 8711c0db646c4f09930414c4e99ade2c, State: Running, Role: LEADER
I20250626 01:57:50.698069 17156 consensus_queue.cc:237] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" member_type: VOTER last_known_addr { host: "127.16.19.196" port: 36263 } } peers { permanent_uuid: "564ffe4de0df4f54bfe8b725c5fac685" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 44795 } } peers { permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 } }
I20250626 01:57:49.597438 17139 server_base.cc:1048] running on GCE node
W20250626 01:57:49.595309 17150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:57:49.595212 17148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:57:50.838071 17139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 01:57:50.841665 17139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 01:57:50.843205 17139 hybrid_clock.cc:648] HybridClock initialized: now 1750903070843150 us; error 53 us; skew 500 ppm
I20250626 01:57:50.844426 17139 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:57:50.852671 17139 webserver.cc:469] Webserver started at http://127.16.19.254:43057/ using document root <none> and password file <none>
I20250626 01:57:50.853984 17139 fs_manager.cc:362] Metadata directory not provided
I20250626 01:57:50.854291 17139 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:57:50.866158 17139 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.004s sys 0.004s
I20250626 01:57:50.872390 17166 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:57:50.873557 17139 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250626 01:57:50.873881 17139 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
uuid: "904ef2bacbb14a33b522e0bf8c3b3c0d"
format_stamp: "Formatted at 2025-06-26 01:57:40 on dist-test-slave-h4rq"
I20250626 01:57:50.875792 17139 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:57:50.931067 17139 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:57:50.932545 17139 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 01:57:50.933044 17139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:57:51.021174 17139 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.254:43727
I20250626 01:57:51.021354 17217 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.254:43727 every 8 connection(s)
I20250626 01:57:51.024333 17139 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250626 01:57:51.034145 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 17139
I20250626 01:57:51.034817 16463 kudu-admin-test.cc:735] Forcing unsafe config change on tserver d6154bdb58394449bbca4ce76fb2356e
I20250626 01:57:51.035677 17218 sys_catalog.cc:263] Verifying existing consensus state
I20250626 01:57:51.040930 17218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d: Bootstrap starting.
I20250626 01:57:51.096004 17218 log.cc:826] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d: Log is configured to *not* fsync() on all Append() calls
W20250626 01:57:51.114246 16588 consensus_peers.cc:487] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c -> Peer 564ffe4de0df4f54bfe8b725c5fac685 (127.16.19.194:44795): Couldn't send request to peer 564ffe4de0df4f54bfe8b725c5fac685. Status: Network error: Client connection negotiation failed: client connection to 127.16.19.194:44795: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 01:57:51.127393 17218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:57:51.128436 17218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d: Bootstrap complete.
I20250626 01:57:51.162688 17218 raft_consensus.cc:357] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "904ef2bacbb14a33b522e0bf8c3b3c0d" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 43727 } }
I20250626 01:57:51.166544 17218 raft_consensus.cc:738] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 904ef2bacbb14a33b522e0bf8c3b3c0d, State: Initialized, Role: FOLLOWER
I20250626 01:57:51.167734 17218 consensus_queue.cc:260] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "904ef2bacbb14a33b522e0bf8c3b3c0d" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 43727 } }
I20250626 01:57:51.168541 17218 raft_consensus.cc:397] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 01:57:51.168952 17218 raft_consensus.cc:491] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 01:57:51.169389 17218 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:57:51.180356 17218 raft_consensus.cc:513] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "904ef2bacbb14a33b522e0bf8c3b3c0d" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 43727 } }
I20250626 01:57:51.181402 17218 leader_election.cc:304] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 904ef2bacbb14a33b522e0bf8c3b3c0d; no voters:
I20250626 01:57:51.183712 17218 leader_election.cc:290] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d [CANDIDATE]: Term 2 election: Requested vote from peers
I20250626 01:57:51.184082 17222 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d [term 2 FOLLOWER]: Leader election won for term 2
I20250626 01:57:51.190136 17222 raft_consensus.cc:695] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d [term 2 LEADER]: Becoming Leader. State: Replica: 904ef2bacbb14a33b522e0bf8c3b3c0d, State: Running, Role: LEADER
I20250626 01:57:51.191171 17222 consensus_queue.cc:237] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "904ef2bacbb14a33b522e0bf8c3b3c0d" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 43727 } }
I20250626 01:57:51.197710 17218 sys_catalog.cc:564] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d [sys.catalog]: configured and running, proceeding with master startup.
I20250626 01:57:51.212509 17224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "904ef2bacbb14a33b522e0bf8c3b3c0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "904ef2bacbb14a33b522e0bf8c3b3c0d" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 43727 } } }
I20250626 01:57:51.216347 17224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d [sys.catalog]: This master's current role is: LEADER
I20250626 01:57:51.213901 17223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 904ef2bacbb14a33b522e0bf8c3b3c0d. Latest consensus state: current_term: 2 leader_uuid: "904ef2bacbb14a33b522e0bf8c3b3c0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "904ef2bacbb14a33b522e0bf8c3b3c0d" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 43727 } } }
I20250626 01:57:51.218274 17223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d [sys.catalog]: This master's current role is: LEADER
I20250626 01:57:51.226471 17229 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 01:57:51.250517 17229 catalog_manager.cc:671] Loaded metadata for table TestTable [id=39738b9209294ab6a5f7102b84047fa9]
I20250626 01:57:51.259384 17229 tablet_loader.cc:96] loaded metadata for tablet 75c7fafa68214e0c82edbb7d3479774d (table TestTable [id=39738b9209294ab6a5f7102b84047fa9])
I20250626 01:57:51.261200 17229 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 01:57:51.266321 17229 catalog_manager.cc:1261] Loaded cluster ID: 26edc70c6efe4ccb950f893589f214e4
I20250626 01:57:51.266706 17229 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 01:57:51.275022 17229 catalog_manager.cc:1506] Loading token signing keys...
I20250626 01:57:51.280275 17229 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 904ef2bacbb14a33b522e0bf8c3b3c0d: Loaded TSK: 0
I20250626 01:57:51.282035 17229 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 01:57:51.293725 17053 raft_consensus.cc:1273] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e [term 2 FOLLOWER]: Refusing update from remote peer 8711c0db646c4f09930414c4e99ade2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 01:57:51.295957 17156 consensus_queue.cc:1035] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" member_type: VOTER last_known_addr { host: "127.16.19.196" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 01:57:51.383867 17098 heartbeater.cc:344] Connected to a master server at 127.16.19.254:43727
I20250626 01:57:51.391360 17183 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" instance_seqno: 1750903068243623) as {username='slave'} at 127.16.19.196:60641; Asking this server to re-register.
I20250626 01:57:51.393357 17098 heartbeater.cc:461] Registering TS with master...
I20250626 01:57:51.394057 17098 heartbeater.cc:507] Master 127.16.19.254:43727 requested a full tablet report, sending...
I20250626 01:57:51.397882 17182 ts_manager.cc:194] Registered new tserver with Master: d6154bdb58394449bbca4ce76fb2356e (127.16.19.196:36263)
I20250626 01:57:51.406790 17182 catalog_manager.cc:5582] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e reported cstate change: term changed from 1 to 2, leader changed from 564ffe4de0df4f54bfe8b725c5fac685 (127.16.19.194) to 8711c0db646c4f09930414c4e99ade2c (127.16.19.193). New cstate: current_term: 2 leader_uuid: "8711c0db646c4f09930414c4e99ade2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" member_type: VOTER last_known_addr { host: "127.16.19.196" port: 36263 } } peers { permanent_uuid: "564ffe4de0df4f54bfe8b725c5fac685" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 44795 } } peers { permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 } } }
I20250626 01:57:51.436061 16698 heartbeater.cc:344] Connected to a master server at 127.16.19.254:43727
I20250626 01:57:51.442765 17183 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" instance_seqno: 1750903062757664) as {username='slave'} at 127.16.19.193:55545; Asking this server to re-register.
I20250626 01:57:51.444257 16698 heartbeater.cc:461] Registering TS with master...
I20250626 01:57:51.445008 16698 heartbeater.cc:507] Master 127.16.19.254:43727 requested a full tablet report, sending...
I20250626 01:57:51.448242 17183 ts_manager.cc:194] Registered new tserver with Master: 8711c0db646c4f09930414c4e99ade2c (127.16.19.193:33947)
I20250626 01:57:51.529811 16964 heartbeater.cc:344] Connected to a master server at 127.16.19.254:43727
I20250626 01:57:51.531612 17182 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "95051b8c263847058d44f0e678adbee5" instance_seqno: 1750903066435384) as {username='slave'} at 127.16.19.195:54647; Asking this server to re-register.
I20250626 01:57:51.532694 16964 heartbeater.cc:461] Registering TS with master...
I20250626 01:57:51.533212 16964 heartbeater.cc:507] Master 127.16.19.254:43727 requested a full tablet report, sending...
I20250626 01:57:51.534873 17182 ts_manager.cc:194] Registered new tserver with Master: 95051b8c263847058d44f0e678adbee5 (127.16.19.195:37143)
W20250626 01:57:51.593837 17220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:57:51.594473 17220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:57:51.626109 17220 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250626 01:57:53.074247 17254 debug-util.cc:398] Leaking SignalData structure 0x7b0800036040 after lost signal to thread 17220
W20250626 01:57:53.074796 17254 kernel_stack_watchdog.cc:198] Thread 17220 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:641 for 402ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250626 01:57:53.215018 17220 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.545s user 0.572s sys 0.959s
W20250626 01:57:53.346094 17220 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.676s user 0.577s sys 0.968s
I20250626 01:57:53.398434 17053 tablet_service.cc:1905] Received UnsafeChangeConfig RPC: dest_uuid: "d6154bdb58394449bbca4ce76fb2356e"
tablet_id: "75c7fafa68214e0c82edbb7d3479774d"
caller_id: "kudu-tools"
new_config {
peers {
permanent_uuid: "8711c0db646c4f09930414c4e99ade2c"
}
peers {
permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e"
}
}
from {username='slave'} at 127.0.0.1:45124
W20250626 01:57:53.399542 17053 raft_consensus.cc:2216] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e [term 2 FOLLOWER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" member_type: VOTER last_known_addr { host: "127.16.19.196" port: 36263 } } peers { permanent_uuid: "564ffe4de0df4f54bfe8b725c5fac685" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 44795 } } peers { permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 } }NEW CONFIG: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" member_type: VOTER last_known_addr { host: "127.16.19.196" port: 36263 } } peers { permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 } } unsafe_config_change: true
I20250626 01:57:53.400344 17053 raft_consensus.cc:3058] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e [term 2 FOLLOWER]: Advancing to term 3
I20250626 01:57:53.520107 17053 raft_consensus.cc:1238] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e [term 3 FOLLOWER]: Rejecting Update request from peer 8711c0db646c4f09930414c4e99ade2c for earlier term 2. Current term is 3. Ops: []
I20250626 01:57:53.521416 17246 consensus_queue.cc:1046] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" member_type: VOTER last_known_addr { host: "127.16.19.196" port: 36263 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 01:57:53.522951 17273 raft_consensus.cc:3053] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [term 2 LEADER]: Stepping down as leader of term 2
I20250626 01:57:53.523240 17273 raft_consensus.cc:738] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8711c0db646c4f09930414c4e99ade2c, State: Running, Role: LEADER
I20250626 01:57:53.523936 17273 consensus_queue.cc:260] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" member_type: VOTER last_known_addr { host: "127.16.19.196" port: 36263 } } peers { permanent_uuid: "564ffe4de0df4f54bfe8b725c5fac685" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 44795 } } peers { permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 } }
I20250626 01:57:53.524952 17273 raft_consensus.cc:3058] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [term 2 FOLLOWER]: Advancing to term 3
I20250626 01:57:54.851455 17276 raft_consensus.cc:491] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:57:54.851851 17276 raft_consensus.cc:513] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" member_type: VOTER last_known_addr { host: "127.16.19.196" port: 36263 } } peers { permanent_uuid: "564ffe4de0df4f54bfe8b725c5fac685" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 44795 } } peers { permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 } }
I20250626 01:57:54.853216 17276 leader_election.cc:290] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d6154bdb58394449bbca4ce76fb2356e (127.16.19.196:36263), 564ffe4de0df4f54bfe8b725c5fac685 (127.16.19.194:44795)
I20250626 01:57:54.854120 17053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75c7fafa68214e0c82edbb7d3479774d" candidate_uuid: "8711c0db646c4f09930414c4e99ade2c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d6154bdb58394449bbca4ce76fb2356e" is_pre_election: true
W20250626 01:57:54.858299 16588 leader_election.cc:336] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 564ffe4de0df4f54bfe8b725c5fac685 (127.16.19.194:44795): Network error: Client connection negotiation failed: client connection to 127.16.19.194:44795: connect: Connection refused (error 111)
I20250626 01:57:54.858611 16588 leader_election.cc:304] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8711c0db646c4f09930414c4e99ade2c; no voters: 564ffe4de0df4f54bfe8b725c5fac685, d6154bdb58394449bbca4ce76fb2356e
I20250626 01:57:54.859202 17276 raft_consensus.cc:2747] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 01:57:54.907907 17279 raft_consensus.cc:491] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools)
I20250626 01:57:54.908365 17279 raft_consensus.cc:513] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" member_type: VOTER last_known_addr { host: "127.16.19.196" port: 36263 } } peers { permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 } } unsafe_config_change: true
I20250626 01:57:54.910235 17279 leader_election.cc:290] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8711c0db646c4f09930414c4e99ade2c (127.16.19.193:33947)
I20250626 01:57:54.919157 16653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75c7fafa68214e0c82edbb7d3479774d" candidate_uuid: "d6154bdb58394449bbca4ce76fb2356e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8711c0db646c4f09930414c4e99ade2c" is_pre_election: true
I20250626 01:57:54.919768 16653 raft_consensus.cc:2466] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6154bdb58394449bbca4ce76fb2356e in term 3.
I20250626 01:57:54.920898 16988 leader_election.cc:304] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8711c0db646c4f09930414c4e99ade2c, d6154bdb58394449bbca4ce76fb2356e; no voters:
I20250626 01:57:54.921536 17279 raft_consensus.cc:2802] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 01:57:54.921793 17279 raft_consensus.cc:491] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e [term 3 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools)
I20250626 01:57:54.922061 17279 raft_consensus.cc:3058] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e [term 3 FOLLOWER]: Advancing to term 4
I20250626 01:57:54.926427 17279 raft_consensus.cc:513] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" member_type: VOTER last_known_addr { host: "127.16.19.196" port: 36263 } } peers { permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 } } unsafe_config_change: true
I20250626 01:57:54.927457 17279 leader_election.cc:290] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e [CANDIDATE]: Term 4 election: Requested vote from peers 8711c0db646c4f09930414c4e99ade2c (127.16.19.193:33947)
I20250626 01:57:54.928436 16653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75c7fafa68214e0c82edbb7d3479774d" candidate_uuid: "d6154bdb58394449bbca4ce76fb2356e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8711c0db646c4f09930414c4e99ade2c"
I20250626 01:57:54.928884 16653 raft_consensus.cc:3058] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [term 3 FOLLOWER]: Advancing to term 4
I20250626 01:57:54.932915 16653 raft_consensus.cc:2466] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6154bdb58394449bbca4ce76fb2356e in term 4.
I20250626 01:57:54.933781 16988 leader_election.cc:304] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8711c0db646c4f09930414c4e99ade2c, d6154bdb58394449bbca4ce76fb2356e; no voters:
I20250626 01:57:54.934417 17279 raft_consensus.cc:2802] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e [term 4 FOLLOWER]: Leader election won for term 4
I20250626 01:57:54.935200 17279 raft_consensus.cc:695] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e [term 4 LEADER]: Becoming Leader. State: Replica: d6154bdb58394449bbca4ce76fb2356e, State: Running, Role: LEADER
I20250626 01:57:54.935963 17279 consensus_queue.cc:237] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" member_type: VOTER last_known_addr { host: "127.16.19.196" port: 36263 } } peers { permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 } } unsafe_config_change: true
I20250626 01:57:54.942476 17182 catalog_manager.cc:5582] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e reported cstate change: term changed from 2 to 4, leader changed from 8711c0db646c4f09930414c4e99ade2c (127.16.19.193) to d6154bdb58394449bbca4ce76fb2356e (127.16.19.196), now has a pending config: VOTER d6154bdb58394449bbca4ce76fb2356e (127.16.19.196), VOTER 8711c0db646c4f09930414c4e99ade2c (127.16.19.193). New cstate: current_term: 4 leader_uuid: "d6154bdb58394449bbca4ce76fb2356e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" member_type: VOTER last_known_addr { host: "127.16.19.196" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "564ffe4de0df4f54bfe8b725c5fac685" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 44795 } } peers { permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" member_type: VOTER last_known_addr { host: "127.16.19.196" port: 36263 } } peers { permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 } } unsafe_config_change: true }
I20250626 01:57:55.483348 16653 raft_consensus.cc:1273] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [term 4 FOLLOWER]: Refusing update from remote peer d6154bdb58394449bbca4ce76fb2356e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250626 01:57:55.484663 17289 consensus_queue.cc:1035] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 01:57:55.492705 17292 raft_consensus.cc:2953] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e [term 4 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER 564ffe4de0df4f54bfe8b725c5fac685 (127.16.19.194) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" member_type: VOTER last_known_addr { host: "127.16.19.196" port: 36263 } } peers { permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 } } unsafe_config_change: true }
I20250626 01:57:55.493757 16653 raft_consensus.cc:2953] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [term 4 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER 564ffe4de0df4f54bfe8b725c5fac685 (127.16.19.194) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" member_type: VOTER last_known_addr { host: "127.16.19.196" port: 36263 } } peers { permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 } } unsafe_config_change: true }
I20250626 01:57:55.502907 17182 catalog_manager.cc:5582] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e reported cstate change: config changed from index -1 to 3, VOTER 564ffe4de0df4f54bfe8b725c5fac685 (127.16.19.194) evicted, no longer has a pending config: VOTER d6154bdb58394449bbca4ce76fb2356e (127.16.19.196), VOTER 8711c0db646c4f09930414c4e99ade2c (127.16.19.193). New cstate: current_term: 4 leader_uuid: "d6154bdb58394449bbca4ce76fb2356e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" member_type: VOTER last_known_addr { host: "127.16.19.196" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
W20250626 01:57:55.513139 17182 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet 75c7fafa68214e0c82edbb7d3479774d on TS 564ffe4de0df4f54bfe8b725c5fac685: Not found: failed to reset TS proxy: Could not find TS for UUID 564ffe4de0df4f54bfe8b725c5fac685
I20250626 01:57:55.529842 17053 consensus_queue.cc:237] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" member_type: VOTER last_known_addr { host: "127.16.19.196" port: 36263 } } peers { permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 } } peers { permanent_uuid: "95051b8c263847058d44f0e678adbee5" member_type: NON_VOTER last_known_addr { host: "127.16.19.195" port: 37143 } attrs { promote: true } } unsafe_config_change: true
I20250626 01:57:55.534548 16653 raft_consensus.cc:1273] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [term 4 FOLLOWER]: Refusing update from remote peer d6154bdb58394449bbca4ce76fb2356e: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250626 01:57:55.536084 17292 consensus_queue.cc:1035] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 01:57:55.542471 17289 raft_consensus.cc:2953] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER 95051b8c263847058d44f0e678adbee5 (127.16.19.195) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" member_type: VOTER last_known_addr { host: "127.16.19.196" port: 36263 } } peers { permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 } } peers { permanent_uuid: "95051b8c263847058d44f0e678adbee5" member_type: NON_VOTER last_known_addr { host: "127.16.19.195" port: 37143 } attrs { promote: true } } unsafe_config_change: true }
I20250626 01:57:55.544261 16653 raft_consensus.cc:2953] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER 95051b8c263847058d44f0e678adbee5 (127.16.19.195) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" member_type: VOTER last_known_addr { host: "127.16.19.196" port: 36263 } } peers { permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 } } peers { permanent_uuid: "95051b8c263847058d44f0e678adbee5" member_type: NON_VOTER last_known_addr { host: "127.16.19.195" port: 37143 } attrs { promote: true } } unsafe_config_change: true }
W20250626 01:57:55.548240 16989 consensus_peers.cc:487] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e -> Peer 95051b8c263847058d44f0e678adbee5 (127.16.19.195:37143): Couldn't send request to peer 95051b8c263847058d44f0e678adbee5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 75c7fafa68214e0c82edbb7d3479774d. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:57:55.550252 17167 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 75c7fafa68214e0c82edbb7d3479774d with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250626 01:57:55.553241 17183 catalog_manager.cc:5582] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e reported cstate change: config changed from index 3 to 5, NON_VOTER 95051b8c263847058d44f0e678adbee5 (127.16.19.195) added. New cstate: current_term: 4 leader_uuid: "d6154bdb58394449bbca4ce76fb2356e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" member_type: VOTER last_known_addr { host: "127.16.19.196" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95051b8c263847058d44f0e678adbee5" member_type: NON_VOTER last_known_addr { host: "127.16.19.195" port: 37143 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true }
W20250626 01:57:55.565584 17168 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 75c7fafa68214e0c82edbb7d3479774d on TS 564ffe4de0df4f54bfe8b725c5fac685 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 564ffe4de0df4f54bfe8b725c5fac685
I20250626 01:57:56.056663 17302 ts_tablet_manager.cc:927] T 75c7fafa68214e0c82edbb7d3479774d P 95051b8c263847058d44f0e678adbee5: Initiating tablet copy from peer d6154bdb58394449bbca4ce76fb2356e (127.16.19.196:36263)
I20250626 01:57:56.059837 17302 tablet_copy_client.cc:323] T 75c7fafa68214e0c82edbb7d3479774d P 95051b8c263847058d44f0e678adbee5: tablet copy: Beginning tablet copy session from remote peer at address 127.16.19.196:36263
I20250626 01:57:56.070106 17073 tablet_copy_service.cc:140] P d6154bdb58394449bbca4ce76fb2356e: Received BeginTabletCopySession request for tablet 75c7fafa68214e0c82edbb7d3479774d from peer 95051b8c263847058d44f0e678adbee5 ({username='slave'} at 127.16.19.195:54175)
I20250626 01:57:56.070501 17073 tablet_copy_service.cc:161] P d6154bdb58394449bbca4ce76fb2356e: Beginning new tablet copy session on tablet 75c7fafa68214e0c82edbb7d3479774d from peer 95051b8c263847058d44f0e678adbee5 at {username='slave'} at 127.16.19.195:54175: session id = 95051b8c263847058d44f0e678adbee5-75c7fafa68214e0c82edbb7d3479774d
I20250626 01:57:56.075187 17073 tablet_copy_source_session.cc:215] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 01:57:56.079949 17302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75c7fafa68214e0c82edbb7d3479774d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:57:56.098477 17302 tablet_copy_client.cc:806] T 75c7fafa68214e0c82edbb7d3479774d P 95051b8c263847058d44f0e678adbee5: tablet copy: Starting download of 0 data blocks...
I20250626 01:57:56.099033 17302 tablet_copy_client.cc:670] T 75c7fafa68214e0c82edbb7d3479774d P 95051b8c263847058d44f0e678adbee5: tablet copy: Starting download of 1 WAL segments...
I20250626 01:57:56.102619 17302 tablet_copy_client.cc:538] T 75c7fafa68214e0c82edbb7d3479774d P 95051b8c263847058d44f0e678adbee5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 01:57:56.108566 17302 tablet_bootstrap.cc:492] T 75c7fafa68214e0c82edbb7d3479774d P 95051b8c263847058d44f0e678adbee5: Bootstrap starting.
I20250626 01:57:56.120851 17302 log.cc:826] T 75c7fafa68214e0c82edbb7d3479774d P 95051b8c263847058d44f0e678adbee5: Log is configured to *not* fsync() on all Append() calls
I20250626 01:57:56.131497 17302 tablet_bootstrap.cc:492] T 75c7fafa68214e0c82edbb7d3479774d P 95051b8c263847058d44f0e678adbee5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:57:56.132233 17302 tablet_bootstrap.cc:492] T 75c7fafa68214e0c82edbb7d3479774d P 95051b8c263847058d44f0e678adbee5: Bootstrap complete.
I20250626 01:57:56.132864 17302 ts_tablet_manager.cc:1397] T 75c7fafa68214e0c82edbb7d3479774d P 95051b8c263847058d44f0e678adbee5: Time spent bootstrapping tablet: real 0.025s user 0.024s sys 0.003s
I20250626 01:57:56.150673 17302 raft_consensus.cc:357] T 75c7fafa68214e0c82edbb7d3479774d P 95051b8c263847058d44f0e678adbee5 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" member_type: VOTER last_known_addr { host: "127.16.19.196" port: 36263 } } peers { permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 } } peers { permanent_uuid: "95051b8c263847058d44f0e678adbee5" member_type: NON_VOTER last_known_addr { host: "127.16.19.195" port: 37143 } attrs { promote: true } } unsafe_config_change: true
I20250626 01:57:56.151589 17302 raft_consensus.cc:738] T 75c7fafa68214e0c82edbb7d3479774d P 95051b8c263847058d44f0e678adbee5 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 95051b8c263847058d44f0e678adbee5, State: Initialized, Role: LEARNER
I20250626 01:57:56.152402 17302 consensus_queue.cc:260] T 75c7fafa68214e0c82edbb7d3479774d P 95051b8c263847058d44f0e678adbee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" member_type: VOTER last_known_addr { host: "127.16.19.196" port: 36263 } } peers { permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 } } peers { permanent_uuid: "95051b8c263847058d44f0e678adbee5" member_type: NON_VOTER last_known_addr { host: "127.16.19.195" port: 37143 } attrs { promote: true } } unsafe_config_change: true
I20250626 01:57:56.155611 17302 ts_tablet_manager.cc:1428] T 75c7fafa68214e0c82edbb7d3479774d P 95051b8c263847058d44f0e678adbee5: Time spent starting tablet: real 0.022s user 0.017s sys 0.004s
I20250626 01:57:56.157287 17073 tablet_copy_service.cc:342] P d6154bdb58394449bbca4ce76fb2356e: Request end of tablet copy session 95051b8c263847058d44f0e678adbee5-75c7fafa68214e0c82edbb7d3479774d received from {username='slave'} at 127.16.19.195:54175
I20250626 01:57:56.157723 17073 tablet_copy_service.cc:434] P d6154bdb58394449bbca4ce76fb2356e: ending tablet copy session 95051b8c263847058d44f0e678adbee5-75c7fafa68214e0c82edbb7d3479774d on tablet 75c7fafa68214e0c82edbb7d3479774d with peer 95051b8c263847058d44f0e678adbee5
I20250626 01:57:56.538583 16919 raft_consensus.cc:1215] T 75c7fafa68214e0c82edbb7d3479774d P 95051b8c263847058d44f0e678adbee5 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.4->[4.5-4.5] Dedup: 4.5->[]
W20250626 01:57:56.731992 17168 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 75c7fafa68214e0c82edbb7d3479774d on TS 564ffe4de0df4f54bfe8b725c5fac685 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 564ffe4de0df4f54bfe8b725c5fac685
I20250626 01:57:56.943949 17309 raft_consensus.cc:1062] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e: attempting to promote NON_VOTER 95051b8c263847058d44f0e678adbee5 to VOTER
I20250626 01:57:56.945875 17309 consensus_queue.cc:237] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" member_type: VOTER last_known_addr { host: "127.16.19.196" port: 36263 } } peers { permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 } } peers { permanent_uuid: "95051b8c263847058d44f0e678adbee5" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 37143 } attrs { promote: false } } unsafe_config_change: true
I20250626 01:57:56.950809 16919 raft_consensus.cc:1273] T 75c7fafa68214e0c82edbb7d3479774d P 95051b8c263847058d44f0e678adbee5 [term 4 LEARNER]: Refusing update from remote peer d6154bdb58394449bbca4ce76fb2356e: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250626 01:57:56.950950 16653 raft_consensus.cc:1273] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [term 4 FOLLOWER]: Refusing update from remote peer d6154bdb58394449bbca4ce76fb2356e: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250626 01:57:56.952091 17292 consensus_queue.cc:1035] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e [LEADER]: Connected to new peer: Peer: permanent_uuid: "95051b8c263847058d44f0e678adbee5" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 37143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250626 01:57:56.952878 17308 consensus_queue.cc:1035] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250626 01:57:56.959172 17292 raft_consensus.cc:2953] T 75c7fafa68214e0c82edbb7d3479774d P d6154bdb58394449bbca4ce76fb2356e [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 95051b8c263847058d44f0e678adbee5 (127.16.19.195) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" member_type: VOTER last_known_addr { host: "127.16.19.196" port: 36263 } } peers { permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 } } peers { permanent_uuid: "95051b8c263847058d44f0e678adbee5" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 37143 } attrs { promote: false } } unsafe_config_change: true }
I20250626 01:57:56.960778 16653 raft_consensus.cc:2953] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 95051b8c263847058d44f0e678adbee5 (127.16.19.195) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" member_type: VOTER last_known_addr { host: "127.16.19.196" port: 36263 } } peers { permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 } } peers { permanent_uuid: "95051b8c263847058d44f0e678adbee5" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 37143 } attrs { promote: false } } unsafe_config_change: true }
I20250626 01:57:56.961964 16919 raft_consensus.cc:2953] T 75c7fafa68214e0c82edbb7d3479774d P 95051b8c263847058d44f0e678adbee5 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 95051b8c263847058d44f0e678adbee5 (127.16.19.195) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" member_type: VOTER last_known_addr { host: "127.16.19.196" port: 36263 } } peers { permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 } } peers { permanent_uuid: "95051b8c263847058d44f0e678adbee5" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 37143 } attrs { promote: false } } unsafe_config_change: true }
I20250626 01:57:56.971067 17183 catalog_manager.cc:5582] T 75c7fafa68214e0c82edbb7d3479774d P 8711c0db646c4f09930414c4e99ade2c reported cstate change: config changed from index 5 to 6, 95051b8c263847058d44f0e678adbee5 (127.16.19.195) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "d6154bdb58394449bbca4ce76fb2356e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e" member_type: VOTER last_known_addr { host: "127.16.19.196" port: 36263 } } peers { permanent_uuid: "8711c0db646c4f09930414c4e99ade2c" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 33947 } } peers { permanent_uuid: "95051b8c263847058d44f0e678adbee5" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 37143 } attrs { promote: false } } unsafe_config_change: true }
I20250626 01:57:57.072834 16463 kudu-admin-test.cc:751] Waiting for Master to see new config...
I20250626 01:57:57.087013 16463 kudu-admin-test.cc:756] Tablet locations:
tablet_locations {
tablet_id: "75c7fafa68214e0c82edbb7d3479774d"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: LEADER
}
interned_replicas {
ts_info_idx: 1
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "d6154bdb58394449bbca4ce76fb2356e"
rpc_addresses {
host: "127.16.19.196"
port: 36263
}
}
ts_infos {
permanent_uuid: "8711c0db646c4f09930414c4e99ade2c"
rpc_addresses {
host: "127.16.19.193"
port: 33947
}
}
ts_infos {
permanent_uuid: "95051b8c263847058d44f0e678adbee5"
rpc_addresses {
host: "127.16.19.195"
port: 37143
}
}
I20250626 01:57:57.089165 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 16569
I20250626 01:57:57.113933 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 16835
I20250626 01:57:57.142176 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 16968
I20250626 01:57:57.167311 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 17139
2025-06-26T01:57:57Z chronyd exiting
[ OK ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes (19448 ms)
[ RUN ] AdminCliTest.TestGracefulSpecificLeaderStepDown
I20250626 01:57:57.225035 16463 test_util.cc:276] Using random seed: -470531535
I20250626 01:57:57.230695 16463 ts_itest-base.cc:115] Starting cluster with:
I20250626 01:57:57.230877 16463 ts_itest-base.cc:116] --------------
I20250626 01:57:57.231055 16463 ts_itest-base.cc:117] 3 tablet servers
I20250626 01:57:57.231200 16463 ts_itest-base.cc:118] 3 replicas per TS
I20250626 01:57:57.231343 16463 ts_itest-base.cc:119] --------------
2025-06-26T01:57:57Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-26T01:57:57Z Disabled control of system clock
I20250626 01:57:57.267246 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.19.254:45487
--webserver_interface=127.16.19.254
--webserver_port=0
--builtin_ntp_servers=127.16.19.212:46813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.19.254:45487
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250626 01:57:57.564237 17328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:57:57.564895 17328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 01:57:57.565378 17328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:57:57.597774 17328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 01:57:57.598124 17328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 01:57:57.598378 17328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 01:57:57.598613 17328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 01:57:57.633901 17328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:46813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.16.19.254:45487
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.19.254:45487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.19.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 01:57:57.635362 17328 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 01:57:57.637061 17328 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:57:57.651010 17335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:57:57.653268 17337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:57:57.651705 17334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:57:58.830380 17336 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 01:57:58.830478 17328 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 01:57:58.834374 17328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 01:57:58.837055 17328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 01:57:58.838445 17328 hybrid_clock.cc:648] HybridClock initialized: now 1750903078838398 us; error 56 us; skew 500 ppm
I20250626 01:57:58.839294 17328 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:57:58.848882 17328 webserver.cc:469] Webserver started at http://127.16.19.254:44735/ using document root <none> and password file <none>
I20250626 01:57:58.849845 17328 fs_manager.cc:362] Metadata directory not provided
I20250626 01:57:58.850075 17328 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:57:58.850528 17328 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:57:58.854983 17328 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "3ba7f26e71fd42e3954dc2cc6960476e"
format_stamp: "Formatted at 2025-06-26 01:57:58 on dist-test-slave-h4rq"
I20250626 01:57:58.856078 17328 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "3ba7f26e71fd42e3954dc2cc6960476e"
format_stamp: "Formatted at 2025-06-26 01:57:58 on dist-test-slave-h4rq"
I20250626 01:57:58.863224 17328 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20250626 01:57:58.868968 17345 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:57:58.870064 17328 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250626 01:57:58.870383 17328 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
uuid: "3ba7f26e71fd42e3954dc2cc6960476e"
format_stamp: "Formatted at 2025-06-26 01:57:58 on dist-test-slave-h4rq"
I20250626 01:57:58.870720 17328 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:57:58.918526 17328 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:57:58.920032 17328 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 01:57:58.920464 17328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:57:58.992901 17328 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.254:45487
I20250626 01:57:58.992982 17396 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.254:45487 every 8 connection(s)
I20250626 01:57:58.995693 17328 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250626 01:57:59.000490 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 17328
I20250626 01:57:59.001102 17397 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:57:59.001163 16463 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250626 01:57:59.025982 17397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ba7f26e71fd42e3954dc2cc6960476e: Bootstrap starting.
I20250626 01:57:59.031627 17397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ba7f26e71fd42e3954dc2cc6960476e: Neither blocks nor log segments found. Creating new log.
I20250626 01:57:59.033373 17397 log.cc:826] T 00000000000000000000000000000000 P 3ba7f26e71fd42e3954dc2cc6960476e: Log is configured to *not* fsync() on all Append() calls
I20250626 01:57:59.037904 17397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ba7f26e71fd42e3954dc2cc6960476e: No bootstrap required, opened a new log
I20250626 01:57:59.055743 17397 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3ba7f26e71fd42e3954dc2cc6960476e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba7f26e71fd42e3954dc2cc6960476e" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 45487 } }
I20250626 01:57:59.056424 17397 raft_consensus.cc:383] T 00000000000000000000000000000000 P 3ba7f26e71fd42e3954dc2cc6960476e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:57:59.056706 17397 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3ba7f26e71fd42e3954dc2cc6960476e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ba7f26e71fd42e3954dc2cc6960476e, State: Initialized, Role: FOLLOWER
I20250626 01:57:59.057529 17397 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ba7f26e71fd42e3954dc2cc6960476e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba7f26e71fd42e3954dc2cc6960476e" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 45487 } }
I20250626 01:57:59.058145 17397 raft_consensus.cc:397] T 00000000000000000000000000000000 P 3ba7f26e71fd42e3954dc2cc6960476e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 01:57:59.058483 17397 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3ba7f26e71fd42e3954dc2cc6960476e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 01:57:59.058800 17397 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 3ba7f26e71fd42e3954dc2cc6960476e [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:57:59.062875 17397 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3ba7f26e71fd42e3954dc2cc6960476e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba7f26e71fd42e3954dc2cc6960476e" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 45487 } }
I20250626 01:57:59.063632 17397 leader_election.cc:304] T 00000000000000000000000000000000 P 3ba7f26e71fd42e3954dc2cc6960476e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ba7f26e71fd42e3954dc2cc6960476e; no voters:
I20250626 01:57:59.065608 17397 leader_election.cc:290] T 00000000000000000000000000000000 P 3ba7f26e71fd42e3954dc2cc6960476e [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 01:57:59.066308 17402 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 3ba7f26e71fd42e3954dc2cc6960476e [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:57:59.069149 17402 raft_consensus.cc:695] T 00000000000000000000000000000000 P 3ba7f26e71fd42e3954dc2cc6960476e [term 1 LEADER]: Becoming Leader. State: Replica: 3ba7f26e71fd42e3954dc2cc6960476e, State: Running, Role: LEADER
I20250626 01:57:59.070118 17402 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ba7f26e71fd42e3954dc2cc6960476e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba7f26e71fd42e3954dc2cc6960476e" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 45487 } }
I20250626 01:57:59.070673 17397 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ba7f26e71fd42e3954dc2cc6960476e [sys.catalog]: configured and running, proceeding with master startup.
I20250626 01:57:59.083038 17404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ba7f26e71fd42e3954dc2cc6960476e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ba7f26e71fd42e3954dc2cc6960476e. Latest consensus state: current_term: 1 leader_uuid: "3ba7f26e71fd42e3954dc2cc6960476e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba7f26e71fd42e3954dc2cc6960476e" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 45487 } } }
I20250626 01:57:59.083657 17404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ba7f26e71fd42e3954dc2cc6960476e [sys.catalog]: This master's current role is: LEADER
I20250626 01:57:59.085608 17403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ba7f26e71fd42e3954dc2cc6960476e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ba7f26e71fd42e3954dc2cc6960476e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba7f26e71fd42e3954dc2cc6960476e" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 45487 } } }
I20250626 01:57:59.086277 17403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ba7f26e71fd42e3954dc2cc6960476e [sys.catalog]: This master's current role is: LEADER
I20250626 01:57:59.086553 17412 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 01:57:59.099653 17412 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 01:57:59.114913 17412 catalog_manager.cc:1349] Generated new cluster ID: b272e6d4ce3e4cbeb200017b1ad0ba25
I20250626 01:57:59.115234 17412 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 01:57:59.144379 17412 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 01:57:59.145967 17412 catalog_manager.cc:1506] Loading token signing keys...
I20250626 01:57:59.157742 17412 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 3ba7f26e71fd42e3954dc2cc6960476e: Generated new TSK 0
I20250626 01:57:59.158645 17412 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 01:57:59.180677 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.19.193:0
--local_ip_for_outbound_sockets=127.16.19.193
--webserver_interface=127.16.19.193
--webserver_port=0
--tserver_master_addrs=127.16.19.254:45487
--builtin_ntp_servers=127.16.19.212:46813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250626 01:57:59.485353 17421 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250626 01:57:59.486018 17421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:57:59.486275 17421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 01:57:59.486785 17421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:57:59.518317 17421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 01:57:59.519253 17421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.19.193
I20250626 01:57:59.554663 17421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:46813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.19.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.19.193
--webserver_port=0
--tserver_master_addrs=127.16.19.254:45487
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.19.193
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 01:57:59.556043 17421 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 01:57:59.557739 17421 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:57:59.575137 17428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:57:59.575445 17427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:57:59.577507 17430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:58:00.756285 17429 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 01:58:00.756819 17421 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 01:58:00.761648 17421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 01:58:00.764545 17421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 01:58:00.766058 17421 hybrid_clock.cc:648] HybridClock initialized: now 1750903080765990 us; error 83 us; skew 500 ppm
I20250626 01:58:00.767161 17421 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:58:00.775069 17421 webserver.cc:469] Webserver started at http://127.16.19.193:46207/ using document root <none> and password file <none>
I20250626 01:58:00.776352 17421 fs_manager.cc:362] Metadata directory not provided
I20250626 01:58:00.776657 17421 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:58:00.777245 17421 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:58:00.783859 17421 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "60170cc2b9514176b92fe5ea741b92cc"
format_stamp: "Formatted at 2025-06-26 01:58:00 on dist-test-slave-h4rq"
I20250626 01:58:00.785626 17421 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "60170cc2b9514176b92fe5ea741b92cc"
format_stamp: "Formatted at 2025-06-26 01:58:00 on dist-test-slave-h4rq"
I20250626 01:58:00.795497 17421 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.005s sys 0.004s
I20250626 01:58:00.802811 17437 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:58:00.804076 17421 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20250626 01:58:00.804455 17421 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "60170cc2b9514176b92fe5ea741b92cc"
format_stamp: "Formatted at 2025-06-26 01:58:00 on dist-test-slave-h4rq"
I20250626 01:58:00.804955 17421 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:58:00.871716 17421 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:58:00.873481 17421 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 01:58:00.873927 17421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:58:00.876540 17421 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:58:00.880861 17421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:58:00.881072 17421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:58:00.881316 17421 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:58:00.881475 17421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:58:01.021507 17421 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.193:37005
I20250626 01:58:01.021618 17549 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.193:37005 every 8 connection(s)
I20250626 01:58:01.024050 17421 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250626 01:58:01.033248 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 17421
I20250626 01:58:01.033622 16463 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250626 01:58:01.040233 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.19.194:0
--local_ip_for_outbound_sockets=127.16.19.194
--webserver_interface=127.16.19.194
--webserver_port=0
--tserver_master_addrs=127.16.19.254:45487
--builtin_ntp_servers=127.16.19.212:46813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250626 01:58:01.045343 17550 heartbeater.cc:344] Connected to a master server at 127.16.19.254:45487
I20250626 01:58:01.045913 17550 heartbeater.cc:461] Registering TS with master...
I20250626 01:58:01.047267 17550 heartbeater.cc:507] Master 127.16.19.254:45487 requested a full tablet report, sending...
I20250626 01:58:01.049929 17362 ts_manager.cc:194] Registered new tserver with Master: 60170cc2b9514176b92fe5ea741b92cc (127.16.19.193:37005)
I20250626 01:58:01.051921 17362 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.19.193:43653
W20250626 01:58:01.342097 17554 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250626 01:58:01.342780 17554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:58:01.343051 17554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 01:58:01.343540 17554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:58:01.374836 17554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 01:58:01.375705 17554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.19.194
I20250626 01:58:01.412009 17554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:46813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.19.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.16.19.194
--webserver_port=0
--tserver_master_addrs=127.16.19.254:45487
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.19.194
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 01:58:01.413427 17554 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 01:58:01.415256 17554 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:58:01.430966 17561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:58:02.055992 17550 heartbeater.cc:499] Master 127.16.19.254:45487 was elected leader, sending a full tablet report...
W20250626 01:58:01.430966 17560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:58:02.596529 17562 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250626 01:58:02.598245 17563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:58:02.598309 17554 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 01:58:02.600893 17554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 01:58:02.603593 17554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 01:58:02.605186 17554 hybrid_clock.cc:648] HybridClock initialized: now 1750903082605113 us; error 87 us; skew 500 ppm
I20250626 01:58:02.606009 17554 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:58:02.612695 17554 webserver.cc:469] Webserver started at http://127.16.19.194:46433/ using document root <none> and password file <none>
I20250626 01:58:02.613623 17554 fs_manager.cc:362] Metadata directory not provided
I20250626 01:58:02.613835 17554 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:58:02.614291 17554 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:58:02.618752 17554 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "b7044b05a5cb4b96b8f98a35837832f1"
format_stamp: "Formatted at 2025-06-26 01:58:02 on dist-test-slave-h4rq"
I20250626 01:58:02.619850 17554 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "b7044b05a5cb4b96b8f98a35837832f1"
format_stamp: "Formatted at 2025-06-26 01:58:02 on dist-test-slave-h4rq"
I20250626 01:58:02.626982 17554 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20250626 01:58:02.632550 17570 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:58:02.633675 17554 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20250626 01:58:02.634002 17554 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "b7044b05a5cb4b96b8f98a35837832f1"
format_stamp: "Formatted at 2025-06-26 01:58:02 on dist-test-slave-h4rq"
I20250626 01:58:02.634320 17554 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:58:02.684552 17554 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:58:02.686053 17554 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 01:58:02.686503 17554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:58:02.689064 17554 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:58:02.693053 17554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:58:02.693236 17554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:58:02.693519 17554 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:58:02.693662 17554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:58:02.828728 17554 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.194:39305
I20250626 01:58:02.828826 17682 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.194:39305 every 8 connection(s)
I20250626 01:58:02.831264 17554 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250626 01:58:02.837980 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 17554
I20250626 01:58:02.838501 16463 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250626 01:58:02.844877 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.19.195:0
--local_ip_for_outbound_sockets=127.16.19.195
--webserver_interface=127.16.19.195
--webserver_port=0
--tserver_master_addrs=127.16.19.254:45487
--builtin_ntp_servers=127.16.19.212:46813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250626 01:58:02.853050 17683 heartbeater.cc:344] Connected to a master server at 127.16.19.254:45487
I20250626 01:58:02.853500 17683 heartbeater.cc:461] Registering TS with master...
I20250626 01:58:02.854508 17683 heartbeater.cc:507] Master 127.16.19.254:45487 requested a full tablet report, sending...
I20250626 01:58:02.856783 17362 ts_manager.cc:194] Registered new tserver with Master: b7044b05a5cb4b96b8f98a35837832f1 (127.16.19.194:39305)
I20250626 01:58:02.858018 17362 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.19.194:49171
W20250626 01:58:03.146447 17687 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250626 01:58:03.147085 17687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:58:03.147346 17687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 01:58:03.147845 17687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:58:03.180075 17687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 01:58:03.180974 17687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.19.195
I20250626 01:58:03.215365 17687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:46813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.19.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.16.19.195
--webserver_port=0
--tserver_master_addrs=127.16.19.254:45487
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.19.195
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 01:58:03.216714 17687 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 01:58:03.218297 17687 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:58:03.234501 17694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:58:03.861353 17683 heartbeater.cc:499] Master 127.16.19.254:45487 was elected leader, sending a full tablet report...
W20250626 01:58:03.235086 17696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:58:03.236207 17687 server_base.cc:1048] running on GCE node
W20250626 01:58:03.234532 17693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:58:04.404804 17687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 01:58:04.406999 17687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 01:58:04.408363 17687 hybrid_clock.cc:648] HybridClock initialized: now 1750903084408341 us; error 44 us; skew 500 ppm
I20250626 01:58:04.409211 17687 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:58:04.418493 17687 webserver.cc:469] Webserver started at http://127.16.19.195:36515/ using document root <none> and password file <none>
I20250626 01:58:04.419483 17687 fs_manager.cc:362] Metadata directory not provided
I20250626 01:58:04.419683 17687 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:58:04.420171 17687 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:58:04.424741 17687 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "c25ce3ff37714b2bb3247b1e1e0b2888"
format_stamp: "Formatted at 2025-06-26 01:58:04 on dist-test-slave-h4rq"
I20250626 01:58:04.425891 17687 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "c25ce3ff37714b2bb3247b1e1e0b2888"
format_stamp: "Formatted at 2025-06-26 01:58:04 on dist-test-slave-h4rq"
I20250626 01:58:04.433445 17687 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250626 01:58:04.439555 17703 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:58:04.440692 17687 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20250626 01:58:04.441028 17687 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "c25ce3ff37714b2bb3247b1e1e0b2888"
format_stamp: "Formatted at 2025-06-26 01:58:04 on dist-test-slave-h4rq"
I20250626 01:58:04.441372 17687 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:58:04.493521 17687 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:58:04.494956 17687 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 01:58:04.495384 17687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:58:04.497959 17687 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:58:04.501909 17687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:58:04.502102 17687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:58:04.502380 17687 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:58:04.502537 17687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:58:04.635656 17687 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.195:32815
I20250626 01:58:04.635766 17815 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.195:32815 every 8 connection(s)
I20250626 01:58:04.638234 17687 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250626 01:58:04.639523 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 17687
I20250626 01:58:04.640081 16463 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250626 01:58:04.659546 17816 heartbeater.cc:344] Connected to a master server at 127.16.19.254:45487
I20250626 01:58:04.659982 17816 heartbeater.cc:461] Registering TS with master...
I20250626 01:58:04.660959 17816 heartbeater.cc:507] Master 127.16.19.254:45487 requested a full tablet report, sending...
I20250626 01:58:04.662827 17362 ts_manager.cc:194] Registered new tserver with Master: c25ce3ff37714b2bb3247b1e1e0b2888 (127.16.19.195:32815)
I20250626 01:58:04.664024 17362 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.19.195:34221
I20250626 01:58:04.674541 16463 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 01:58:04.709040 17362 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42554:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250626 01:58:04.728067 17362 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 01:58:04.781471 17485 tablet_service.cc:1468] Processing CreateTablet for tablet 9839ce94f2ad453e9d07902599652b9c (DEFAULT_TABLE table=TestTable [id=1697091a996d4a579437336c4068f36d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 01:58:04.783555 17485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9839ce94f2ad453e9d07902599652b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:58:04.788532 17751 tablet_service.cc:1468] Processing CreateTablet for tablet 9839ce94f2ad453e9d07902599652b9c (DEFAULT_TABLE table=TestTable [id=1697091a996d4a579437336c4068f36d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 01:58:04.789201 17618 tablet_service.cc:1468] Processing CreateTablet for tablet 9839ce94f2ad453e9d07902599652b9c (DEFAULT_TABLE table=TestTable [id=1697091a996d4a579437336c4068f36d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 01:58:04.790433 17751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9839ce94f2ad453e9d07902599652b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:58:04.790962 17618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9839ce94f2ad453e9d07902599652b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:58:04.811049 17835 tablet_bootstrap.cc:492] T 9839ce94f2ad453e9d07902599652b9c P c25ce3ff37714b2bb3247b1e1e0b2888: Bootstrap starting.
I20250626 01:58:04.812288 17836 tablet_bootstrap.cc:492] T 9839ce94f2ad453e9d07902599652b9c P 60170cc2b9514176b92fe5ea741b92cc: Bootstrap starting.
I20250626 01:58:04.818395 17836 tablet_bootstrap.cc:654] T 9839ce94f2ad453e9d07902599652b9c P 60170cc2b9514176b92fe5ea741b92cc: Neither blocks nor log segments found. Creating new log.
I20250626 01:58:04.820010 17835 tablet_bootstrap.cc:654] T 9839ce94f2ad453e9d07902599652b9c P c25ce3ff37714b2bb3247b1e1e0b2888: Neither blocks nor log segments found. Creating new log.
I20250626 01:58:04.820147 17836 log.cc:826] T 9839ce94f2ad453e9d07902599652b9c P 60170cc2b9514176b92fe5ea741b92cc: Log is configured to *not* fsync() on all Append() calls
I20250626 01:58:04.822640 17835 log.cc:826] T 9839ce94f2ad453e9d07902599652b9c P c25ce3ff37714b2bb3247b1e1e0b2888: Log is configured to *not* fsync() on all Append() calls
I20250626 01:58:04.824200 17837 tablet_bootstrap.cc:492] T 9839ce94f2ad453e9d07902599652b9c P b7044b05a5cb4b96b8f98a35837832f1: Bootstrap starting.
I20250626 01:58:04.828119 17836 tablet_bootstrap.cc:492] T 9839ce94f2ad453e9d07902599652b9c P 60170cc2b9514176b92fe5ea741b92cc: No bootstrap required, opened a new log
I20250626 01:58:04.828727 17836 ts_tablet_manager.cc:1397] T 9839ce94f2ad453e9d07902599652b9c P 60170cc2b9514176b92fe5ea741b92cc: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.006s
I20250626 01:58:04.829233 17835 tablet_bootstrap.cc:492] T 9839ce94f2ad453e9d07902599652b9c P c25ce3ff37714b2bb3247b1e1e0b2888: No bootstrap required, opened a new log
I20250626 01:58:04.829887 17835 ts_tablet_manager.cc:1397] T 9839ce94f2ad453e9d07902599652b9c P c25ce3ff37714b2bb3247b1e1e0b2888: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.009s
I20250626 01:58:04.832831 17837 tablet_bootstrap.cc:654] T 9839ce94f2ad453e9d07902599652b9c P b7044b05a5cb4b96b8f98a35837832f1: Neither blocks nor log segments found. Creating new log.
I20250626 01:58:04.835278 17837 log.cc:826] T 9839ce94f2ad453e9d07902599652b9c P b7044b05a5cb4b96b8f98a35837832f1: Log is configured to *not* fsync() on all Append() calls
I20250626 01:58:04.841822 17837 tablet_bootstrap.cc:492] T 9839ce94f2ad453e9d07902599652b9c P b7044b05a5cb4b96b8f98a35837832f1: No bootstrap required, opened a new log
I20250626 01:58:04.842437 17837 ts_tablet_manager.cc:1397] T 9839ce94f2ad453e9d07902599652b9c P b7044b05a5cb4b96b8f98a35837832f1: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.008s
I20250626 01:58:04.857411 17836 raft_consensus.cc:357] T 9839ce94f2ad453e9d07902599652b9c P 60170cc2b9514176b92fe5ea741b92cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60170cc2b9514176b92fe5ea741b92cc" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 37005 } } peers { permanent_uuid: "c25ce3ff37714b2bb3247b1e1e0b2888" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 32815 } } peers { permanent_uuid: "b7044b05a5cb4b96b8f98a35837832f1" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39305 } }
I20250626 01:58:04.858718 17836 raft_consensus.cc:738] T 9839ce94f2ad453e9d07902599652b9c P 60170cc2b9514176b92fe5ea741b92cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60170cc2b9514176b92fe5ea741b92cc, State: Initialized, Role: FOLLOWER
I20250626 01:58:04.858695 17835 raft_consensus.cc:357] T 9839ce94f2ad453e9d07902599652b9c P c25ce3ff37714b2bb3247b1e1e0b2888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60170cc2b9514176b92fe5ea741b92cc" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 37005 } } peers { permanent_uuid: "c25ce3ff37714b2bb3247b1e1e0b2888" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 32815 } } peers { permanent_uuid: "b7044b05a5cb4b96b8f98a35837832f1" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39305 } }
I20250626 01:58:04.859733 17835 raft_consensus.cc:738] T 9839ce94f2ad453e9d07902599652b9c P c25ce3ff37714b2bb3247b1e1e0b2888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c25ce3ff37714b2bb3247b1e1e0b2888, State: Initialized, Role: FOLLOWER
I20250626 01:58:04.859551 17836 consensus_queue.cc:260] T 9839ce94f2ad453e9d07902599652b9c P 60170cc2b9514176b92fe5ea741b92cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60170cc2b9514176b92fe5ea741b92cc" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 37005 } } peers { permanent_uuid: "c25ce3ff37714b2bb3247b1e1e0b2888" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 32815 } } peers { permanent_uuid: "b7044b05a5cb4b96b8f98a35837832f1" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39305 } }
I20250626 01:58:04.860481 17835 consensus_queue.cc:260] T 9839ce94f2ad453e9d07902599652b9c P c25ce3ff37714b2bb3247b1e1e0b2888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60170cc2b9514176b92fe5ea741b92cc" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 37005 } } peers { permanent_uuid: "c25ce3ff37714b2bb3247b1e1e0b2888" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 32815 } } peers { permanent_uuid: "b7044b05a5cb4b96b8f98a35837832f1" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39305 } }
I20250626 01:58:04.870358 17836 ts_tablet_manager.cc:1428] T 9839ce94f2ad453e9d07902599652b9c P 60170cc2b9514176b92fe5ea741b92cc: Time spent starting tablet: real 0.041s user 0.027s sys 0.010s
I20250626 01:58:04.875998 17816 heartbeater.cc:499] Master 127.16.19.254:45487 was elected leader, sending a full tablet report...
I20250626 01:58:04.877641 17835 ts_tablet_manager.cc:1428] T 9839ce94f2ad453e9d07902599652b9c P c25ce3ff37714b2bb3247b1e1e0b2888: Time spent starting tablet: real 0.047s user 0.028s sys 0.009s
I20250626 01:58:04.878197 17837 raft_consensus.cc:357] T 9839ce94f2ad453e9d07902599652b9c P b7044b05a5cb4b96b8f98a35837832f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60170cc2b9514176b92fe5ea741b92cc" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 37005 } } peers { permanent_uuid: "c25ce3ff37714b2bb3247b1e1e0b2888" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 32815 } } peers { permanent_uuid: "b7044b05a5cb4b96b8f98a35837832f1" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39305 } }
I20250626 01:58:04.879099 17837 raft_consensus.cc:738] T 9839ce94f2ad453e9d07902599652b9c P b7044b05a5cb4b96b8f98a35837832f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7044b05a5cb4b96b8f98a35837832f1, State: Initialized, Role: FOLLOWER
I20250626 01:58:04.879807 17837 consensus_queue.cc:260] T 9839ce94f2ad453e9d07902599652b9c P b7044b05a5cb4b96b8f98a35837832f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60170cc2b9514176b92fe5ea741b92cc" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 37005 } } peers { permanent_uuid: "c25ce3ff37714b2bb3247b1e1e0b2888" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 32815 } } peers { permanent_uuid: "b7044b05a5cb4b96b8f98a35837832f1" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39305 } }
I20250626 01:58:04.883369 17837 ts_tablet_manager.cc:1428] T 9839ce94f2ad453e9d07902599652b9c P b7044b05a5cb4b96b8f98a35837832f1: Time spent starting tablet: real 0.041s user 0.031s sys 0.001s
W20250626 01:58:04.893153 17817 tablet.cc:2378] T 9839ce94f2ad453e9d07902599652b9c P c25ce3ff37714b2bb3247b1e1e0b2888: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 01:58:04.900913 16463 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 01:58:04.904268 16463 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 60170cc2b9514176b92fe5ea741b92cc to finish bootstrapping
I20250626 01:58:04.917109 16463 ts_itest-base.cc:246] Waiting for 1 tablets on tserver b7044b05a5cb4b96b8f98a35837832f1 to finish bootstrapping
I20250626 01:58:04.927306 16463 ts_itest-base.cc:246] Waiting for 1 tablets on tserver c25ce3ff37714b2bb3247b1e1e0b2888 to finish bootstrapping
I20250626 01:58:04.971181 17505 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "9839ce94f2ad453e9d07902599652b9c"
dest_uuid: "60170cc2b9514176b92fe5ea741b92cc"
from {username='slave'} at 127.0.0.1:56044
I20250626 01:58:04.971740 17505 raft_consensus.cc:491] T 9839ce94f2ad453e9d07902599652b9c P 60170cc2b9514176b92fe5ea741b92cc [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:58:04.972021 17505 raft_consensus.cc:3058] T 9839ce94f2ad453e9d07902599652b9c P 60170cc2b9514176b92fe5ea741b92cc [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:58:04.976490 17505 raft_consensus.cc:513] T 9839ce94f2ad453e9d07902599652b9c P 60170cc2b9514176b92fe5ea741b92cc [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60170cc2b9514176b92fe5ea741b92cc" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 37005 } } peers { permanent_uuid: "c25ce3ff37714b2bb3247b1e1e0b2888" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 32815 } } peers { permanent_uuid: "b7044b05a5cb4b96b8f98a35837832f1" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39305 } }
I20250626 01:58:04.978731 17505 leader_election.cc:290] T 9839ce94f2ad453e9d07902599652b9c P 60170cc2b9514176b92fe5ea741b92cc [CANDIDATE]: Term 1 election: Requested vote from peers c25ce3ff37714b2bb3247b1e1e0b2888 (127.16.19.195:32815), b7044b05a5cb4b96b8f98a35837832f1 (127.16.19.194:39305)
I20250626 01:58:04.990633 16463 cluster_itest_util.cc:257] Not converged past 1 yet: 0.0 0.0 0.0
I20250626 01:58:04.991701 17771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9839ce94f2ad453e9d07902599652b9c" candidate_uuid: "60170cc2b9514176b92fe5ea741b92cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c25ce3ff37714b2bb3247b1e1e0b2888"
I20250626 01:58:04.992467 17771 raft_consensus.cc:3058] T 9839ce94f2ad453e9d07902599652b9c P c25ce3ff37714b2bb3247b1e1e0b2888 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:58:04.994246 17638 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9839ce94f2ad453e9d07902599652b9c" candidate_uuid: "60170cc2b9514176b92fe5ea741b92cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b7044b05a5cb4b96b8f98a35837832f1"
I20250626 01:58:04.995033 17638 raft_consensus.cc:3058] T 9839ce94f2ad453e9d07902599652b9c P b7044b05a5cb4b96b8f98a35837832f1 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:58:04.999675 17771 raft_consensus.cc:2466] T 9839ce94f2ad453e9d07902599652b9c P c25ce3ff37714b2bb3247b1e1e0b2888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60170cc2b9514176b92fe5ea741b92cc in term 1.
I20250626 01:58:05.001147 17439 leader_election.cc:304] T 9839ce94f2ad453e9d07902599652b9c P 60170cc2b9514176b92fe5ea741b92cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60170cc2b9514176b92fe5ea741b92cc, c25ce3ff37714b2bb3247b1e1e0b2888; no voters:
I20250626 01:58:05.001868 17638 raft_consensus.cc:2466] T 9839ce94f2ad453e9d07902599652b9c P b7044b05a5cb4b96b8f98a35837832f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60170cc2b9514176b92fe5ea741b92cc in term 1.
I20250626 01:58:05.002183 17841 raft_consensus.cc:2802] T 9839ce94f2ad453e9d07902599652b9c P 60170cc2b9514176b92fe5ea741b92cc [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:58:05.004245 17841 raft_consensus.cc:695] T 9839ce94f2ad453e9d07902599652b9c P 60170cc2b9514176b92fe5ea741b92cc [term 1 LEADER]: Becoming Leader. State: Replica: 60170cc2b9514176b92fe5ea741b92cc, State: Running, Role: LEADER
I20250626 01:58:05.005254 17841 consensus_queue.cc:237] T 9839ce94f2ad453e9d07902599652b9c P 60170cc2b9514176b92fe5ea741b92cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60170cc2b9514176b92fe5ea741b92cc" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 37005 } } peers { permanent_uuid: "c25ce3ff37714b2bb3247b1e1e0b2888" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 32815 } } peers { permanent_uuid: "b7044b05a5cb4b96b8f98a35837832f1" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39305 } }
I20250626 01:58:05.018412 17362 catalog_manager.cc:5582] T 9839ce94f2ad453e9d07902599652b9c P 60170cc2b9514176b92fe5ea741b92cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 60170cc2b9514176b92fe5ea741b92cc (127.16.19.193). New cstate: current_term: 1 leader_uuid: "60170cc2b9514176b92fe5ea741b92cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60170cc2b9514176b92fe5ea741b92cc" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 37005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c25ce3ff37714b2bb3247b1e1e0b2888" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 32815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7044b05a5cb4b96b8f98a35837832f1" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39305 } health_report { overall_health: UNKNOWN } } }
W20250626 01:58:05.032706 17551 tablet.cc:2378] T 9839ce94f2ad453e9d07902599652b9c P 60170cc2b9514176b92fe5ea741b92cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250626 01:58:05.088124 17684 tablet.cc:2378] T 9839ce94f2ad453e9d07902599652b9c P b7044b05a5cb4b96b8f98a35837832f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 01:58:05.096526 16463 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250626 01:58:05.301908 16463 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250626 01:58:05.508693 17841 consensus_queue.cc:1035] T 9839ce94f2ad453e9d07902599652b9c P 60170cc2b9514176b92fe5ea741b92cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7044b05a5cb4b96b8f98a35837832f1" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:58:05.525485 17841 consensus_queue.cc:1035] T 9839ce94f2ad453e9d07902599652b9c P 60170cc2b9514176b92fe5ea741b92cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c25ce3ff37714b2bb3247b1e1e0b2888" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 32815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:58:07.308532 17505 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "9839ce94f2ad453e9d07902599652b9c"
dest_uuid: "60170cc2b9514176b92fe5ea741b92cc"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51262
I20250626 01:58:07.309121 17505 raft_consensus.cc:604] T 9839ce94f2ad453e9d07902599652b9c P 60170cc2b9514176b92fe5ea741b92cc [term 1 LEADER]: Received request to transfer leadership
I20250626 01:58:07.577950 17879 raft_consensus.cc:991] T 9839ce94f2ad453e9d07902599652b9c P 60170cc2b9514176b92fe5ea741b92cc: : Instructing follower c25ce3ff37714b2bb3247b1e1e0b2888 to start an election
I20250626 01:58:07.578353 17863 raft_consensus.cc:1079] T 9839ce94f2ad453e9d07902599652b9c P 60170cc2b9514176b92fe5ea741b92cc [term 1 LEADER]: Signalling peer c25ce3ff37714b2bb3247b1e1e0b2888 to start an election
I20250626 01:58:07.579751 17771 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "9839ce94f2ad453e9d07902599652b9c"
dest_uuid: "c25ce3ff37714b2bb3247b1e1e0b2888"
from {username='slave'} at 127.16.19.193:50489
I20250626 01:58:07.580284 17771 raft_consensus.cc:491] T 9839ce94f2ad453e9d07902599652b9c P c25ce3ff37714b2bb3247b1e1e0b2888 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:58:07.580547 17771 raft_consensus.cc:3058] T 9839ce94f2ad453e9d07902599652b9c P c25ce3ff37714b2bb3247b1e1e0b2888 [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:58:07.584689 17771 raft_consensus.cc:513] T 9839ce94f2ad453e9d07902599652b9c P c25ce3ff37714b2bb3247b1e1e0b2888 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60170cc2b9514176b92fe5ea741b92cc" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 37005 } } peers { permanent_uuid: "c25ce3ff37714b2bb3247b1e1e0b2888" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 32815 } } peers { permanent_uuid: "b7044b05a5cb4b96b8f98a35837832f1" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39305 } }
I20250626 01:58:07.586835 17771 leader_election.cc:290] T 9839ce94f2ad453e9d07902599652b9c P c25ce3ff37714b2bb3247b1e1e0b2888 [CANDIDATE]: Term 2 election: Requested vote from peers 60170cc2b9514176b92fe5ea741b92cc (127.16.19.193:37005), b7044b05a5cb4b96b8f98a35837832f1 (127.16.19.194:39305)
I20250626 01:58:07.597564 17505 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9839ce94f2ad453e9d07902599652b9c" candidate_uuid: "c25ce3ff37714b2bb3247b1e1e0b2888" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "60170cc2b9514176b92fe5ea741b92cc"
I20250626 01:58:07.597788 17638 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9839ce94f2ad453e9d07902599652b9c" candidate_uuid: "c25ce3ff37714b2bb3247b1e1e0b2888" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b7044b05a5cb4b96b8f98a35837832f1"
I20250626 01:58:07.598248 17505 raft_consensus.cc:3053] T 9839ce94f2ad453e9d07902599652b9c P 60170cc2b9514176b92fe5ea741b92cc [term 1 LEADER]: Stepping down as leader of term 1
I20250626 01:58:07.598316 17638 raft_consensus.cc:3058] T 9839ce94f2ad453e9d07902599652b9c P b7044b05a5cb4b96b8f98a35837832f1 [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:58:07.598572 17505 raft_consensus.cc:738] T 9839ce94f2ad453e9d07902599652b9c P 60170cc2b9514176b92fe5ea741b92cc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 60170cc2b9514176b92fe5ea741b92cc, State: Running, Role: LEADER
I20250626 01:58:07.599227 17505 consensus_queue.cc:260] T 9839ce94f2ad453e9d07902599652b9c P 60170cc2b9514176b92fe5ea741b92cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60170cc2b9514176b92fe5ea741b92cc" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 37005 } } peers { permanent_uuid: "c25ce3ff37714b2bb3247b1e1e0b2888" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 32815 } } peers { permanent_uuid: "b7044b05a5cb4b96b8f98a35837832f1" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39305 } }
I20250626 01:58:07.600483 17505 raft_consensus.cc:3058] T 9839ce94f2ad453e9d07902599652b9c P 60170cc2b9514176b92fe5ea741b92cc [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:58:07.604843 17638 raft_consensus.cc:2466] T 9839ce94f2ad453e9d07902599652b9c P b7044b05a5cb4b96b8f98a35837832f1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c25ce3ff37714b2bb3247b1e1e0b2888 in term 2.
I20250626 01:58:07.606171 17705 leader_election.cc:304] T 9839ce94f2ad453e9d07902599652b9c P c25ce3ff37714b2bb3247b1e1e0b2888 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7044b05a5cb4b96b8f98a35837832f1, c25ce3ff37714b2bb3247b1e1e0b2888; no voters:
I20250626 01:58:07.607076 17505 raft_consensus.cc:2466] T 9839ce94f2ad453e9d07902599652b9c P 60170cc2b9514176b92fe5ea741b92cc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c25ce3ff37714b2bb3247b1e1e0b2888 in term 2.
I20250626 01:58:07.608559 17883 raft_consensus.cc:2802] T 9839ce94f2ad453e9d07902599652b9c P c25ce3ff37714b2bb3247b1e1e0b2888 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 01:58:07.609906 17883 raft_consensus.cc:695] T 9839ce94f2ad453e9d07902599652b9c P c25ce3ff37714b2bb3247b1e1e0b2888 [term 2 LEADER]: Becoming Leader. State: Replica: c25ce3ff37714b2bb3247b1e1e0b2888, State: Running, Role: LEADER
I20250626 01:58:07.610658 17883 consensus_queue.cc:237] T 9839ce94f2ad453e9d07902599652b9c P c25ce3ff37714b2bb3247b1e1e0b2888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60170cc2b9514176b92fe5ea741b92cc" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 37005 } } peers { permanent_uuid: "c25ce3ff37714b2bb3247b1e1e0b2888" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 32815 } } peers { permanent_uuid: "b7044b05a5cb4b96b8f98a35837832f1" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39305 } }
I20250626 01:58:07.618563 17361 catalog_manager.cc:5582] T 9839ce94f2ad453e9d07902599652b9c P c25ce3ff37714b2bb3247b1e1e0b2888 reported cstate change: term changed from 1 to 2, leader changed from 60170cc2b9514176b92fe5ea741b92cc (127.16.19.193) to c25ce3ff37714b2bb3247b1e1e0b2888 (127.16.19.195). New cstate: current_term: 2 leader_uuid: "c25ce3ff37714b2bb3247b1e1e0b2888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60170cc2b9514176b92fe5ea741b92cc" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 37005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c25ce3ff37714b2bb3247b1e1e0b2888" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 32815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7044b05a5cb4b96b8f98a35837832f1" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39305 } health_report { overall_health: UNKNOWN } } }
I20250626 01:58:08.029244 17638 raft_consensus.cc:1273] T 9839ce94f2ad453e9d07902599652b9c P b7044b05a5cb4b96b8f98a35837832f1 [term 2 FOLLOWER]: Refusing update from remote peer c25ce3ff37714b2bb3247b1e1e0b2888: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 01:58:08.030777 17883 consensus_queue.cc:1035] T 9839ce94f2ad453e9d07902599652b9c P c25ce3ff37714b2bb3247b1e1e0b2888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7044b05a5cb4b96b8f98a35837832f1" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 01:58:08.049491 17505 raft_consensus.cc:1273] T 9839ce94f2ad453e9d07902599652b9c P 60170cc2b9514176b92fe5ea741b92cc [term 2 FOLLOWER]: Refusing update from remote peer c25ce3ff37714b2bb3247b1e1e0b2888: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 01:58:08.051209 17884 consensus_queue.cc:1035] T 9839ce94f2ad453e9d07902599652b9c P c25ce3ff37714b2bb3247b1e1e0b2888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60170cc2b9514176b92fe5ea741b92cc" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250626 01:58:10.044710 17505 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "9839ce94f2ad453e9d07902599652b9c"
dest_uuid: "60170cc2b9514176b92fe5ea741b92cc"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51274
I20250626 01:58:10.045620 17505 raft_consensus.cc:604] T 9839ce94f2ad453e9d07902599652b9c P 60170cc2b9514176b92fe5ea741b92cc [term 2 FOLLOWER]: Received request to transfer leadership
I20250626 01:58:10.046128 17505 raft_consensus.cc:612] T 9839ce94f2ad453e9d07902599652b9c P 60170cc2b9514176b92fe5ea741b92cc [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250626 01:58:11.086995 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 17421
I20250626 01:58:11.112402 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 17554
I20250626 01:58:11.140035 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 17687
I20250626 01:58:11.165282 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 17328
2025-06-26T01:58:11Z chronyd exiting
[ OK ] AdminCliTest.TestGracefulSpecificLeaderStepDown (13994 ms)
[ RUN ] AdminCliTest.TestDescribeTableColumnFlags
I20250626 01:58:11.219769 16463 test_util.cc:276] Using random seed: -456536808
I20250626 01:58:11.224227 16463 ts_itest-base.cc:115] Starting cluster with:
I20250626 01:58:11.224416 16463 ts_itest-base.cc:116] --------------
I20250626 01:58:11.224563 16463 ts_itest-base.cc:117] 3 tablet servers
I20250626 01:58:11.224761 16463 ts_itest-base.cc:118] 3 replicas per TS
I20250626 01:58:11.224924 16463 ts_itest-base.cc:119] --------------
2025-06-26T01:58:11Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-26T01:58:11Z Disabled control of system clock
I20250626 01:58:11.263917 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.19.254:36099
--webserver_interface=127.16.19.254
--webserver_port=0
--builtin_ntp_servers=127.16.19.212:33037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.19.254:36099 with env {}
W20250626 01:58:11.566974 17925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:58:11.567600 17925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 01:58:11.568089 17925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:58:11.599632 17925 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 01:58:11.599987 17925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 01:58:11.600251 17925 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 01:58:11.600494 17925 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 01:58:11.636366 17925 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:33037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.19.254:36099
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.19.254:36099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.19.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 01:58:11.637750 17925 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 01:58:11.639446 17925 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:58:11.655270 17934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:58:11.655329 17932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:58:11.657021 17925 server_base.cc:1048] running on GCE node
W20250626 01:58:11.655391 17931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:58:12.847061 17925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 01:58:12.849792 17925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 01:58:12.851176 17925 hybrid_clock.cc:648] HybridClock initialized: now 1750903092851118 us; error 85 us; skew 500 ppm
I20250626 01:58:12.851979 17925 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:58:12.858953 17925 webserver.cc:469] Webserver started at http://127.16.19.254:41479/ using document root <none> and password file <none>
I20250626 01:58:12.859934 17925 fs_manager.cc:362] Metadata directory not provided
I20250626 01:58:12.860173 17925 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:58:12.860683 17925 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:58:12.865276 17925 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "ac887e2e2f8e43de8ce038b26545d64b"
format_stamp: "Formatted at 2025-06-26 01:58:12 on dist-test-slave-h4rq"
I20250626 01:58:12.866390 17925 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "ac887e2e2f8e43de8ce038b26545d64b"
format_stamp: "Formatted at 2025-06-26 01:58:12 on dist-test-slave-h4rq"
I20250626 01:58:12.873773 17925 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250626 01:58:12.879972 17941 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:58:12.881366 17925 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250626 01:58:12.881703 17925 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ac887e2e2f8e43de8ce038b26545d64b"
format_stamp: "Formatted at 2025-06-26 01:58:12 on dist-test-slave-h4rq"
I20250626 01:58:12.882048 17925 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:58:12.951220 17925 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:58:12.952790 17925 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 01:58:12.953244 17925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:58:13.026470 17925 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.254:36099
I20250626 01:58:13.026546 17992 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.254:36099 every 8 connection(s)
I20250626 01:58:13.029196 17925 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250626 01:58:13.031971 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 17925
I20250626 01:58:13.032500 16463 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250626 01:58:13.035524 17993 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:58:13.059332 17993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac887e2e2f8e43de8ce038b26545d64b: Bootstrap starting.
I20250626 01:58:13.065202 17993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac887e2e2f8e43de8ce038b26545d64b: Neither blocks nor log segments found. Creating new log.
I20250626 01:58:13.066900 17993 log.cc:826] T 00000000000000000000000000000000 P ac887e2e2f8e43de8ce038b26545d64b: Log is configured to *not* fsync() on all Append() calls
I20250626 01:58:13.071518 17993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac887e2e2f8e43de8ce038b26545d64b: No bootstrap required, opened a new log
I20250626 01:58:13.090378 17993 raft_consensus.cc:357] T 00000000000000000000000000000000 P ac887e2e2f8e43de8ce038b26545d64b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac887e2e2f8e43de8ce038b26545d64b" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 36099 } }
I20250626 01:58:13.091095 17993 raft_consensus.cc:383] T 00000000000000000000000000000000 P ac887e2e2f8e43de8ce038b26545d64b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:58:13.091353 17993 raft_consensus.cc:738] T 00000000000000000000000000000000 P ac887e2e2f8e43de8ce038b26545d64b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac887e2e2f8e43de8ce038b26545d64b, State: Initialized, Role: FOLLOWER
I20250626 01:58:13.092116 17993 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac887e2e2f8e43de8ce038b26545d64b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac887e2e2f8e43de8ce038b26545d64b" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 36099 } }
I20250626 01:58:13.092680 17993 raft_consensus.cc:397] T 00000000000000000000000000000000 P ac887e2e2f8e43de8ce038b26545d64b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 01:58:13.092957 17993 raft_consensus.cc:491] T 00000000000000000000000000000000 P ac887e2e2f8e43de8ce038b26545d64b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 01:58:13.093271 17993 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ac887e2e2f8e43de8ce038b26545d64b [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:58:13.098348 17993 raft_consensus.cc:513] T 00000000000000000000000000000000 P ac887e2e2f8e43de8ce038b26545d64b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac887e2e2f8e43de8ce038b26545d64b" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 36099 } }
I20250626 01:58:13.099069 17993 leader_election.cc:304] T 00000000000000000000000000000000 P ac887e2e2f8e43de8ce038b26545d64b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac887e2e2f8e43de8ce038b26545d64b; no voters:
I20250626 01:58:13.100821 17993 leader_election.cc:290] T 00000000000000000000000000000000 P ac887e2e2f8e43de8ce038b26545d64b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 01:58:13.101552 17998 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ac887e2e2f8e43de8ce038b26545d64b [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:58:13.103947 17998 raft_consensus.cc:695] T 00000000000000000000000000000000 P ac887e2e2f8e43de8ce038b26545d64b [term 1 LEADER]: Becoming Leader. State: Replica: ac887e2e2f8e43de8ce038b26545d64b, State: Running, Role: LEADER
I20250626 01:58:13.104823 17998 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac887e2e2f8e43de8ce038b26545d64b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac887e2e2f8e43de8ce038b26545d64b" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 36099 } }
I20250626 01:58:13.106029 17993 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac887e2e2f8e43de8ce038b26545d64b [sys.catalog]: configured and running, proceeding with master startup.
I20250626 01:58:13.116922 17999 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac887e2e2f8e43de8ce038b26545d64b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac887e2e2f8e43de8ce038b26545d64b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac887e2e2f8e43de8ce038b26545d64b" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 36099 } } }
I20250626 01:58:13.119051 17999 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac887e2e2f8e43de8ce038b26545d64b [sys.catalog]: This master's current role is: LEADER
I20250626 01:58:13.118422 18000 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac887e2e2f8e43de8ce038b26545d64b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac887e2e2f8e43de8ce038b26545d64b. Latest consensus state: current_term: 1 leader_uuid: "ac887e2e2f8e43de8ce038b26545d64b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac887e2e2f8e43de8ce038b26545d64b" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 36099 } } }
I20250626 01:58:13.120671 18000 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac887e2e2f8e43de8ce038b26545d64b [sys.catalog]: This master's current role is: LEADER
I20250626 01:58:13.130673 18007 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 01:58:13.143347 18007 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 01:58:13.159435 18007 catalog_manager.cc:1349] Generated new cluster ID: a98878ffbbce418da81689d73bc03d79
I20250626 01:58:13.159713 18007 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 01:58:13.186479 18007 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 01:58:13.187938 18007 catalog_manager.cc:1506] Loading token signing keys...
I20250626 01:58:13.203061 18007 catalog_manager.cc:5955] T 00000000000000000000000000000000 P ac887e2e2f8e43de8ce038b26545d64b: Generated new TSK 0
I20250626 01:58:13.204035 18007 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 01:58:13.217438 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.19.193:0
--local_ip_for_outbound_sockets=127.16.19.193
--webserver_interface=127.16.19.193
--webserver_port=0
--tserver_master_addrs=127.16.19.254:36099
--builtin_ntp_servers=127.16.19.212:33037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250626 01:58:13.545364 18018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:58:13.545892 18018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 01:58:13.546418 18018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:58:13.580001 18018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 01:58:13.580945 18018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.19.193
I20250626 01:58:13.617529 18018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:33037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.19.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.19.193
--webserver_port=0
--tserver_master_addrs=127.16.19.254:36099
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.19.193
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 01:58:13.618927 18018 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 01:58:13.621100 18018 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:58:13.639761 18025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:58:13.643661 18027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:58:13.644335 18018 server_base.cc:1048] running on GCE node
W20250626 01:58:13.641683 18024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:58:14.853129 18018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 01:58:14.855890 18018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 01:58:14.857375 18018 hybrid_clock.cc:648] HybridClock initialized: now 1750903094857311 us; error 83 us; skew 500 ppm
I20250626 01:58:14.858240 18018 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:58:14.865669 18018 webserver.cc:469] Webserver started at http://127.16.19.193:45729/ using document root <none> and password file <none>
I20250626 01:58:14.866626 18018 fs_manager.cc:362] Metadata directory not provided
I20250626 01:58:14.866837 18018 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:58:14.867355 18018 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:58:14.872038 18018 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "d00e5cdd9be049cfb3d836852b287f2a"
format_stamp: "Formatted at 2025-06-26 01:58:14 on dist-test-slave-h4rq"
I20250626 01:58:14.873243 18018 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "d00e5cdd9be049cfb3d836852b287f2a"
format_stamp: "Formatted at 2025-06-26 01:58:14 on dist-test-slave-h4rq"
W20250626 01:58:14.876149 17989 debug-util.cc:398] Leaking SignalData structure 0x7b080006f060 after lost signal to thread 17926
W20250626 01:58:14.876891 17989 debug-util.cc:398] Leaking SignalData structure 0x7b080006f7a0 after lost signal to thread 17992
I20250626 01:58:14.882880 18018 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.009s sys 0.000s
I20250626 01:58:14.893473 18034 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:58:14.895020 18018 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20250626 01:58:14.895457 18018 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d00e5cdd9be049cfb3d836852b287f2a"
format_stamp: "Formatted at 2025-06-26 01:58:14 on dist-test-slave-h4rq"
I20250626 01:58:14.895938 18018 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:58:14.968086 18018 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:58:14.970513 18018 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 01:58:14.971228 18018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:58:14.975199 18018 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:58:14.981964 18018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:58:14.982372 18018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250626 01:58:14.982753 18018 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:58:14.983026 18018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:58:15.158187 18018 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.193:34023
I20250626 01:58:15.158233 18146 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.193:34023 every 8 connection(s)
I20250626 01:58:15.162503 18018 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250626 01:58:15.165181 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 18018
I20250626 01:58:15.165694 16463 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250626 01:58:15.175779 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.19.194:0
--local_ip_for_outbound_sockets=127.16.19.194
--webserver_interface=127.16.19.194
--webserver_port=0
--tserver_master_addrs=127.16.19.254:36099
--builtin_ntp_servers=127.16.19.212:33037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 01:58:15.195706 18147 heartbeater.cc:344] Connected to a master server at 127.16.19.254:36099
I20250626 01:58:15.196381 18147 heartbeater.cc:461] Registering TS with master...
I20250626 01:58:15.198041 18147 heartbeater.cc:507] Master 127.16.19.254:36099 requested a full tablet report, sending...
I20250626 01:58:15.201766 17958 ts_manager.cc:194] Registered new tserver with Master: d00e5cdd9be049cfb3d836852b287f2a (127.16.19.193:34023)
I20250626 01:58:15.204536 17958 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.19.193:58271
W20250626 01:58:15.485702 18151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:58:15.486232 18151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 01:58:15.486753 18151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:58:15.519944 18151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 01:58:15.520938 18151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.19.194
I20250626 01:58:15.557188 18151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:33037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.19.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.16.19.194
--webserver_port=0
--tserver_master_addrs=127.16.19.254:36099
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.19.194
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 01:58:15.558549 18151 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 01:58:15.560164 18151 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:58:15.578141 18158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:58:16.209049 18147 heartbeater.cc:499] Master 127.16.19.254:36099 was elected leader, sending a full tablet report...
W20250626 01:58:15.579655 18160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:58:15.579025 18157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:58:16.757103 18159 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 01:58:16.757331 18151 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 01:58:16.761689 18151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 01:58:16.764657 18151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 01:58:16.766216 18151 hybrid_clock.cc:648] HybridClock initialized: now 1750903096766164 us; error 56 us; skew 500 ppm
I20250626 01:58:16.767362 18151 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:58:16.775503 18151 webserver.cc:469] Webserver started at http://127.16.19.194:39775/ using document root <none> and password file <none>
I20250626 01:58:16.776911 18151 fs_manager.cc:362] Metadata directory not provided
I20250626 01:58:16.777207 18151 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:58:16.777834 18151 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:58:16.785310 18151 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "db76bdb9bbbe42e2a32c2598a6398f19"
format_stamp: "Formatted at 2025-06-26 01:58:16 on dist-test-slave-h4rq"
I20250626 01:58:16.786918 18151 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "db76bdb9bbbe42e2a32c2598a6398f19"
format_stamp: "Formatted at 2025-06-26 01:58:16 on dist-test-slave-h4rq"
I20250626 01:58:16.797116 18151 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.011s sys 0.000s
I20250626 01:58:16.805259 18167 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:58:16.806581 18151 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250626 01:58:16.807008 18151 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "db76bdb9bbbe42e2a32c2598a6398f19"
format_stamp: "Formatted at 2025-06-26 01:58:16 on dist-test-slave-h4rq"
I20250626 01:58:16.807531 18151 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:58:16.870666 18151 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:58:16.872094 18151 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 01:58:16.872545 18151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:58:16.875695 18151 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:58:16.879938 18151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:58:16.880158 18151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:58:16.880414 18151 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:58:16.880569 18151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:58:17.030736 18151 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.194:41169
I20250626 01:58:17.030835 18279 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.194:41169 every 8 connection(s)
I20250626 01:58:17.033336 18151 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250626 01:58:17.042366 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 18151
I20250626 01:58:17.042785 16463 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250626 01:58:17.049455 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.19.195:0
--local_ip_for_outbound_sockets=127.16.19.195
--webserver_interface=127.16.19.195
--webserver_port=0
--tserver_master_addrs=127.16.19.254:36099
--builtin_ntp_servers=127.16.19.212:33037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 01:58:17.055697 18280 heartbeater.cc:344] Connected to a master server at 127.16.19.254:36099
I20250626 01:58:17.056269 18280 heartbeater.cc:461] Registering TS with master...
I20250626 01:58:17.057442 18280 heartbeater.cc:507] Master 127.16.19.254:36099 requested a full tablet report, sending...
I20250626 01:58:17.059725 17958 ts_manager.cc:194] Registered new tserver with Master: db76bdb9bbbe42e2a32c2598a6398f19 (127.16.19.194:41169)
I20250626 01:58:17.061017 17958 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.19.194:59813
W20250626 01:58:17.356024 18284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:58:17.356565 18284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 01:58:17.357126 18284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:58:17.388034 18284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 01:58:17.388979 18284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.19.195
I20250626 01:58:17.423808 18284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:33037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.19.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.16.19.195
--webserver_port=0
--tserver_master_addrs=127.16.19.254:36099
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.19.195
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 01:58:17.425184 18284 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 01:58:17.426823 18284 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:58:17.442615 18290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:58:18.064363 18280 heartbeater.cc:499] Master 127.16.19.254:36099 was elected leader, sending a full tablet report...
W20250626 01:58:17.442755 18291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:58:17.442646 18293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:58:18.659883 18292 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1217 milliseconds
I20250626 01:58:18.659991 18284 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 01:58:18.661198 18284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 01:58:18.663826 18284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 01:58:18.665304 18284 hybrid_clock.cc:648] HybridClock initialized: now 1750903098665264 us; error 59 us; skew 500 ppm
I20250626 01:58:18.666110 18284 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:58:18.672762 18284 webserver.cc:469] Webserver started at http://127.16.19.195:37975/ using document root <none> and password file <none>
I20250626 01:58:18.673722 18284 fs_manager.cc:362] Metadata directory not provided
I20250626 01:58:18.673923 18284 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:58:18.674407 18284 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:58:18.678810 18284 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "c07e3ae9d7b84044820f9e655e82556c"
format_stamp: "Formatted at 2025-06-26 01:58:18 on dist-test-slave-h4rq"
I20250626 01:58:18.679894 18284 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "c07e3ae9d7b84044820f9e655e82556c"
format_stamp: "Formatted at 2025-06-26 01:58:18 on dist-test-slave-h4rq"
I20250626 01:58:18.687165 18284 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250626 01:58:18.692785 18301 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:58:18.693854 18284 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250626 01:58:18.694169 18284 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "c07e3ae9d7b84044820f9e655e82556c"
format_stamp: "Formatted at 2025-06-26 01:58:18 on dist-test-slave-h4rq"
I20250626 01:58:18.694517 18284 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:58:18.753489 18284 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:58:18.755070 18284 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 01:58:18.755545 18284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:58:18.758091 18284 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:58:18.762235 18284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:58:18.762452 18284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:58:18.762701 18284 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:58:18.762861 18284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:58:18.900211 18284 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.195:45111
I20250626 01:58:18.900331 18413 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.195:45111 every 8 connection(s)
I20250626 01:58:18.903597 18284 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250626 01:58:18.908555 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 18284
I20250626 01:58:18.909322 16463 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250626 01:58:18.932516 18414 heartbeater.cc:344] Connected to a master server at 127.16.19.254:36099
I20250626 01:58:18.933012 18414 heartbeater.cc:461] Registering TS with master...
I20250626 01:58:18.934052 18414 heartbeater.cc:507] Master 127.16.19.254:36099 requested a full tablet report, sending...
I20250626 01:58:18.936744 17958 ts_manager.cc:194] Registered new tserver with Master: c07e3ae9d7b84044820f9e655e82556c (127.16.19.195:45111)
I20250626 01:58:18.938308 17958 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.19.195:56517
I20250626 01:58:18.945775 16463 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 01:58:18.981791 17958 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44008:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250626 01:58:19.000967 17958 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 01:58:19.054438 18082 tablet_service.cc:1468] Processing CreateTablet for tablet ae27453d5aea4f648e2de78e1dc87b06 (DEFAULT_TABLE table=TestTable [id=9c5ceab7bff34b93a5855b748d9b0cb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 01:58:19.056727 18082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae27453d5aea4f648e2de78e1dc87b06. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:58:19.060935 18349 tablet_service.cc:1468] Processing CreateTablet for tablet ae27453d5aea4f648e2de78e1dc87b06 (DEFAULT_TABLE table=TestTable [id=9c5ceab7bff34b93a5855b748d9b0cb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 01:58:19.063009 18349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae27453d5aea4f648e2de78e1dc87b06. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:58:19.066547 18215 tablet_service.cc:1468] Processing CreateTablet for tablet ae27453d5aea4f648e2de78e1dc87b06 (DEFAULT_TABLE table=TestTable [id=9c5ceab7bff34b93a5855b748d9b0cb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 01:58:19.069037 18215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae27453d5aea4f648e2de78e1dc87b06. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:58:19.086789 18432 tablet_bootstrap.cc:492] T ae27453d5aea4f648e2de78e1dc87b06 P c07e3ae9d7b84044820f9e655e82556c: Bootstrap starting.
I20250626 01:58:19.089154 18433 tablet_bootstrap.cc:492] T ae27453d5aea4f648e2de78e1dc87b06 P d00e5cdd9be049cfb3d836852b287f2a: Bootstrap starting.
I20250626 01:58:19.096426 18432 tablet_bootstrap.cc:654] T ae27453d5aea4f648e2de78e1dc87b06 P c07e3ae9d7b84044820f9e655e82556c: Neither blocks nor log segments found. Creating new log.
I20250626 01:58:19.096952 18433 tablet_bootstrap.cc:654] T ae27453d5aea4f648e2de78e1dc87b06 P d00e5cdd9be049cfb3d836852b287f2a: Neither blocks nor log segments found. Creating new log.
I20250626 01:58:19.098062 18434 tablet_bootstrap.cc:492] T ae27453d5aea4f648e2de78e1dc87b06 P db76bdb9bbbe42e2a32c2598a6398f19: Bootstrap starting.
I20250626 01:58:19.099493 18433 log.cc:826] T ae27453d5aea4f648e2de78e1dc87b06 P d00e5cdd9be049cfb3d836852b287f2a: Log is configured to *not* fsync() on all Append() calls
I20250626 01:58:19.099494 18432 log.cc:826] T ae27453d5aea4f648e2de78e1dc87b06 P c07e3ae9d7b84044820f9e655e82556c: Log is configured to *not* fsync() on all Append() calls
I20250626 01:58:19.105443 18434 tablet_bootstrap.cc:654] T ae27453d5aea4f648e2de78e1dc87b06 P db76bdb9bbbe42e2a32c2598a6398f19: Neither blocks nor log segments found. Creating new log.
I20250626 01:58:19.105532 18432 tablet_bootstrap.cc:492] T ae27453d5aea4f648e2de78e1dc87b06 P c07e3ae9d7b84044820f9e655e82556c: No bootstrap required, opened a new log
I20250626 01:58:19.106092 18433 tablet_bootstrap.cc:492] T ae27453d5aea4f648e2de78e1dc87b06 P d00e5cdd9be049cfb3d836852b287f2a: No bootstrap required, opened a new log
I20250626 01:58:19.106104 18432 ts_tablet_manager.cc:1397] T ae27453d5aea4f648e2de78e1dc87b06 P c07e3ae9d7b84044820f9e655e82556c: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.006s
I20250626 01:58:19.106698 18433 ts_tablet_manager.cc:1397] T ae27453d5aea4f648e2de78e1dc87b06 P d00e5cdd9be049cfb3d836852b287f2a: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.004s
I20250626 01:58:19.108006 18434 log.cc:826] T ae27453d5aea4f648e2de78e1dc87b06 P db76bdb9bbbe42e2a32c2598a6398f19: Log is configured to *not* fsync() on all Append() calls
I20250626 01:58:19.114056 18434 tablet_bootstrap.cc:492] T ae27453d5aea4f648e2de78e1dc87b06 P db76bdb9bbbe42e2a32c2598a6398f19: No bootstrap required, opened a new log
I20250626 01:58:19.114626 18434 ts_tablet_manager.cc:1397] T ae27453d5aea4f648e2de78e1dc87b06 P db76bdb9bbbe42e2a32c2598a6398f19: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.004s
I20250626 01:58:19.126294 18432 raft_consensus.cc:357] T ae27453d5aea4f648e2de78e1dc87b06 P c07e3ae9d7b84044820f9e655e82556c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00e5cdd9be049cfb3d836852b287f2a" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 34023 } } peers { permanent_uuid: "c07e3ae9d7b84044820f9e655e82556c" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 45111 } } peers { permanent_uuid: "db76bdb9bbbe42e2a32c2598a6398f19" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 41169 } }
I20250626 01:58:19.127008 18432 raft_consensus.cc:383] T ae27453d5aea4f648e2de78e1dc87b06 P c07e3ae9d7b84044820f9e655e82556c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:58:19.127286 18432 raft_consensus.cc:738] T ae27453d5aea4f648e2de78e1dc87b06 P c07e3ae9d7b84044820f9e655e82556c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c07e3ae9d7b84044820f9e655e82556c, State: Initialized, Role: FOLLOWER
I20250626 01:58:19.128110 18432 consensus_queue.cc:260] T ae27453d5aea4f648e2de78e1dc87b06 P c07e3ae9d7b84044820f9e655e82556c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00e5cdd9be049cfb3d836852b287f2a" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 34023 } } peers { permanent_uuid: "c07e3ae9d7b84044820f9e655e82556c" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 45111 } } peers { permanent_uuid: "db76bdb9bbbe42e2a32c2598a6398f19" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 41169 } }
I20250626 01:58:19.131448 18414 heartbeater.cc:499] Master 127.16.19.254:36099 was elected leader, sending a full tablet report...
I20250626 01:58:19.132424 18432 ts_tablet_manager.cc:1428] T ae27453d5aea4f648e2de78e1dc87b06 P c07e3ae9d7b84044820f9e655e82556c: Time spent starting tablet: real 0.026s user 0.024s sys 0.001s
I20250626 01:58:19.134280 18433 raft_consensus.cc:357] T ae27453d5aea4f648e2de78e1dc87b06 P d00e5cdd9be049cfb3d836852b287f2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00e5cdd9be049cfb3d836852b287f2a" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 34023 } } peers { permanent_uuid: "c07e3ae9d7b84044820f9e655e82556c" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 45111 } } peers { permanent_uuid: "db76bdb9bbbe42e2a32c2598a6398f19" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 41169 } }
I20250626 01:58:19.135233 18433 raft_consensus.cc:383] T ae27453d5aea4f648e2de78e1dc87b06 P d00e5cdd9be049cfb3d836852b287f2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:58:19.135540 18433 raft_consensus.cc:738] T ae27453d5aea4f648e2de78e1dc87b06 P d00e5cdd9be049cfb3d836852b287f2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d00e5cdd9be049cfb3d836852b287f2a, State: Initialized, Role: FOLLOWER
I20250626 01:58:19.136441 18433 consensus_queue.cc:260] T ae27453d5aea4f648e2de78e1dc87b06 P d00e5cdd9be049cfb3d836852b287f2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00e5cdd9be049cfb3d836852b287f2a" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 34023 } } peers { permanent_uuid: "c07e3ae9d7b84044820f9e655e82556c" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 45111 } } peers { permanent_uuid: "db76bdb9bbbe42e2a32c2598a6398f19" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 41169 } }
I20250626 01:58:19.141112 18433 ts_tablet_manager.cc:1428] T ae27453d5aea4f648e2de78e1dc87b06 P d00e5cdd9be049cfb3d836852b287f2a: Time spent starting tablet: real 0.034s user 0.030s sys 0.004s
I20250626 01:58:19.142681 18434 raft_consensus.cc:357] T ae27453d5aea4f648e2de78e1dc87b06 P db76bdb9bbbe42e2a32c2598a6398f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00e5cdd9be049cfb3d836852b287f2a" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 34023 } } peers { permanent_uuid: "c07e3ae9d7b84044820f9e655e82556c" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 45111 } } peers { permanent_uuid: "db76bdb9bbbe42e2a32c2598a6398f19" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 41169 } }
I20250626 01:58:19.143595 18434 raft_consensus.cc:383] T ae27453d5aea4f648e2de78e1dc87b06 P db76bdb9bbbe42e2a32c2598a6398f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:58:19.144685 18434 raft_consensus.cc:738] T ae27453d5aea4f648e2de78e1dc87b06 P db76bdb9bbbe42e2a32c2598a6398f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db76bdb9bbbe42e2a32c2598a6398f19, State: Initialized, Role: FOLLOWER
I20250626 01:58:19.145816 18434 consensus_queue.cc:260] T ae27453d5aea4f648e2de78e1dc87b06 P db76bdb9bbbe42e2a32c2598a6398f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00e5cdd9be049cfb3d836852b287f2a" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 34023 } } peers { permanent_uuid: "c07e3ae9d7b84044820f9e655e82556c" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 45111 } } peers { permanent_uuid: "db76bdb9bbbe42e2a32c2598a6398f19" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 41169 } }
I20250626 01:58:19.149839 18434 ts_tablet_manager.cc:1428] T ae27453d5aea4f648e2de78e1dc87b06 P db76bdb9bbbe42e2a32c2598a6398f19: Time spent starting tablet: real 0.035s user 0.027s sys 0.008s
W20250626 01:58:19.168658 18415 tablet.cc:2378] T ae27453d5aea4f648e2de78e1dc87b06 P c07e3ae9d7b84044820f9e655e82556c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250626 01:58:19.176199 18148 tablet.cc:2378] T ae27453d5aea4f648e2de78e1dc87b06 P d00e5cdd9be049cfb3d836852b287f2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 01:58:19.199162 18440 raft_consensus.cc:491] T ae27453d5aea4f648e2de78e1dc87b06 P db76bdb9bbbe42e2a32c2598a6398f19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:58:19.199625 18440 raft_consensus.cc:513] T ae27453d5aea4f648e2de78e1dc87b06 P db76bdb9bbbe42e2a32c2598a6398f19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00e5cdd9be049cfb3d836852b287f2a" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 34023 } } peers { permanent_uuid: "c07e3ae9d7b84044820f9e655e82556c" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 45111 } } peers { permanent_uuid: "db76bdb9bbbe42e2a32c2598a6398f19" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 41169 } }
I20250626 01:58:19.202162 18440 leader_election.cc:290] T ae27453d5aea4f648e2de78e1dc87b06 P db76bdb9bbbe42e2a32c2598a6398f19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d00e5cdd9be049cfb3d836852b287f2a (127.16.19.193:34023), c07e3ae9d7b84044820f9e655e82556c (127.16.19.195:45111)
I20250626 01:58:19.213739 18102 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae27453d5aea4f648e2de78e1dc87b06" candidate_uuid: "db76bdb9bbbe42e2a32c2598a6398f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d00e5cdd9be049cfb3d836852b287f2a" is_pre_election: true
I20250626 01:58:19.214478 18102 raft_consensus.cc:2466] T ae27453d5aea4f648e2de78e1dc87b06 P d00e5cdd9be049cfb3d836852b287f2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db76bdb9bbbe42e2a32c2598a6398f19 in term 0.
I20250626 01:58:19.215404 18369 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae27453d5aea4f648e2de78e1dc87b06" candidate_uuid: "db76bdb9bbbe42e2a32c2598a6398f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c07e3ae9d7b84044820f9e655e82556c" is_pre_election: true
I20250626 01:58:19.215720 18171 leader_election.cc:304] T ae27453d5aea4f648e2de78e1dc87b06 P db76bdb9bbbe42e2a32c2598a6398f19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d00e5cdd9be049cfb3d836852b287f2a, db76bdb9bbbe42e2a32c2598a6398f19; no voters:
I20250626 01:58:19.216204 18369 raft_consensus.cc:2466] T ae27453d5aea4f648e2de78e1dc87b06 P c07e3ae9d7b84044820f9e655e82556c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db76bdb9bbbe42e2a32c2598a6398f19 in term 0.
I20250626 01:58:19.216691 18440 raft_consensus.cc:2802] T ae27453d5aea4f648e2de78e1dc87b06 P db76bdb9bbbe42e2a32c2598a6398f19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:58:19.216974 18440 raft_consensus.cc:491] T ae27453d5aea4f648e2de78e1dc87b06 P db76bdb9bbbe42e2a32c2598a6398f19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:58:19.217269 18440 raft_consensus.cc:3058] T ae27453d5aea4f648e2de78e1dc87b06 P db76bdb9bbbe42e2a32c2598a6398f19 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:58:19.221565 18440 raft_consensus.cc:513] T ae27453d5aea4f648e2de78e1dc87b06 P db76bdb9bbbe42e2a32c2598a6398f19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00e5cdd9be049cfb3d836852b287f2a" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 34023 } } peers { permanent_uuid: "c07e3ae9d7b84044820f9e655e82556c" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 45111 } } peers { permanent_uuid: "db76bdb9bbbe42e2a32c2598a6398f19" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 41169 } }
I20250626 01:58:19.222904 18440 leader_election.cc:290] T ae27453d5aea4f648e2de78e1dc87b06 P db76bdb9bbbe42e2a32c2598a6398f19 [CANDIDATE]: Term 1 election: Requested vote from peers d00e5cdd9be049cfb3d836852b287f2a (127.16.19.193:34023), c07e3ae9d7b84044820f9e655e82556c (127.16.19.195:45111)
I20250626 01:58:19.223753 18102 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae27453d5aea4f648e2de78e1dc87b06" candidate_uuid: "db76bdb9bbbe42e2a32c2598a6398f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d00e5cdd9be049cfb3d836852b287f2a"
I20250626 01:58:19.223985 18369 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae27453d5aea4f648e2de78e1dc87b06" candidate_uuid: "db76bdb9bbbe42e2a32c2598a6398f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c07e3ae9d7b84044820f9e655e82556c"
I20250626 01:58:19.224344 18102 raft_consensus.cc:3058] T ae27453d5aea4f648e2de78e1dc87b06 P d00e5cdd9be049cfb3d836852b287f2a [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:58:19.224511 18369 raft_consensus.cc:3058] T ae27453d5aea4f648e2de78e1dc87b06 P c07e3ae9d7b84044820f9e655e82556c [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:58:19.231482 18102 raft_consensus.cc:2466] T ae27453d5aea4f648e2de78e1dc87b06 P d00e5cdd9be049cfb3d836852b287f2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db76bdb9bbbe42e2a32c2598a6398f19 in term 1.
I20250626 01:58:19.231482 18369 raft_consensus.cc:2466] T ae27453d5aea4f648e2de78e1dc87b06 P c07e3ae9d7b84044820f9e655e82556c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db76bdb9bbbe42e2a32c2598a6398f19 in term 1.
I20250626 01:58:19.232551 18171 leader_election.cc:304] T ae27453d5aea4f648e2de78e1dc87b06 P db76bdb9bbbe42e2a32c2598a6398f19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d00e5cdd9be049cfb3d836852b287f2a, db76bdb9bbbe42e2a32c2598a6398f19; no voters:
I20250626 01:58:19.233212 18440 raft_consensus.cc:2802] T ae27453d5aea4f648e2de78e1dc87b06 P db76bdb9bbbe42e2a32c2598a6398f19 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:58:19.234786 18440 raft_consensus.cc:695] T ae27453d5aea4f648e2de78e1dc87b06 P db76bdb9bbbe42e2a32c2598a6398f19 [term 1 LEADER]: Becoming Leader. State: Replica: db76bdb9bbbe42e2a32c2598a6398f19, State: Running, Role: LEADER
I20250626 01:58:19.235533 18440 consensus_queue.cc:237] T ae27453d5aea4f648e2de78e1dc87b06 P db76bdb9bbbe42e2a32c2598a6398f19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00e5cdd9be049cfb3d836852b287f2a" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 34023 } } peers { permanent_uuid: "c07e3ae9d7b84044820f9e655e82556c" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 45111 } } peers { permanent_uuid: "db76bdb9bbbe42e2a32c2598a6398f19" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 41169 } }
I20250626 01:58:19.246896 17956 catalog_manager.cc:5582] T ae27453d5aea4f648e2de78e1dc87b06 P db76bdb9bbbe42e2a32c2598a6398f19 reported cstate change: term changed from 0 to 1, leader changed from <none> to db76bdb9bbbe42e2a32c2598a6398f19 (127.16.19.194). New cstate: current_term: 1 leader_uuid: "db76bdb9bbbe42e2a32c2598a6398f19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00e5cdd9be049cfb3d836852b287f2a" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 34023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c07e3ae9d7b84044820f9e655e82556c" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 45111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db76bdb9bbbe42e2a32c2598a6398f19" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 41169 } health_report { overall_health: HEALTHY } } }
W20250626 01:58:19.291033 18281 tablet.cc:2378] T ae27453d5aea4f648e2de78e1dc87b06 P db76bdb9bbbe42e2a32c2598a6398f19: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 01:58:19.295961 16463 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 01:58:19.299311 16463 ts_itest-base.cc:246] Waiting for 1 tablets on tserver d00e5cdd9be049cfb3d836852b287f2a to finish bootstrapping
I20250626 01:58:19.311996 16463 ts_itest-base.cc:246] Waiting for 1 tablets on tserver db76bdb9bbbe42e2a32c2598a6398f19 to finish bootstrapping
I20250626 01:58:19.323493 16463 ts_itest-base.cc:246] Waiting for 1 tablets on tserver c07e3ae9d7b84044820f9e655e82556c to finish bootstrapping
I20250626 01:58:19.337617 17956 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44008:
name: "TestAnotherTable"
schema {
columns {
name: "foo"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "bar"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
comment: "comment for bar"
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "foo"
}
}
}
W20250626 01:58:19.339241 17956 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAnotherTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 01:58:19.356379 18082 tablet_service.cc:1468] Processing CreateTablet for tablet 26cf3d3b86cb412c822e99f2d9bd796c (DEFAULT_TABLE table=TestAnotherTable [id=b24833265cdc47e184149d155666d5d1]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250626 01:58:19.356833 18215 tablet_service.cc:1468] Processing CreateTablet for tablet 26cf3d3b86cb412c822e99f2d9bd796c (DEFAULT_TABLE table=TestAnotherTable [id=b24833265cdc47e184149d155666d5d1]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250626 01:58:19.357555 18082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26cf3d3b86cb412c822e99f2d9bd796c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:58:19.357877 18215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26cf3d3b86cb412c822e99f2d9bd796c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:58:19.357873 18349 tablet_service.cc:1468] Processing CreateTablet for tablet 26cf3d3b86cb412c822e99f2d9bd796c (DEFAULT_TABLE table=TestAnotherTable [id=b24833265cdc47e184149d155666d5d1]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250626 01:58:19.359014 18349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26cf3d3b86cb412c822e99f2d9bd796c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:58:19.369621 18432 tablet_bootstrap.cc:492] T 26cf3d3b86cb412c822e99f2d9bd796c P c07e3ae9d7b84044820f9e655e82556c: Bootstrap starting.
I20250626 01:58:19.372365 18434 tablet_bootstrap.cc:492] T 26cf3d3b86cb412c822e99f2d9bd796c P db76bdb9bbbe42e2a32c2598a6398f19: Bootstrap starting.
I20250626 01:58:19.372373 18433 tablet_bootstrap.cc:492] T 26cf3d3b86cb412c822e99f2d9bd796c P d00e5cdd9be049cfb3d836852b287f2a: Bootstrap starting.
I20250626 01:58:19.376058 18432 tablet_bootstrap.cc:654] T 26cf3d3b86cb412c822e99f2d9bd796c P c07e3ae9d7b84044820f9e655e82556c: Neither blocks nor log segments found. Creating new log.
I20250626 01:58:19.378782 18434 tablet_bootstrap.cc:654] T 26cf3d3b86cb412c822e99f2d9bd796c P db76bdb9bbbe42e2a32c2598a6398f19: Neither blocks nor log segments found. Creating new log.
I20250626 01:58:19.378758 18433 tablet_bootstrap.cc:654] T 26cf3d3b86cb412c822e99f2d9bd796c P d00e5cdd9be049cfb3d836852b287f2a: Neither blocks nor log segments found. Creating new log.
I20250626 01:58:19.381430 18432 tablet_bootstrap.cc:492] T 26cf3d3b86cb412c822e99f2d9bd796c P c07e3ae9d7b84044820f9e655e82556c: No bootstrap required, opened a new log
I20250626 01:58:19.381896 18432 ts_tablet_manager.cc:1397] T 26cf3d3b86cb412c822e99f2d9bd796c P c07e3ae9d7b84044820f9e655e82556c: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.005s
I20250626 01:58:19.385874 18432 raft_consensus.cc:357] T 26cf3d3b86cb412c822e99f2d9bd796c P c07e3ae9d7b84044820f9e655e82556c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00e5cdd9be049cfb3d836852b287f2a" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 34023 } } peers { permanent_uuid: "db76bdb9bbbe42e2a32c2598a6398f19" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 41169 } } peers { permanent_uuid: "c07e3ae9d7b84044820f9e655e82556c" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 45111 } }
I20250626 01:58:19.386492 18432 raft_consensus.cc:383] T 26cf3d3b86cb412c822e99f2d9bd796c P c07e3ae9d7b84044820f9e655e82556c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:58:19.386777 18432 raft_consensus.cc:738] T 26cf3d3b86cb412c822e99f2d9bd796c P c07e3ae9d7b84044820f9e655e82556c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c07e3ae9d7b84044820f9e655e82556c, State: Initialized, Role: FOLLOWER
I20250626 01:58:19.387053 18433 tablet_bootstrap.cc:492] T 26cf3d3b86cb412c822e99f2d9bd796c P d00e5cdd9be049cfb3d836852b287f2a: No bootstrap required, opened a new log
I20250626 01:58:19.387426 18433 ts_tablet_manager.cc:1397] T 26cf3d3b86cb412c822e99f2d9bd796c P d00e5cdd9be049cfb3d836852b287f2a: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.003s
I20250626 01:58:19.387779 18434 tablet_bootstrap.cc:492] T 26cf3d3b86cb412c822e99f2d9bd796c P db76bdb9bbbe42e2a32c2598a6398f19: No bootstrap required, opened a new log
I20250626 01:58:19.387614 18432 consensus_queue.cc:260] T 26cf3d3b86cb412c822e99f2d9bd796c P c07e3ae9d7b84044820f9e655e82556c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00e5cdd9be049cfb3d836852b287f2a" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 34023 } } peers { permanent_uuid: "db76bdb9bbbe42e2a32c2598a6398f19" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 41169 } } peers { permanent_uuid: "c07e3ae9d7b84044820f9e655e82556c" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 45111 } }
I20250626 01:58:19.388311 18434 ts_tablet_manager.cc:1397] T 26cf3d3b86cb412c822e99f2d9bd796c P db76bdb9bbbe42e2a32c2598a6398f19: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.003s
I20250626 01:58:19.389995 18432 ts_tablet_manager.cc:1428] T 26cf3d3b86cb412c822e99f2d9bd796c P c07e3ae9d7b84044820f9e655e82556c: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20250626 01:58:19.389927 18433 raft_consensus.cc:357] T 26cf3d3b86cb412c822e99f2d9bd796c P d00e5cdd9be049cfb3d836852b287f2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00e5cdd9be049cfb3d836852b287f2a" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 34023 } } peers { permanent_uuid: "db76bdb9bbbe42e2a32c2598a6398f19" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 41169 } } peers { permanent_uuid: "c07e3ae9d7b84044820f9e655e82556c" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 45111 } }
I20250626 01:58:19.390661 18433 raft_consensus.cc:383] T 26cf3d3b86cb412c822e99f2d9bd796c P d00e5cdd9be049cfb3d836852b287f2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:58:19.390974 18433 raft_consensus.cc:738] T 26cf3d3b86cb412c822e99f2d9bd796c P d00e5cdd9be049cfb3d836852b287f2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d00e5cdd9be049cfb3d836852b287f2a, State: Initialized, Role: FOLLOWER
I20250626 01:58:19.391804 18433 consensus_queue.cc:260] T 26cf3d3b86cb412c822e99f2d9bd796c P d00e5cdd9be049cfb3d836852b287f2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00e5cdd9be049cfb3d836852b287f2a" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 34023 } } peers { permanent_uuid: "db76bdb9bbbe42e2a32c2598a6398f19" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 41169 } } peers { permanent_uuid: "c07e3ae9d7b84044820f9e655e82556c" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 45111 } }
I20250626 01:58:19.392546 18434 raft_consensus.cc:357] T 26cf3d3b86cb412c822e99f2d9bd796c P db76bdb9bbbe42e2a32c2598a6398f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00e5cdd9be049cfb3d836852b287f2a" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 34023 } } peers { permanent_uuid: "db76bdb9bbbe42e2a32c2598a6398f19" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 41169 } } peers { permanent_uuid: "c07e3ae9d7b84044820f9e655e82556c" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 45111 } }
I20250626 01:58:19.393292 18434 raft_consensus.cc:383] T 26cf3d3b86cb412c822e99f2d9bd796c P db76bdb9bbbe42e2a32c2598a6398f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:58:19.393518 18434 raft_consensus.cc:738] T 26cf3d3b86cb412c822e99f2d9bd796c P db76bdb9bbbe42e2a32c2598a6398f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db76bdb9bbbe42e2a32c2598a6398f19, State: Initialized, Role: FOLLOWER
I20250626 01:58:19.394301 18434 consensus_queue.cc:260] T 26cf3d3b86cb412c822e99f2d9bd796c P db76bdb9bbbe42e2a32c2598a6398f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00e5cdd9be049cfb3d836852b287f2a" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 34023 } } peers { permanent_uuid: "db76bdb9bbbe42e2a32c2598a6398f19" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 41169 } } peers { permanent_uuid: "c07e3ae9d7b84044820f9e655e82556c" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 45111 } }
I20250626 01:58:19.394912 18433 ts_tablet_manager.cc:1428] T 26cf3d3b86cb412c822e99f2d9bd796c P d00e5cdd9be049cfb3d836852b287f2a: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20250626 01:58:19.396591 18434 ts_tablet_manager.cc:1428] T 26cf3d3b86cb412c822e99f2d9bd796c P db76bdb9bbbe42e2a32c2598a6398f19: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20250626 01:58:19.543429 18439 raft_consensus.cc:491] T 26cf3d3b86cb412c822e99f2d9bd796c P d00e5cdd9be049cfb3d836852b287f2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:58:19.543867 18439 raft_consensus.cc:513] T 26cf3d3b86cb412c822e99f2d9bd796c P d00e5cdd9be049cfb3d836852b287f2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00e5cdd9be049cfb3d836852b287f2a" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 34023 } } peers { permanent_uuid: "db76bdb9bbbe42e2a32c2598a6398f19" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 41169 } } peers { permanent_uuid: "c07e3ae9d7b84044820f9e655e82556c" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 45111 } }
I20250626 01:58:19.546327 18439 leader_election.cc:290] T 26cf3d3b86cb412c822e99f2d9bd796c P d00e5cdd9be049cfb3d836852b287f2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db76bdb9bbbe42e2a32c2598a6398f19 (127.16.19.194:41169), c07e3ae9d7b84044820f9e655e82556c (127.16.19.195:45111)
I20250626 01:58:19.558597 18235 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cf3d3b86cb412c822e99f2d9bd796c" candidate_uuid: "d00e5cdd9be049cfb3d836852b287f2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db76bdb9bbbe42e2a32c2598a6398f19" is_pre_election: true
I20250626 01:58:19.559295 18235 raft_consensus.cc:2466] T 26cf3d3b86cb412c822e99f2d9bd796c P db76bdb9bbbe42e2a32c2598a6398f19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d00e5cdd9be049cfb3d836852b287f2a in term 0.
I20250626 01:58:19.560170 18369 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cf3d3b86cb412c822e99f2d9bd796c" candidate_uuid: "d00e5cdd9be049cfb3d836852b287f2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c07e3ae9d7b84044820f9e655e82556c" is_pre_election: true
I20250626 01:58:19.560477 18038 leader_election.cc:304] T 26cf3d3b86cb412c822e99f2d9bd796c P d00e5cdd9be049cfb3d836852b287f2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d00e5cdd9be049cfb3d836852b287f2a, db76bdb9bbbe42e2a32c2598a6398f19; no voters:
I20250626 01:58:19.560809 18369 raft_consensus.cc:2466] T 26cf3d3b86cb412c822e99f2d9bd796c P c07e3ae9d7b84044820f9e655e82556c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d00e5cdd9be049cfb3d836852b287f2a in term 0.
I20250626 01:58:19.561529 18439 raft_consensus.cc:2802] T 26cf3d3b86cb412c822e99f2d9bd796c P d00e5cdd9be049cfb3d836852b287f2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:58:19.561892 18439 raft_consensus.cc:491] T 26cf3d3b86cb412c822e99f2d9bd796c P d00e5cdd9be049cfb3d836852b287f2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:58:19.562168 18439 raft_consensus.cc:3058] T 26cf3d3b86cb412c822e99f2d9bd796c P d00e5cdd9be049cfb3d836852b287f2a [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:58:19.566463 18439 raft_consensus.cc:513] T 26cf3d3b86cb412c822e99f2d9bd796c P d00e5cdd9be049cfb3d836852b287f2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00e5cdd9be049cfb3d836852b287f2a" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 34023 } } peers { permanent_uuid: "db76bdb9bbbe42e2a32c2598a6398f19" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 41169 } } peers { permanent_uuid: "c07e3ae9d7b84044820f9e655e82556c" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 45111 } }
I20250626 01:58:19.567822 18439 leader_election.cc:290] T 26cf3d3b86cb412c822e99f2d9bd796c P d00e5cdd9be049cfb3d836852b287f2a [CANDIDATE]: Term 1 election: Requested vote from peers db76bdb9bbbe42e2a32c2598a6398f19 (127.16.19.194:41169), c07e3ae9d7b84044820f9e655e82556c (127.16.19.195:45111)
I20250626 01:58:19.568768 18235 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cf3d3b86cb412c822e99f2d9bd796c" candidate_uuid: "d00e5cdd9be049cfb3d836852b287f2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db76bdb9bbbe42e2a32c2598a6398f19"
I20250626 01:58:19.568959 18369 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cf3d3b86cb412c822e99f2d9bd796c" candidate_uuid: "d00e5cdd9be049cfb3d836852b287f2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c07e3ae9d7b84044820f9e655e82556c"
I20250626 01:58:19.569521 18235 raft_consensus.cc:3058] T 26cf3d3b86cb412c822e99f2d9bd796c P db76bdb9bbbe42e2a32c2598a6398f19 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:58:19.569586 18369 raft_consensus.cc:3058] T 26cf3d3b86cb412c822e99f2d9bd796c P c07e3ae9d7b84044820f9e655e82556c [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:58:19.576224 18369 raft_consensus.cc:2466] T 26cf3d3b86cb412c822e99f2d9bd796c P c07e3ae9d7b84044820f9e655e82556c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d00e5cdd9be049cfb3d836852b287f2a in term 1.
I20250626 01:58:19.576224 18235 raft_consensus.cc:2466] T 26cf3d3b86cb412c822e99f2d9bd796c P db76bdb9bbbe42e2a32c2598a6398f19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d00e5cdd9be049cfb3d836852b287f2a in term 1.
I20250626 01:58:19.577323 18038 leader_election.cc:304] T 26cf3d3b86cb412c822e99f2d9bd796c P d00e5cdd9be049cfb3d836852b287f2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c07e3ae9d7b84044820f9e655e82556c, d00e5cdd9be049cfb3d836852b287f2a; no voters:
I20250626 01:58:19.577956 18439 raft_consensus.cc:2802] T 26cf3d3b86cb412c822e99f2d9bd796c P d00e5cdd9be049cfb3d836852b287f2a [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:58:19.579895 18439 raft_consensus.cc:695] T 26cf3d3b86cb412c822e99f2d9bd796c P d00e5cdd9be049cfb3d836852b287f2a [term 1 LEADER]: Becoming Leader. State: Replica: d00e5cdd9be049cfb3d836852b287f2a, State: Running, Role: LEADER
I20250626 01:58:19.580812 18439 consensus_queue.cc:237] T 26cf3d3b86cb412c822e99f2d9bd796c P d00e5cdd9be049cfb3d836852b287f2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00e5cdd9be049cfb3d836852b287f2a" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 34023 } } peers { permanent_uuid: "db76bdb9bbbe42e2a32c2598a6398f19" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 41169 } } peers { permanent_uuid: "c07e3ae9d7b84044820f9e655e82556c" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 45111 } }
I20250626 01:58:19.591639 17956 catalog_manager.cc:5582] T 26cf3d3b86cb412c822e99f2d9bd796c P d00e5cdd9be049cfb3d836852b287f2a reported cstate change: term changed from 0 to 1, leader changed from <none> to d00e5cdd9be049cfb3d836852b287f2a (127.16.19.193). New cstate: current_term: 1 leader_uuid: "d00e5cdd9be049cfb3d836852b287f2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00e5cdd9be049cfb3d836852b287f2a" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 34023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db76bdb9bbbe42e2a32c2598a6398f19" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 41169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c07e3ae9d7b84044820f9e655e82556c" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 45111 } health_report { overall_health: UNKNOWN } } }
I20250626 01:58:19.723043 18440 consensus_queue.cc:1035] T ae27453d5aea4f648e2de78e1dc87b06 P db76bdb9bbbe42e2a32c2598a6398f19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07e3ae9d7b84044820f9e655e82556c" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 45111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 01:58:19.741844 18443 consensus_queue.cc:1035] T ae27453d5aea4f648e2de78e1dc87b06 P db76bdb9bbbe42e2a32c2598a6398f19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d00e5cdd9be049cfb3d836852b287f2a" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 34023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250626 01:58:19.972586 18453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:58:19.973249 18453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250626 01:58:20.003129 18439 consensus_queue.cc:1035] T 26cf3d3b86cb412c822e99f2d9bd796c P d00e5cdd9be049cfb3d836852b287f2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07e3ae9d7b84044820f9e655e82556c" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 45111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 01:58:20.008791 18453 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250626 01:58:20.017531 18451 consensus_queue.cc:1035] T 26cf3d3b86cb412c822e99f2d9bd796c P d00e5cdd9be049cfb3d836852b287f2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db76bdb9bbbe42e2a32c2598a6398f19" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 41169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 01:58:21.337936 18453 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.270s user 0.494s sys 0.775s
W20250626 01:58:21.338251 18453 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.270s user 0.494s sys 0.775s
W20250626 01:58:22.442752 18276 debug-util.cc:398] Leaking SignalData structure 0x7b08000ac540 after lost signal to thread 18152
W20250626 01:58:22.443591 18276 debug-util.cc:398] Leaking SignalData structure 0x7b08000c5e80 after lost signal to thread 18279
W20250626 01:58:22.939994 18476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:58:22.940937 18476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:58:23.008301 18476 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250626 01:58:24.466970 18481 debug-util.cc:398] Leaking SignalData structure 0x7b0800036040 after lost signal to thread 18476
W20250626 01:58:24.467692 18481 kernel_stack_watchdog.cc:198] Thread 18476 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:641 for 404ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250626 01:58:24.631557 18476 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.571s user 0.561s sys 0.951s
W20250626 01:58:24.807957 18476 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.747s user 0.564s sys 0.962s
W20250626 01:58:26.198585 18491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:58:26.199124 18491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:58:26.230499 18491 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250626 01:58:27.514070 18491 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.241s user 0.477s sys 0.762s
W20250626 01:58:27.514477 18491 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.241s user 0.477s sys 0.762s
W20250626 01:58:28.906096 18506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:58:28.906986 18506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:58:28.950881 18506 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250626 01:58:30.316677 18506 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.323s user 0.510s sys 0.809s
W20250626 01:58:30.317106 18506 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.324s user 0.513s sys 0.809s
I20250626 01:58:31.387293 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 18018
I20250626 01:58:31.417250 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 18151
I20250626 01:58:31.446144 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 18284
I20250626 01:58:31.471905 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 17925
2025-06-26T01:58:31Z chronyd exiting
[ OK ] AdminCliTest.TestDescribeTableColumnFlags (20315 ms)
[ RUN ] AdminCliTest.TestAuthzResetCacheNotAuthorized
I20250626 01:58:31.535123 16463 test_util.cc:276] Using random seed: -436221450
I20250626 01:58:31.539323 16463 ts_itest-base.cc:115] Starting cluster with:
I20250626 01:58:31.539512 16463 ts_itest-base.cc:116] --------------
I20250626 01:58:31.539666 16463 ts_itest-base.cc:117] 3 tablet servers
I20250626 01:58:31.539801 16463 ts_itest-base.cc:118] 3 replicas per TS
I20250626 01:58:31.539942 16463 ts_itest-base.cc:119] --------------
2025-06-26T01:58:31Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-26T01:58:31Z Disabled control of system clock
I20250626 01:58:31.577244 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.19.254:33149
--webserver_interface=127.16.19.254
--webserver_port=0
--builtin_ntp_servers=127.16.19.212:45271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.19.254:33149
--superuser_acl=no-such-user with env {}
W20250626 01:58:31.876396 18526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:58:31.877002 18526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 01:58:31.877475 18526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:58:31.909150 18526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 01:58:31.909474 18526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 01:58:31.909683 18526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 01:58:31.909879 18526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 01:58:31.946813 18526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:45271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.19.254:33149
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.19.254:33149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.19.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 01:58:31.948076 18526 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 01:58:31.950170 18526 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:58:31.967128 18533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:58:31.967130 18532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:58:31.967428 18535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:58:33.147425 18534 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 01:58:33.147518 18526 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 01:58:33.152002 18526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 01:58:33.155118 18526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 01:58:33.156548 18526 hybrid_clock.cc:648] HybridClock initialized: now 1750903113156487 us; error 75 us; skew 500 ppm
I20250626 01:58:33.157406 18526 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:58:33.169030 18526 webserver.cc:469] Webserver started at http://127.16.19.254:38437/ using document root <none> and password file <none>
I20250626 01:58:33.169972 18526 fs_manager.cc:362] Metadata directory not provided
I20250626 01:58:33.170166 18526 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:58:33.170596 18526 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:58:33.175019 18526 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "c3fe8f92566e4c9ba673e1f14d0712d3"
format_stamp: "Formatted at 2025-06-26 01:58:33 on dist-test-slave-h4rq"
I20250626 01:58:33.176075 18526 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "c3fe8f92566e4c9ba673e1f14d0712d3"
format_stamp: "Formatted at 2025-06-26 01:58:33 on dist-test-slave-h4rq"
I20250626 01:58:33.183254 18526 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20250626 01:58:33.188881 18542 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:58:33.189916 18526 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250626 01:58:33.190222 18526 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
uuid: "c3fe8f92566e4c9ba673e1f14d0712d3"
format_stamp: "Formatted at 2025-06-26 01:58:33 on dist-test-slave-h4rq"
I20250626 01:58:33.190526 18526 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:58:33.258678 18526 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:58:33.260231 18526 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 01:58:33.260689 18526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:58:33.330582 18526 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.254:33149
I20250626 01:58:33.330646 18593 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.254:33149 every 8 connection(s)
I20250626 01:58:33.333374 18526 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250626 01:58:33.338662 18594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:58:33.341339 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 18526
I20250626 01:58:33.341729 16463 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250626 01:58:33.359903 18594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3fe8f92566e4c9ba673e1f14d0712d3: Bootstrap starting.
I20250626 01:58:33.366153 18594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3fe8f92566e4c9ba673e1f14d0712d3: Neither blocks nor log segments found. Creating new log.
I20250626 01:58:33.367854 18594 log.cc:826] T 00000000000000000000000000000000 P c3fe8f92566e4c9ba673e1f14d0712d3: Log is configured to *not* fsync() on all Append() calls
I20250626 01:58:33.372503 18594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3fe8f92566e4c9ba673e1f14d0712d3: No bootstrap required, opened a new log
I20250626 01:58:33.390362 18594 raft_consensus.cc:357] T 00000000000000000000000000000000 P c3fe8f92566e4c9ba673e1f14d0712d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3fe8f92566e4c9ba673e1f14d0712d3" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 33149 } }
I20250626 01:58:33.391148 18594 raft_consensus.cc:383] T 00000000000000000000000000000000 P c3fe8f92566e4c9ba673e1f14d0712d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:58:33.391386 18594 raft_consensus.cc:738] T 00000000000000000000000000000000 P c3fe8f92566e4c9ba673e1f14d0712d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3fe8f92566e4c9ba673e1f14d0712d3, State: Initialized, Role: FOLLOWER
I20250626 01:58:33.392089 18594 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3fe8f92566e4c9ba673e1f14d0712d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3fe8f92566e4c9ba673e1f14d0712d3" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 33149 } }
I20250626 01:58:33.392602 18594 raft_consensus.cc:397] T 00000000000000000000000000000000 P c3fe8f92566e4c9ba673e1f14d0712d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 01:58:33.392911 18594 raft_consensus.cc:491] T 00000000000000000000000000000000 P c3fe8f92566e4c9ba673e1f14d0712d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 01:58:33.393198 18594 raft_consensus.cc:3058] T 00000000000000000000000000000000 P c3fe8f92566e4c9ba673e1f14d0712d3 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:58:33.397260 18594 raft_consensus.cc:513] T 00000000000000000000000000000000 P c3fe8f92566e4c9ba673e1f14d0712d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3fe8f92566e4c9ba673e1f14d0712d3" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 33149 } }
I20250626 01:58:33.397984 18594 leader_election.cc:304] T 00000000000000000000000000000000 P c3fe8f92566e4c9ba673e1f14d0712d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3fe8f92566e4c9ba673e1f14d0712d3; no voters:
I20250626 01:58:33.399857 18594 leader_election.cc:290] T 00000000000000000000000000000000 P c3fe8f92566e4c9ba673e1f14d0712d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 01:58:33.400508 18599 raft_consensus.cc:2802] T 00000000000000000000000000000000 P c3fe8f92566e4c9ba673e1f14d0712d3 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:58:33.402746 18599 raft_consensus.cc:695] T 00000000000000000000000000000000 P c3fe8f92566e4c9ba673e1f14d0712d3 [term 1 LEADER]: Becoming Leader. State: Replica: c3fe8f92566e4c9ba673e1f14d0712d3, State: Running, Role: LEADER
I20250626 01:58:33.403492 18599 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3fe8f92566e4c9ba673e1f14d0712d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3fe8f92566e4c9ba673e1f14d0712d3" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 33149 } }
I20250626 01:58:33.404480 18594 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3fe8f92566e4c9ba673e1f14d0712d3 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 01:58:33.414654 18601 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3fe8f92566e4c9ba673e1f14d0712d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3fe8f92566e4c9ba673e1f14d0712d3. Latest consensus state: current_term: 1 leader_uuid: "c3fe8f92566e4c9ba673e1f14d0712d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3fe8f92566e4c9ba673e1f14d0712d3" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 33149 } } }
I20250626 01:58:33.415436 18601 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3fe8f92566e4c9ba673e1f14d0712d3 [sys.catalog]: This master's current role is: LEADER
I20250626 01:58:33.414659 18600 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3fe8f92566e4c9ba673e1f14d0712d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3fe8f92566e4c9ba673e1f14d0712d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3fe8f92566e4c9ba673e1f14d0712d3" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 33149 } } }
I20250626 01:58:33.415819 18600 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3fe8f92566e4c9ba673e1f14d0712d3 [sys.catalog]: This master's current role is: LEADER
I20250626 01:58:33.419265 18607 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 01:58:33.430655 18607 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 01:58:33.447710 18607 catalog_manager.cc:1349] Generated new cluster ID: b0b4a99066844b5a827387e562c7ec48
I20250626 01:58:33.447969 18607 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 01:58:33.467295 18607 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 01:58:33.469334 18607 catalog_manager.cc:1506] Loading token signing keys...
I20250626 01:58:33.493741 18607 catalog_manager.cc:5955] T 00000000000000000000000000000000 P c3fe8f92566e4c9ba673e1f14d0712d3: Generated new TSK 0
I20250626 01:58:33.494925 18607 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 01:58:33.512959 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.19.193:0
--local_ip_for_outbound_sockets=127.16.19.193
--webserver_interface=127.16.19.193
--webserver_port=0
--tserver_master_addrs=127.16.19.254:33149
--builtin_ntp_servers=127.16.19.212:45271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250626 01:58:33.834791 18618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:58:33.835325 18618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 01:58:33.835825 18618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:58:33.868249 18618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 01:58:33.869661 18618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.19.193
I20250626 01:58:33.905613 18618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:45271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.19.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.19.193
--webserver_port=0
--tserver_master_addrs=127.16.19.254:33149
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.19.193
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 01:58:33.907235 18618 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 01:58:33.909026 18618 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:58:33.926988 18625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:58:33.927512 18624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:58:33.929246 18618 server_base.cc:1048] running on GCE node
W20250626 01:58:33.928361 18627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:58:35.126161 18618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 01:58:35.128921 18618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 01:58:35.130394 18618 hybrid_clock.cc:648] HybridClock initialized: now 1750903115130327 us; error 79 us; skew 500 ppm
I20250626 01:58:35.131291 18618 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:58:35.139065 18618 webserver.cc:469] Webserver started at http://127.16.19.193:32983/ using document root <none> and password file <none>
I20250626 01:58:35.140035 18618 fs_manager.cc:362] Metadata directory not provided
I20250626 01:58:35.140259 18618 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:58:35.140796 18618 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:58:35.145551 18618 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "eabb181d28354629b61e387eecb73943"
format_stamp: "Formatted at 2025-06-26 01:58:35 on dist-test-slave-h4rq"
I20250626 01:58:35.146749 18618 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "eabb181d28354629b61e387eecb73943"
format_stamp: "Formatted at 2025-06-26 01:58:35 on dist-test-slave-h4rq"
I20250626 01:58:35.154973 18618 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.003s
I20250626 01:58:35.161408 18634 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:58:35.162600 18618 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250626 01:58:35.162931 18618 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "eabb181d28354629b61e387eecb73943"
format_stamp: "Formatted at 2025-06-26 01:58:35 on dist-test-slave-h4rq"
I20250626 01:58:35.163269 18618 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:58:35.214826 18618 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:58:35.216368 18618 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 01:58:35.216816 18618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:58:35.219282 18618 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:58:35.223387 18618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:58:35.223702 18618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250626 01:58:35.223918 18618 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:58:35.224118 18618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:58:35.361044 18618 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.193:37393
I20250626 01:58:35.361167 18746 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.193:37393 every 8 connection(s)
I20250626 01:58:35.363554 18618 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250626 01:58:35.369402 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 18618
I20250626 01:58:35.369814 16463 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250626 01:58:35.375994 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.19.194:0
--local_ip_for_outbound_sockets=127.16.19.194
--webserver_interface=127.16.19.194
--webserver_port=0
--tserver_master_addrs=127.16.19.254:33149
--builtin_ntp_servers=127.16.19.212:45271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 01:58:35.385077 18747 heartbeater.cc:344] Connected to a master server at 127.16.19.254:33149
I20250626 01:58:35.385505 18747 heartbeater.cc:461] Registering TS with master...
I20250626 01:58:35.386494 18747 heartbeater.cc:507] Master 127.16.19.254:33149 requested a full tablet report, sending...
I20250626 01:58:35.388859 18559 ts_manager.cc:194] Registered new tserver with Master: eabb181d28354629b61e387eecb73943 (127.16.19.193:37393)
I20250626 01:58:35.390744 18559 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.19.193:60405
W20250626 01:58:35.672602 18751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:58:35.673146 18751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 01:58:35.673648 18751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:58:35.704957 18751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 01:58:35.705924 18751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.19.194
I20250626 01:58:35.741112 18751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:45271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.19.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.16.19.194
--webserver_port=0
--tserver_master_addrs=127.16.19.254:33149
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.19.194
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 01:58:35.742470 18751 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 01:58:35.744123 18751 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:58:35.758829 18757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:58:36.394169 18747 heartbeater.cc:499] Master 127.16.19.254:33149 was elected leader, sending a full tablet report...
W20250626 01:58:35.758904 18758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:58:36.954110 18760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:58:36.955924 18759 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250626 01:58:36.956849 18751 thread.cc:640] OpenStack (cloud detector) Time spent creating pthread: real 1.200s user 0.425s sys 0.773s
W20250626 01:58:36.957120 18751 thread.cc:606] OpenStack (cloud detector) Time spent starting thread: real 1.201s user 0.425s sys 0.773s
I20250626 01:58:36.957319 18751 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 01:58:36.958423 18751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 01:58:36.960803 18751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 01:58:36.962159 18751 hybrid_clock.cc:648] HybridClock initialized: now 1750903116962130 us; error 47 us; skew 500 ppm
I20250626 01:58:36.962978 18751 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:58:36.971082 18751 webserver.cc:469] Webserver started at http://127.16.19.194:36407/ using document root <none> and password file <none>
I20250626 01:58:36.972378 18751 fs_manager.cc:362] Metadata directory not provided
I20250626 01:58:36.972613 18751 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:58:36.973094 18751 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:58:36.977578 18751 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "972d6bc25d644b52bc2d36bace6943da"
format_stamp: "Formatted at 2025-06-26 01:58:36 on dist-test-slave-h4rq"
I20250626 01:58:36.978703 18751 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "972d6bc25d644b52bc2d36bace6943da"
format_stamp: "Formatted at 2025-06-26 01:58:36 on dist-test-slave-h4rq"
I20250626 01:58:36.986464 18751 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20250626 01:58:36.992779 18767 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:58:36.993898 18751 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250626 01:58:36.994199 18751 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "972d6bc25d644b52bc2d36bace6943da"
format_stamp: "Formatted at 2025-06-26 01:58:36 on dist-test-slave-h4rq"
I20250626 01:58:36.994527 18751 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:58:37.064796 18751 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:58:37.066252 18751 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 01:58:37.066679 18751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:58:37.069130 18751 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:58:37.073252 18751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:58:37.073457 18751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:58:37.073722 18751 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:58:37.073868 18751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:58:37.208264 18751 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.194:44779
I20250626 01:58:37.208367 18879 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.194:44779 every 8 connection(s)
I20250626 01:58:37.210862 18751 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250626 01:58:37.215229 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 18751
I20250626 01:58:37.215801 16463 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250626 01:58:37.221638 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.19.195:0
--local_ip_for_outbound_sockets=127.16.19.195
--webserver_interface=127.16.19.195
--webserver_port=0
--tserver_master_addrs=127.16.19.254:33149
--builtin_ntp_servers=127.16.19.212:45271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 01:58:37.232774 18880 heartbeater.cc:344] Connected to a master server at 127.16.19.254:33149
I20250626 01:58:37.233206 18880 heartbeater.cc:461] Registering TS with master...
I20250626 01:58:37.234233 18880 heartbeater.cc:507] Master 127.16.19.254:33149 requested a full tablet report, sending...
I20250626 01:58:37.236678 18559 ts_manager.cc:194] Registered new tserver with Master: 972d6bc25d644b52bc2d36bace6943da (127.16.19.194:44779)
I20250626 01:58:37.238675 18559 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.19.194:58483
W20250626 01:58:37.529744 18884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:58:37.530253 18884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 01:58:37.530759 18884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:58:37.562734 18884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 01:58:37.563623 18884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.19.195
I20250626 01:58:37.598102 18884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:45271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.19.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.16.19.195
--webserver_port=0
--tserver_master_addrs=127.16.19.254:33149
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.19.195
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 01:58:37.599481 18884 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 01:58:37.601110 18884 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:58:37.618060 18893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:58:38.242364 18880 heartbeater.cc:499] Master 127.16.19.254:33149 was elected leader, sending a full tablet report...
W20250626 01:58:37.618117 18891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:58:37.618594 18890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:58:38.790776 18892 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 01:58:38.790923 18884 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 01:58:38.795081 18884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 01:58:38.797499 18884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 01:58:38.798879 18884 hybrid_clock.cc:648] HybridClock initialized: now 1750903118798828 us; error 70 us; skew 500 ppm
I20250626 01:58:38.799712 18884 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:58:38.811172 18884 webserver.cc:469] Webserver started at http://127.16.19.195:34289/ using document root <none> and password file <none>
I20250626 01:58:38.812131 18884 fs_manager.cc:362] Metadata directory not provided
I20250626 01:58:38.812354 18884 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:58:38.812865 18884 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:58:38.818924 18884 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "e3cd4ea12b534ace8a58080372af5e48"
format_stamp: "Formatted at 2025-06-26 01:58:38 on dist-test-slave-h4rq"
I20250626 01:58:38.820039 18884 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "e3cd4ea12b534ace8a58080372af5e48"
format_stamp: "Formatted at 2025-06-26 01:58:38 on dist-test-slave-h4rq"
I20250626 01:58:38.827270 18884 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250626 01:58:38.832847 18901 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:58:38.833824 18884 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250626 01:58:38.834136 18884 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e3cd4ea12b534ace8a58080372af5e48"
format_stamp: "Formatted at 2025-06-26 01:58:38 on dist-test-slave-h4rq"
I20250626 01:58:38.834475 18884 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:58:38.896687 18884 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:58:38.898211 18884 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 01:58:38.898648 18884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:58:38.901190 18884 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:58:38.905270 18884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:58:38.905489 18884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:58:38.905846 18884 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:58:38.906013 18884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:58:39.042064 18884 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.195:33015
I20250626 01:58:39.042178 19013 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.195:33015 every 8 connection(s)
I20250626 01:58:39.044752 18884 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250626 01:58:39.047637 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 18884
I20250626 01:58:39.048339 16463 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250626 01:58:39.065202 19014 heartbeater.cc:344] Connected to a master server at 127.16.19.254:33149
I20250626 01:58:39.065742 19014 heartbeater.cc:461] Registering TS with master...
I20250626 01:58:39.066927 19014 heartbeater.cc:507] Master 127.16.19.254:33149 requested a full tablet report, sending...
I20250626 01:58:39.069171 18559 ts_manager.cc:194] Registered new tserver with Master: e3cd4ea12b534ace8a58080372af5e48 (127.16.19.195:33015)
I20250626 01:58:39.070372 18559 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.19.195:51183
I20250626 01:58:39.083019 16463 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 01:58:39.116871 18559 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48542:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250626 01:58:39.135978 18559 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 01:58:39.191190 18682 tablet_service.cc:1468] Processing CreateTablet for tablet 025638836773406a8cc091e2ebe3378c (DEFAULT_TABLE table=TestTable [id=e75cc19ee1f94b8e880670ea2eef782a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 01:58:39.193372 18682 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 025638836773406a8cc091e2ebe3378c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:58:39.196981 18815 tablet_service.cc:1468] Processing CreateTablet for tablet 025638836773406a8cc091e2ebe3378c (DEFAULT_TABLE table=TestTable [id=e75cc19ee1f94b8e880670ea2eef782a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 01:58:39.198170 18949 tablet_service.cc:1468] Processing CreateTablet for tablet 025638836773406a8cc091e2ebe3378c (DEFAULT_TABLE table=TestTable [id=e75cc19ee1f94b8e880670ea2eef782a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 01:58:39.200131 18815 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 025638836773406a8cc091e2ebe3378c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:58:39.201054 18949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 025638836773406a8cc091e2ebe3378c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:58:39.229421 19033 tablet_bootstrap.cc:492] T 025638836773406a8cc091e2ebe3378c P eabb181d28354629b61e387eecb73943: Bootstrap starting.
I20250626 01:58:39.233114 19034 tablet_bootstrap.cc:492] T 025638836773406a8cc091e2ebe3378c P 972d6bc25d644b52bc2d36bace6943da: Bootstrap starting.
I20250626 01:58:39.237218 19033 tablet_bootstrap.cc:654] T 025638836773406a8cc091e2ebe3378c P eabb181d28354629b61e387eecb73943: Neither blocks nor log segments found. Creating new log.
I20250626 01:58:39.239763 19033 log.cc:826] T 025638836773406a8cc091e2ebe3378c P eabb181d28354629b61e387eecb73943: Log is configured to *not* fsync() on all Append() calls
I20250626 01:58:39.244865 19035 tablet_bootstrap.cc:492] T 025638836773406a8cc091e2ebe3378c P e3cd4ea12b534ace8a58080372af5e48: Bootstrap starting.
I20250626 01:58:39.249267 19034 tablet_bootstrap.cc:654] T 025638836773406a8cc091e2ebe3378c P 972d6bc25d644b52bc2d36bace6943da: Neither blocks nor log segments found. Creating new log.
I20250626 01:58:39.251289 19033 tablet_bootstrap.cc:492] T 025638836773406a8cc091e2ebe3378c P eabb181d28354629b61e387eecb73943: No bootstrap required, opened a new log
I20250626 01:58:39.251729 19034 log.cc:826] T 025638836773406a8cc091e2ebe3378c P 972d6bc25d644b52bc2d36bace6943da: Log is configured to *not* fsync() on all Append() calls
I20250626 01:58:39.251842 19033 ts_tablet_manager.cc:1397] T 025638836773406a8cc091e2ebe3378c P eabb181d28354629b61e387eecb73943: Time spent bootstrapping tablet: real 0.023s user 0.012s sys 0.009s
I20250626 01:58:39.259943 19035 tablet_bootstrap.cc:654] T 025638836773406a8cc091e2ebe3378c P e3cd4ea12b534ace8a58080372af5e48: Neither blocks nor log segments found. Creating new log.
I20250626 01:58:39.261992 19034 tablet_bootstrap.cc:492] T 025638836773406a8cc091e2ebe3378c P 972d6bc25d644b52bc2d36bace6943da: No bootstrap required, opened a new log
I20250626 01:58:39.262434 19034 ts_tablet_manager.cc:1397] T 025638836773406a8cc091e2ebe3378c P 972d6bc25d644b52bc2d36bace6943da: Time spent bootstrapping tablet: real 0.030s user 0.010s sys 0.010s
I20250626 01:58:39.262341 19035 log.cc:826] T 025638836773406a8cc091e2ebe3378c P e3cd4ea12b534ace8a58080372af5e48: Log is configured to *not* fsync() on all Append() calls
I20250626 01:58:39.268038 19035 tablet_bootstrap.cc:492] T 025638836773406a8cc091e2ebe3378c P e3cd4ea12b534ace8a58080372af5e48: No bootstrap required, opened a new log
I20250626 01:58:39.268678 19035 ts_tablet_manager.cc:1397] T 025638836773406a8cc091e2ebe3378c P e3cd4ea12b534ace8a58080372af5e48: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.015s
I20250626 01:58:39.275379 19033 raft_consensus.cc:357] T 025638836773406a8cc091e2ebe3378c P eabb181d28354629b61e387eecb73943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabb181d28354629b61e387eecb73943" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 37393 } } peers { permanent_uuid: "e3cd4ea12b534ace8a58080372af5e48" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 33015 } } peers { permanent_uuid: "972d6bc25d644b52bc2d36bace6943da" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 44779 } }
I20250626 01:58:39.276077 19033 raft_consensus.cc:383] T 025638836773406a8cc091e2ebe3378c P eabb181d28354629b61e387eecb73943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:58:39.276336 19033 raft_consensus.cc:738] T 025638836773406a8cc091e2ebe3378c P eabb181d28354629b61e387eecb73943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eabb181d28354629b61e387eecb73943, State: Initialized, Role: FOLLOWER
I20250626 01:58:39.277127 19033 consensus_queue.cc:260] T 025638836773406a8cc091e2ebe3378c P eabb181d28354629b61e387eecb73943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabb181d28354629b61e387eecb73943" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 37393 } } peers { permanent_uuid: "e3cd4ea12b534ace8a58080372af5e48" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 33015 } } peers { permanent_uuid: "972d6bc25d644b52bc2d36bace6943da" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 44779 } }
I20250626 01:58:39.281045 19033 ts_tablet_manager.cc:1428] T 025638836773406a8cc091e2ebe3378c P eabb181d28354629b61e387eecb73943: Time spent starting tablet: real 0.029s user 0.021s sys 0.008s
I20250626 01:58:39.288093 19034 raft_consensus.cc:357] T 025638836773406a8cc091e2ebe3378c P 972d6bc25d644b52bc2d36bace6943da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabb181d28354629b61e387eecb73943" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 37393 } } peers { permanent_uuid: "e3cd4ea12b534ace8a58080372af5e48" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 33015 } } peers { permanent_uuid: "972d6bc25d644b52bc2d36bace6943da" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 44779 } }
I20250626 01:58:39.288972 19034 raft_consensus.cc:383] T 025638836773406a8cc091e2ebe3378c P 972d6bc25d644b52bc2d36bace6943da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:58:39.289244 19034 raft_consensus.cc:738] T 025638836773406a8cc091e2ebe3378c P 972d6bc25d644b52bc2d36bace6943da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 972d6bc25d644b52bc2d36bace6943da, State: Initialized, Role: FOLLOWER
I20250626 01:58:39.290001 19034 consensus_queue.cc:260] T 025638836773406a8cc091e2ebe3378c P 972d6bc25d644b52bc2d36bace6943da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabb181d28354629b61e387eecb73943" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 37393 } } peers { permanent_uuid: "e3cd4ea12b534ace8a58080372af5e48" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 33015 } } peers { permanent_uuid: "972d6bc25d644b52bc2d36bace6943da" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 44779 } }
I20250626 01:58:39.294425 19035 raft_consensus.cc:357] T 025638836773406a8cc091e2ebe3378c P e3cd4ea12b534ace8a58080372af5e48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabb181d28354629b61e387eecb73943" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 37393 } } peers { permanent_uuid: "e3cd4ea12b534ace8a58080372af5e48" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 33015 } } peers { permanent_uuid: "972d6bc25d644b52bc2d36bace6943da" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 44779 } }
I20250626 01:58:39.295182 19035 raft_consensus.cc:383] T 025638836773406a8cc091e2ebe3378c P e3cd4ea12b534ace8a58080372af5e48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:58:39.295498 19035 raft_consensus.cc:738] T 025638836773406a8cc091e2ebe3378c P e3cd4ea12b534ace8a58080372af5e48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cd4ea12b534ace8a58080372af5e48, State: Initialized, Role: FOLLOWER
I20250626 01:58:39.296412 19035 consensus_queue.cc:260] T 025638836773406a8cc091e2ebe3378c P e3cd4ea12b534ace8a58080372af5e48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabb181d28354629b61e387eecb73943" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 37393 } } peers { permanent_uuid: "e3cd4ea12b534ace8a58080372af5e48" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 33015 } } peers { permanent_uuid: "972d6bc25d644b52bc2d36bace6943da" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 44779 } }
I20250626 01:58:39.297535 19034 ts_tablet_manager.cc:1428] T 025638836773406a8cc091e2ebe3378c P 972d6bc25d644b52bc2d36bace6943da: Time spent starting tablet: real 0.035s user 0.034s sys 0.002s
I20250626 01:58:39.299417 19014 heartbeater.cc:499] Master 127.16.19.254:33149 was elected leader, sending a full tablet report...
I20250626 01:58:39.300784 19035 ts_tablet_manager.cc:1428] T 025638836773406a8cc091e2ebe3378c P e3cd4ea12b534ace8a58080372af5e48: Time spent starting tablet: real 0.032s user 0.020s sys 0.011s
W20250626 01:58:39.372821 18748 tablet.cc:2378] T 025638836773406a8cc091e2ebe3378c P eabb181d28354629b61e387eecb73943: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 01:58:39.464285 19039 raft_consensus.cc:491] T 025638836773406a8cc091e2ebe3378c P eabb181d28354629b61e387eecb73943 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:58:39.464835 19039 raft_consensus.cc:513] T 025638836773406a8cc091e2ebe3378c P eabb181d28354629b61e387eecb73943 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabb181d28354629b61e387eecb73943" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 37393 } } peers { permanent_uuid: "e3cd4ea12b534ace8a58080372af5e48" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 33015 } } peers { permanent_uuid: "972d6bc25d644b52bc2d36bace6943da" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 44779 } }
I20250626 01:58:39.467152 19039 leader_election.cc:290] T 025638836773406a8cc091e2ebe3378c P eabb181d28354629b61e387eecb73943 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3cd4ea12b534ace8a58080372af5e48 (127.16.19.195:33015), 972d6bc25d644b52bc2d36bace6943da (127.16.19.194:44779)
W20250626 01:58:39.470467 18881 tablet.cc:2378] T 025638836773406a8cc091e2ebe3378c P 972d6bc25d644b52bc2d36bace6943da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 01:58:39.478215 18969 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "025638836773406a8cc091e2ebe3378c" candidate_uuid: "eabb181d28354629b61e387eecb73943" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cd4ea12b534ace8a58080372af5e48" is_pre_election: true
I20250626 01:58:39.478940 18969 raft_consensus.cc:2466] T 025638836773406a8cc091e2ebe3378c P e3cd4ea12b534ace8a58080372af5e48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eabb181d28354629b61e387eecb73943 in term 0.
I20250626 01:58:39.480015 18635 leader_election.cc:304] T 025638836773406a8cc091e2ebe3378c P eabb181d28354629b61e387eecb73943 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3cd4ea12b534ace8a58080372af5e48, eabb181d28354629b61e387eecb73943; no voters:
I20250626 01:58:39.480064 18835 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "025638836773406a8cc091e2ebe3378c" candidate_uuid: "eabb181d28354629b61e387eecb73943" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "972d6bc25d644b52bc2d36bace6943da" is_pre_election: true
I20250626 01:58:39.480753 19039 raft_consensus.cc:2802] T 025638836773406a8cc091e2ebe3378c P eabb181d28354629b61e387eecb73943 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:58:39.480794 18835 raft_consensus.cc:2466] T 025638836773406a8cc091e2ebe3378c P 972d6bc25d644b52bc2d36bace6943da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eabb181d28354629b61e387eecb73943 in term 0.
I20250626 01:58:39.481027 19039 raft_consensus.cc:491] T 025638836773406a8cc091e2ebe3378c P eabb181d28354629b61e387eecb73943 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:58:39.481302 19039 raft_consensus.cc:3058] T 025638836773406a8cc091e2ebe3378c P eabb181d28354629b61e387eecb73943 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:58:39.485759 19039 raft_consensus.cc:513] T 025638836773406a8cc091e2ebe3378c P eabb181d28354629b61e387eecb73943 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabb181d28354629b61e387eecb73943" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 37393 } } peers { permanent_uuid: "e3cd4ea12b534ace8a58080372af5e48" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 33015 } } peers { permanent_uuid: "972d6bc25d644b52bc2d36bace6943da" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 44779 } }
I20250626 01:58:39.487058 19039 leader_election.cc:290] T 025638836773406a8cc091e2ebe3378c P eabb181d28354629b61e387eecb73943 [CANDIDATE]: Term 1 election: Requested vote from peers e3cd4ea12b534ace8a58080372af5e48 (127.16.19.195:33015), 972d6bc25d644b52bc2d36bace6943da (127.16.19.194:44779)
I20250626 01:58:39.487835 18969 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "025638836773406a8cc091e2ebe3378c" candidate_uuid: "eabb181d28354629b61e387eecb73943" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cd4ea12b534ace8a58080372af5e48"
I20250626 01:58:39.487967 18835 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "025638836773406a8cc091e2ebe3378c" candidate_uuid: "eabb181d28354629b61e387eecb73943" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "972d6bc25d644b52bc2d36bace6943da"
I20250626 01:58:39.488234 18969 raft_consensus.cc:3058] T 025638836773406a8cc091e2ebe3378c P e3cd4ea12b534ace8a58080372af5e48 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:58:39.488401 18835 raft_consensus.cc:3058] T 025638836773406a8cc091e2ebe3378c P 972d6bc25d644b52bc2d36bace6943da [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:58:39.492581 18969 raft_consensus.cc:2466] T 025638836773406a8cc091e2ebe3378c P e3cd4ea12b534ace8a58080372af5e48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eabb181d28354629b61e387eecb73943 in term 1.
I20250626 01:58:39.492764 18835 raft_consensus.cc:2466] T 025638836773406a8cc091e2ebe3378c P 972d6bc25d644b52bc2d36bace6943da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eabb181d28354629b61e387eecb73943 in term 1.
I20250626 01:58:39.493455 18635 leader_election.cc:304] T 025638836773406a8cc091e2ebe3378c P eabb181d28354629b61e387eecb73943 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3cd4ea12b534ace8a58080372af5e48, eabb181d28354629b61e387eecb73943; no voters:
I20250626 01:58:39.494045 19039 raft_consensus.cc:2802] T 025638836773406a8cc091e2ebe3378c P eabb181d28354629b61e387eecb73943 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:58:39.495561 19039 raft_consensus.cc:695] T 025638836773406a8cc091e2ebe3378c P eabb181d28354629b61e387eecb73943 [term 1 LEADER]: Becoming Leader. State: Replica: eabb181d28354629b61e387eecb73943, State: Running, Role: LEADER
I20250626 01:58:39.496280 19039 consensus_queue.cc:237] T 025638836773406a8cc091e2ebe3378c P eabb181d28354629b61e387eecb73943 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabb181d28354629b61e387eecb73943" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 37393 } } peers { permanent_uuid: "e3cd4ea12b534ace8a58080372af5e48" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 33015 } } peers { permanent_uuid: "972d6bc25d644b52bc2d36bace6943da" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 44779 } }
I20250626 01:58:39.507227 18558 catalog_manager.cc:5582] T 025638836773406a8cc091e2ebe3378c P eabb181d28354629b61e387eecb73943 reported cstate change: term changed from 0 to 1, leader changed from <none> to eabb181d28354629b61e387eecb73943 (127.16.19.193). New cstate: current_term: 1 leader_uuid: "eabb181d28354629b61e387eecb73943" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabb181d28354629b61e387eecb73943" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 37393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cd4ea12b534ace8a58080372af5e48" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 33015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "972d6bc25d644b52bc2d36bace6943da" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 44779 } health_report { overall_health: UNKNOWN } } }
W20250626 01:58:39.550305 19015 tablet.cc:2378] T 025638836773406a8cc091e2ebe3378c P e3cd4ea12b534ace8a58080372af5e48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 01:58:39.587731 16463 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 01:58:39.590862 16463 ts_itest-base.cc:246] Waiting for 1 tablets on tserver eabb181d28354629b61e387eecb73943 to finish bootstrapping
I20250626 01:58:39.602737 16463 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 972d6bc25d644b52bc2d36bace6943da to finish bootstrapping
I20250626 01:58:39.613461 16463 ts_itest-base.cc:246] Waiting for 1 tablets on tserver e3cd4ea12b534ace8a58080372af5e48 to finish bootstrapping
I20250626 01:58:39.901129 19039 consensus_queue.cc:1035] T 025638836773406a8cc091e2ebe3378c P eabb181d28354629b61e387eecb73943 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cd4ea12b534ace8a58080372af5e48" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 33015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:58:39.924393 19044 consensus_queue.cc:1035] T 025638836773406a8cc091e2ebe3378c P eabb181d28354629b61e387eecb73943 [LEADER]: Connected to new peer: Peer: permanent_uuid: "972d6bc25d644b52bc2d36bace6943da" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 44779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 01:58:41.304312 18558 server_base.cc:1130] Unauthorized access attempt to method kudu.master.MasterService.RefreshAuthzCache from {username='slave'} at 127.0.0.1:48568
I20250626 01:58:42.333626 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 18618
I20250626 01:58:42.368887 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 18751
I20250626 01:58:42.395852 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 18884
I20250626 01:58:42.422703 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 18526
2025-06-26T01:58:42Z chronyd exiting
[ OK ] AdminCliTest.TestAuthzResetCacheNotAuthorized (10939 ms)
[ RUN ] AdminCliTest.TestRebuildTables
I20250626 01:58:42.474713 16463 test_util.cc:276] Using random seed: -425281858
I20250626 01:58:42.478683 16463 ts_itest-base.cc:115] Starting cluster with:
I20250626 01:58:42.478855 16463 ts_itest-base.cc:116] --------------
I20250626 01:58:42.478969 16463 ts_itest-base.cc:117] 3 tablet servers
I20250626 01:58:42.479079 16463 ts_itest-base.cc:118] 3 replicas per TS
I20250626 01:58:42.479198 16463 ts_itest-base.cc:119] --------------
2025-06-26T01:58:42Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-26T01:58:42Z Disabled control of system clock
I20250626 01:58:42.515123 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.19.254:43009
--webserver_interface=127.16.19.254
--webserver_port=0
--builtin_ntp_servers=127.16.19.212:42273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.19.254:43009 with env {}
W20250626 01:58:42.818856 19080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:58:42.819406 19080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 01:58:42.819830 19080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:58:42.852133 19080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 01:58:42.852447 19080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 01:58:42.852718 19080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 01:58:42.852937 19080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 01:58:42.889279 19080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:42273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.19.254:43009
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.19.254:43009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.19.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 01:58:42.890592 19080 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 01:58:42.892262 19080 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:58:42.906597 19086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:58:42.907296 19089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:58:42.908329 19080 server_base.cc:1048] running on GCE node
W20250626 01:58:42.907686 19087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:58:44.109582 19080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 01:58:44.112833 19080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 01:58:44.114305 19080 hybrid_clock.cc:648] HybridClock initialized: now 1750903124114265 us; error 56 us; skew 500 ppm
I20250626 01:58:44.115162 19080 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:58:44.125195 19080 webserver.cc:469] Webserver started at http://127.16.19.254:40755/ using document root <none> and password file <none>
I20250626 01:58:44.126192 19080 fs_manager.cc:362] Metadata directory not provided
I20250626 01:58:44.126389 19080 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:58:44.126914 19080 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:58:44.131472 19080 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "eb90a9aba85a4018bf48a8e64992e8f0"
format_stamp: "Formatted at 2025-06-26 01:58:44 on dist-test-slave-h4rq"
I20250626 01:58:44.132612 19080 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "eb90a9aba85a4018bf48a8e64992e8f0"
format_stamp: "Formatted at 2025-06-26 01:58:44 on dist-test-slave-h4rq"
I20250626 01:58:44.139936 19080 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20250626 01:58:44.145525 19096 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:58:44.146642 19080 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250626 01:58:44.146978 19080 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
uuid: "eb90a9aba85a4018bf48a8e64992e8f0"
format_stamp: "Formatted at 2025-06-26 01:58:44 on dist-test-slave-h4rq"
I20250626 01:58:44.147312 19080 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:58:44.216940 19080 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:58:44.218497 19080 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 01:58:44.218961 19080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:58:44.291390 19080 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.254:43009
I20250626 01:58:44.291450 19147 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.254:43009 every 8 connection(s)
I20250626 01:58:44.294332 19080 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250626 01:58:44.299474 19148 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:58:44.301384 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 19080
I20250626 01:58:44.301806 16463 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250626 01:58:44.323134 19148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0: Bootstrap starting.
I20250626 01:58:44.329375 19148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0: Neither blocks nor log segments found. Creating new log.
I20250626 01:58:44.331365 19148 log.cc:826] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0: Log is configured to *not* fsync() on all Append() calls
I20250626 01:58:44.335804 19148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0: No bootstrap required, opened a new log
I20250626 01:58:44.353830 19148 raft_consensus.cc:357] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb90a9aba85a4018bf48a8e64992e8f0" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 43009 } }
I20250626 01:58:44.354703 19148 raft_consensus.cc:383] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:58:44.354911 19148 raft_consensus.cc:738] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb90a9aba85a4018bf48a8e64992e8f0, State: Initialized, Role: FOLLOWER
I20250626 01:58:44.355515 19148 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb90a9aba85a4018bf48a8e64992e8f0" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 43009 } }
I20250626 01:58:44.355983 19148 raft_consensus.cc:397] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 01:58:44.356204 19148 raft_consensus.cc:491] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 01:58:44.356456 19148 raft_consensus.cc:3058] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:58:44.360456 19148 raft_consensus.cc:513] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb90a9aba85a4018bf48a8e64992e8f0" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 43009 } }
I20250626 01:58:44.361169 19148 leader_election.cc:304] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb90a9aba85a4018bf48a8e64992e8f0; no voters:
I20250626 01:58:44.363124 19148 leader_election.cc:290] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 01:58:44.363884 19153 raft_consensus.cc:2802] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:58:44.366055 19153 raft_consensus.cc:695] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [term 1 LEADER]: Becoming Leader. State: Replica: eb90a9aba85a4018bf48a8e64992e8f0, State: Running, Role: LEADER
I20250626 01:58:44.366776 19153 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb90a9aba85a4018bf48a8e64992e8f0" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 43009 } }
I20250626 01:58:44.367349 19148 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 01:58:44.377665 19154 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb90a9aba85a4018bf48a8e64992e8f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb90a9aba85a4018bf48a8e64992e8f0" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 43009 } } }
I20250626 01:58:44.378481 19154 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [sys.catalog]: This master's current role is: LEADER
I20250626 01:58:44.380470 19155 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb90a9aba85a4018bf48a8e64992e8f0. Latest consensus state: current_term: 1 leader_uuid: "eb90a9aba85a4018bf48a8e64992e8f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb90a9aba85a4018bf48a8e64992e8f0" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 43009 } } }
I20250626 01:58:44.381711 19155 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [sys.catalog]: This master's current role is: LEADER
I20250626 01:58:44.381839 19162 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 01:58:44.393543 19162 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 01:58:44.410712 19162 catalog_manager.cc:1349] Generated new cluster ID: b34015b3edf14cc89b090bae5fb94094
I20250626 01:58:44.411044 19162 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 01:58:44.435943 19162 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 01:58:44.437507 19162 catalog_manager.cc:1506] Loading token signing keys...
I20250626 01:58:44.451735 19162 catalog_manager.cc:5955] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0: Generated new TSK 0
I20250626 01:58:44.452728 19162 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 01:58:44.471915 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.19.193:0
--local_ip_for_outbound_sockets=127.16.19.193
--webserver_interface=127.16.19.193
--webserver_port=0
--tserver_master_addrs=127.16.19.254:43009
--builtin_ntp_servers=127.16.19.212:42273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250626 01:58:44.781631 19172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:58:44.782163 19172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 01:58:44.782662 19172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:58:44.814404 19172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 01:58:44.815441 19172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.19.193
I20250626 01:58:44.851238 19172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:42273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.19.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.19.193
--webserver_port=0
--tserver_master_addrs=127.16.19.254:43009
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.19.193
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 01:58:44.852568 19172 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 01:58:44.854270 19172 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:58:44.872603 19179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:58:44.874614 19178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:58:44.872686 19181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:58:46.184217 19180 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1309 milliseconds
I20250626 01:58:46.184319 19172 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 01:58:46.185563 19172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 01:58:46.188313 19172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 01:58:46.189913 19172 hybrid_clock.cc:648] HybridClock initialized: now 1750903126189854 us; error 64 us; skew 500 ppm
I20250626 01:58:46.190749 19172 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:58:46.198122 19172 webserver.cc:469] Webserver started at http://127.16.19.193:44591/ using document root <none> and password file <none>
I20250626 01:58:46.199072 19172 fs_manager.cc:362] Metadata directory not provided
I20250626 01:58:46.199276 19172 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:58:46.199748 19172 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:58:46.204252 19172 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "f91b0712e19140bd9b9453efa221e865"
format_stamp: "Formatted at 2025-06-26 01:58:46 on dist-test-slave-h4rq"
I20250626 01:58:46.205453 19172 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "f91b0712e19140bd9b9453efa221e865"
format_stamp: "Formatted at 2025-06-26 01:58:46 on dist-test-slave-h4rq"
I20250626 01:58:46.213337 19172 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250626 01:58:46.219576 19188 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:58:46.220767 19172 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20250626 01:58:46.221089 19172 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f91b0712e19140bd9b9453efa221e865"
format_stamp: "Formatted at 2025-06-26 01:58:46 on dist-test-slave-h4rq"
I20250626 01:58:46.221412 19172 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:58:46.275811 19172 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:58:46.277417 19172 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 01:58:46.277843 19172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:58:46.280865 19172 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:58:46.285778 19172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:58:46.285985 19172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:58:46.286247 19172 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:58:46.286403 19172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:58:46.453608 19172 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.193:44115
I20250626 01:58:46.453786 19300 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.193:44115 every 8 connection(s)
I20250626 01:58:46.456208 19172 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250626 01:58:46.459759 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 19172
I20250626 01:58:46.460238 16463 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250626 01:58:46.468068 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.19.194:0
--local_ip_for_outbound_sockets=127.16.19.194
--webserver_interface=127.16.19.194
--webserver_port=0
--tserver_master_addrs=127.16.19.254:43009
--builtin_ntp_servers=127.16.19.212:42273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 01:58:46.491307 19301 heartbeater.cc:344] Connected to a master server at 127.16.19.254:43009
I20250626 01:58:46.491755 19301 heartbeater.cc:461] Registering TS with master...
I20250626 01:58:46.492815 19301 heartbeater.cc:507] Master 127.16.19.254:43009 requested a full tablet report, sending...
I20250626 01:58:46.495303 19113 ts_manager.cc:194] Registered new tserver with Master: f91b0712e19140bd9b9453efa221e865 (127.16.19.193:44115)
I20250626 01:58:46.497326 19113 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.19.193:54409
W20250626 01:58:46.790649 19305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:58:46.791181 19305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 01:58:46.791659 19305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:58:46.822750 19305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 01:58:46.823599 19305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.19.194
I20250626 01:58:46.858282 19305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:42273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.19.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.16.19.194
--webserver_port=0
--tserver_master_addrs=127.16.19.254:43009
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.19.194
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 01:58:46.859580 19305 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 01:58:46.861229 19305 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:58:46.878950 19312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:58:47.500751 19301 heartbeater.cc:499] Master 127.16.19.254:43009 was elected leader, sending a full tablet report...
W20250626 01:58:46.879385 19311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:58:46.879896 19314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:58:46.880839 19305 server_base.cc:1048] running on GCE node
I20250626 01:58:48.050856 19305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 01:58:48.053510 19305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 01:58:48.054989 19305 hybrid_clock.cc:648] HybridClock initialized: now 1750903128054945 us; error 56 us; skew 500 ppm
I20250626 01:58:48.056174 19305 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:58:48.062999 19305 webserver.cc:469] Webserver started at http://127.16.19.194:38337/ using document root <none> and password file <none>
I20250626 01:58:48.063994 19305 fs_manager.cc:362] Metadata directory not provided
I20250626 01:58:48.064204 19305 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:58:48.064756 19305 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:58:48.071514 19305 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "cf4e54b733f84093b1488c82347f0389"
format_stamp: "Formatted at 2025-06-26 01:58:48 on dist-test-slave-h4rq"
I20250626 01:58:48.072757 19305 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "cf4e54b733f84093b1488c82347f0389"
format_stamp: "Formatted at 2025-06-26 01:58:48 on dist-test-slave-h4rq"
I20250626 01:58:48.080816 19305 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20250626 01:58:48.087076 19322 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:58:48.088243 19305 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20250626 01:58:48.088587 19305 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "cf4e54b733f84093b1488c82347f0389"
format_stamp: "Formatted at 2025-06-26 01:58:48 on dist-test-slave-h4rq"
I20250626 01:58:48.089005 19305 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:58:48.150705 19305 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:58:48.152215 19305 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 01:58:48.152693 19305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:58:48.155247 19305 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:58:48.159583 19305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:58:48.159802 19305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:58:48.160077 19305 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:58:48.160236 19305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:58:48.303800 19305 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.194:39201
I20250626 01:58:48.303917 19434 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.194:39201 every 8 connection(s)
I20250626 01:58:48.306522 19305 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250626 01:58:48.311393 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 19305
I20250626 01:58:48.312140 16463 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250626 01:58:48.319629 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.19.195:0
--local_ip_for_outbound_sockets=127.16.19.195
--webserver_interface=127.16.19.195
--webserver_port=0
--tserver_master_addrs=127.16.19.254:43009
--builtin_ntp_servers=127.16.19.212:42273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 01:58:48.331403 19435 heartbeater.cc:344] Connected to a master server at 127.16.19.254:43009
I20250626 01:58:48.331982 19435 heartbeater.cc:461] Registering TS with master...
I20250626 01:58:48.333436 19435 heartbeater.cc:507] Master 127.16.19.254:43009 requested a full tablet report, sending...
I20250626 01:58:48.336127 19113 ts_manager.cc:194] Registered new tserver with Master: cf4e54b733f84093b1488c82347f0389 (127.16.19.194:39201)
I20250626 01:58:48.337543 19113 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.19.194:58341
W20250626 01:58:48.623895 19439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:58:48.624425 19439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 01:58:48.624975 19439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:58:48.656294 19439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 01:58:48.657315 19439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.19.195
I20250626 01:58:48.694221 19439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:42273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.19.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.16.19.195
--webserver_port=0
--tserver_master_addrs=127.16.19.254:43009
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.19.195
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 01:58:48.695638 19439 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 01:58:48.697326 19439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:58:48.712196 19445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:58:49.341712 19435 heartbeater.cc:499] Master 127.16.19.254:43009 was elected leader, sending a full tablet report...
W20250626 01:58:48.714293 19448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:58:48.712301 19446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:58:49.932920 19447 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1217 milliseconds
I20250626 01:58:49.933059 19439 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 01:58:49.934403 19439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 01:58:49.936961 19439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 01:58:49.938400 19439 hybrid_clock.cc:648] HybridClock initialized: now 1750903129938374 us; error 126 us; skew 500 ppm
I20250626 01:58:49.939388 19439 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:58:49.951723 19439 webserver.cc:469] Webserver started at http://127.16.19.195:39177/ using document root <none> and password file <none>
I20250626 01:58:49.952797 19439 fs_manager.cc:362] Metadata directory not provided
I20250626 01:58:49.953037 19439 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:58:49.953531 19439 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:58:49.958348 19439 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "7ab09cbcdaba4e069a36503512251ef7"
format_stamp: "Formatted at 2025-06-26 01:58:49 on dist-test-slave-h4rq"
I20250626 01:58:49.959544 19439 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "7ab09cbcdaba4e069a36503512251ef7"
format_stamp: "Formatted at 2025-06-26 01:58:49 on dist-test-slave-h4rq"
I20250626 01:58:49.967118 19439 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20250626 01:58:49.973305 19455 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:58:49.975062 19439 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20250626 01:58:49.975409 19439 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "7ab09cbcdaba4e069a36503512251ef7"
format_stamp: "Formatted at 2025-06-26 01:58:49 on dist-test-slave-h4rq"
I20250626 01:58:49.975847 19439 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:58:50.028730 19439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:58:50.030212 19439 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 01:58:50.030633 19439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:58:50.033155 19439 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:58:50.037312 19439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:58:50.037532 19439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:58:50.037730 19439 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:58:50.037873 19439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:58:50.178795 19439 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.195:40417
I20250626 01:58:50.178930 19567 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.195:40417 every 8 connection(s)
I20250626 01:58:50.181449 19439 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250626 01:58:50.187131 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 19439
I20250626 01:58:50.187630 16463 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250626 01:58:50.205325 19568 heartbeater.cc:344] Connected to a master server at 127.16.19.254:43009
I20250626 01:58:50.205890 19568 heartbeater.cc:461] Registering TS with master...
I20250626 01:58:50.207146 19568 heartbeater.cc:507] Master 127.16.19.254:43009 requested a full tablet report, sending...
I20250626 01:58:50.209476 19113 ts_manager.cc:194] Registered new tserver with Master: 7ab09cbcdaba4e069a36503512251ef7 (127.16.19.195:40417)
I20250626 01:58:50.210800 19113 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.19.195:42157
I20250626 01:58:50.224179 16463 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 01:58:50.263649 16463 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 01:58:50.263974 16463 test_util.cc:276] Using random seed: -417492586
I20250626 01:58:50.315246 19113 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41800:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 01:58:50.360699 19236 tablet_service.cc:1468] Processing CreateTablet for tablet 242b3d8e6a774185a554cf2603c1e5a0 (DEFAULT_TABLE table=TestTable [id=7b7b07110c7d4790b07a2a345dcf503b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 01:58:50.362187 19236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 242b3d8e6a774185a554cf2603c1e5a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:58:50.382110 19588 tablet_bootstrap.cc:492] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865: Bootstrap starting.
I20250626 01:58:50.387770 19588 tablet_bootstrap.cc:654] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865: Neither blocks nor log segments found. Creating new log.
I20250626 01:58:50.389678 19588 log.cc:826] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865: Log is configured to *not* fsync() on all Append() calls
I20250626 01:58:50.394462 19588 tablet_bootstrap.cc:492] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865: No bootstrap required, opened a new log
I20250626 01:58:50.394901 19588 ts_tablet_manager.cc:1397] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.007s
I20250626 01:58:50.413331 19588 raft_consensus.cc:357] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } }
I20250626 01:58:50.413925 19588 raft_consensus.cc:383] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:58:50.414204 19588 raft_consensus.cc:738] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f91b0712e19140bd9b9453efa221e865, State: Initialized, Role: FOLLOWER
I20250626 01:58:50.414945 19588 consensus_queue.cc:260] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } }
I20250626 01:58:50.415444 19588 raft_consensus.cc:397] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 01:58:50.415755 19588 raft_consensus.cc:491] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 01:58:50.416070 19588 raft_consensus.cc:3058] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:58:50.420858 19588 raft_consensus.cc:513] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } }
I20250626 01:58:50.421609 19588 leader_election.cc:304] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f91b0712e19140bd9b9453efa221e865; no voters:
I20250626 01:58:50.423444 19588 leader_election.cc:290] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 01:58:50.423827 19590 raft_consensus.cc:2802] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:58:50.426192 19590 raft_consensus.cc:695] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [term 1 LEADER]: Becoming Leader. State: Replica: f91b0712e19140bd9b9453efa221e865, State: Running, Role: LEADER
I20250626 01:58:50.427210 19588 ts_tablet_manager.cc:1428] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865: Time spent starting tablet: real 0.032s user 0.019s sys 0.013s
I20250626 01:58:50.427110 19590 consensus_queue.cc:237] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } }
I20250626 01:58:50.441535 19113 catalog_manager.cc:5582] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 reported cstate change: term changed from 0 to 1, leader changed from <none> to f91b0712e19140bd9b9453efa221e865 (127.16.19.193). New cstate: current_term: 1 leader_uuid: "f91b0712e19140bd9b9453efa221e865" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } health_report { overall_health: HEALTHY } } }
I20250626 01:58:50.677201 16463 test_util.cc:276] Using random seed: -417079377
I20250626 01:58:50.702915 19111 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41816:
name: "TestTable1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 01:58:50.734706 19503 tablet_service.cc:1468] Processing CreateTablet for tablet bbe200e5cd43440e83047942538c8c1b (DEFAULT_TABLE table=TestTable1 [id=49f418de1b8e40d2992cf549672011a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 01:58:50.736249 19503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbe200e5cd43440e83047942538c8c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:58:50.757934 19609 tablet_bootstrap.cc:492] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7: Bootstrap starting.
I20250626 01:58:50.763938 19609 tablet_bootstrap.cc:654] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7: Neither blocks nor log segments found. Creating new log.
I20250626 01:58:50.765851 19609 log.cc:826] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7: Log is configured to *not* fsync() on all Append() calls
I20250626 01:58:50.771016 19609 tablet_bootstrap.cc:492] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7: No bootstrap required, opened a new log
I20250626 01:58:50.771564 19609 ts_tablet_manager.cc:1397] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20250626 01:58:50.789776 19609 raft_consensus.cc:357] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } }
I20250626 01:58:50.790426 19609 raft_consensus.cc:383] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:58:50.790661 19609 raft_consensus.cc:738] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ab09cbcdaba4e069a36503512251ef7, State: Initialized, Role: FOLLOWER
I20250626 01:58:50.791411 19609 consensus_queue.cc:260] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } }
I20250626 01:58:50.791936 19609 raft_consensus.cc:397] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 01:58:50.792217 19609 raft_consensus.cc:491] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 01:58:50.792539 19609 raft_consensus.cc:3058] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:58:50.796876 19609 raft_consensus.cc:513] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } }
I20250626 01:58:50.797655 19609 leader_election.cc:304] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ab09cbcdaba4e069a36503512251ef7; no voters:
I20250626 01:58:50.799492 19609 leader_election.cc:290] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 01:58:50.800098 19611 raft_consensus.cc:2802] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:58:50.803287 19568 heartbeater.cc:499] Master 127.16.19.254:43009 was elected leader, sending a full tablet report...
I20250626 01:58:50.802680 19611 raft_consensus.cc:695] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [term 1 LEADER]: Becoming Leader. State: Replica: 7ab09cbcdaba4e069a36503512251ef7, State: Running, Role: LEADER
I20250626 01:58:50.803917 19609 ts_tablet_manager.cc:1428] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7: Time spent starting tablet: real 0.032s user 0.026s sys 0.007s
I20250626 01:58:50.804235 19611 consensus_queue.cc:237] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } }
I20250626 01:58:50.816568 19111 catalog_manager.cc:5582] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ab09cbcdaba4e069a36503512251ef7 (127.16.19.195). New cstate: current_term: 1 leader_uuid: "7ab09cbcdaba4e069a36503512251ef7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } health_report { overall_health: HEALTHY } } }
I20250626 01:58:51.018573 16463 test_util.cc:276] Using random seed: -416738008
I20250626 01:58:51.042201 19113 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41824:
name: "TestTable2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 01:58:51.071935 19370 tablet_service.cc:1468] Processing CreateTablet for tablet d5b1c5efbd044314965eb5c7b1bede3c (DEFAULT_TABLE table=TestTable2 [id=53b4ca6fa1494d44822bc977e13bd91f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 01:58:51.073449 19370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5b1c5efbd044314965eb5c7b1bede3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:58:51.094255 19631 tablet_bootstrap.cc:492] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389: Bootstrap starting.
I20250626 01:58:51.100867 19631 tablet_bootstrap.cc:654] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389: Neither blocks nor log segments found. Creating new log.
I20250626 01:58:51.102881 19631 log.cc:826] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389: Log is configured to *not* fsync() on all Append() calls
I20250626 01:58:51.107964 19631 tablet_bootstrap.cc:492] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389: No bootstrap required, opened a new log
I20250626 01:58:51.108450 19631 ts_tablet_manager.cc:1397] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20250626 01:58:51.127622 19631 raft_consensus.cc:357] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } }
I20250626 01:58:51.128418 19631 raft_consensus.cc:383] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:58:51.128705 19631 raft_consensus.cc:738] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf4e54b733f84093b1488c82347f0389, State: Initialized, Role: FOLLOWER
I20250626 01:58:51.129412 19631 consensus_queue.cc:260] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } }
I20250626 01:58:51.129940 19631 raft_consensus.cc:397] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 01:58:51.130206 19631 raft_consensus.cc:491] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 01:58:51.130514 19631 raft_consensus.cc:3058] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:58:51.135620 19631 raft_consensus.cc:513] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } }
I20250626 01:58:51.136371 19631 leader_election.cc:304] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf4e54b733f84093b1488c82347f0389; no voters:
I20250626 01:58:51.138170 19631 leader_election.cc:290] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 01:58:51.138576 19633 raft_consensus.cc:2802] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:58:51.141184 19633 raft_consensus.cc:695] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389 [term 1 LEADER]: Becoming Leader. State: Replica: cf4e54b733f84093b1488c82347f0389, State: Running, Role: LEADER
I20250626 01:58:51.142189 19633 consensus_queue.cc:237] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } }
I20250626 01:58:51.142735 19631 ts_tablet_manager.cc:1428] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389: Time spent starting tablet: real 0.034s user 0.024s sys 0.008s
I20250626 01:58:51.154410 19113 catalog_manager.cc:5582] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf4e54b733f84093b1488c82347f0389 (127.16.19.194). New cstate: current_term: 1 leader_uuid: "cf4e54b733f84093b1488c82347f0389" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } health_report { overall_health: HEALTHY } } }
I20250626 01:58:51.396183 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 19080
W20250626 01:58:51.459468 19301 heartbeater.cc:646] Failed to heartbeat to 127.16.19.254:43009 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.19.254:43009: connect: Connection refused (error 111)
W20250626 01:58:51.832439 19568 heartbeater.cc:646] Failed to heartbeat to 127.16.19.254:43009 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.19.254:43009: connect: Connection refused (error 111)
W20250626 01:58:52.170984 19435 heartbeater.cc:646] Failed to heartbeat to 127.16.19.254:43009 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.19.254:43009: connect: Connection refused (error 111)
I20250626 01:58:56.290650 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 19172
I20250626 01:58:56.315203 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 19305
I20250626 01:58:56.343178 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 19439
I20250626 01:58:56.370142 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.19.254:43009
--webserver_interface=127.16.19.254
--webserver_port=40755
--builtin_ntp_servers=127.16.19.212:42273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.19.254:43009 with env {}
W20250626 01:58:56.677381 19711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:58:56.678030 19711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 01:58:56.678512 19711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:58:56.710345 19711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 01:58:56.710693 19711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 01:58:56.710974 19711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 01:58:56.711228 19711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 01:58:56.747867 19711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:42273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.19.254:43009
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.19.254:43009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.19.254
--webserver_port=40755
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 01:58:56.749253 19711 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 01:58:56.750963 19711 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:58:56.765576 19718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:58:56.766844 19717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:58:56.767825 19711 server_base.cc:1048] running on GCE node
W20250626 01:58:56.767513 19720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:58:57.942552 19711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 01:58:57.945274 19711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 01:58:57.946686 19711 hybrid_clock.cc:648] HybridClock initialized: now 1750903137946645 us; error 60 us; skew 500 ppm
I20250626 01:58:57.947504 19711 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:58:57.960897 19711 webserver.cc:469] Webserver started at http://127.16.19.254:40755/ using document root <none> and password file <none>
I20250626 01:58:57.962608 19711 fs_manager.cc:362] Metadata directory not provided
I20250626 01:58:57.963068 19711 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:58:57.978941 19711 fs_manager.cc:714] Time spent opening directory manager: real 0.008s user 0.007s sys 0.002s
I20250626 01:58:57.986550 19728 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:58:57.988127 19711 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20250626 01:58:57.988664 19711 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
uuid: "eb90a9aba85a4018bf48a8e64992e8f0"
format_stamp: "Formatted at 2025-06-26 01:58:44 on dist-test-slave-h4rq"
I20250626 01:58:57.991794 19711 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:58:58.075367 19711 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:58:58.076980 19711 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 01:58:58.077406 19711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:58:58.150600 19711 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.254:43009
I20250626 01:58:58.150734 19779 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.254:43009 every 8 connection(s)
I20250626 01:58:58.153544 19711 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250626 01:58:58.159932 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 19711
I20250626 01:58:58.162212 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.19.193:44115
--local_ip_for_outbound_sockets=127.16.19.193
--tserver_master_addrs=127.16.19.254:43009
--webserver_port=44591
--webserver_interface=127.16.19.193
--builtin_ntp_servers=127.16.19.212:42273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 01:58:58.165939 19780 sys_catalog.cc:263] Verifying existing consensus state
I20250626 01:58:58.173825 19780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0: Bootstrap starting.
I20250626 01:58:58.184547 19780 log.cc:826] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0: Log is configured to *not* fsync() on all Append() calls
I20250626 01:58:58.233036 19780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=13 ignored=0} mutations{seen=10 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:58:58.233875 19780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0: Bootstrap complete.
I20250626 01:58:58.254844 19780 raft_consensus.cc:357] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb90a9aba85a4018bf48a8e64992e8f0" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 43009 } }
I20250626 01:58:58.257005 19780 raft_consensus.cc:738] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb90a9aba85a4018bf48a8e64992e8f0, State: Initialized, Role: FOLLOWER
I20250626 01:58:58.257915 19780 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb90a9aba85a4018bf48a8e64992e8f0" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 43009 } }
I20250626 01:58:58.258450 19780 raft_consensus.cc:397] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 01:58:58.258728 19780 raft_consensus.cc:491] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 01:58:58.259042 19780 raft_consensus.cc:3058] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [term 2 FOLLOWER]: Advancing to term 3
I20250626 01:58:58.264771 19780 raft_consensus.cc:513] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb90a9aba85a4018bf48a8e64992e8f0" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 43009 } }
I20250626 01:58:58.265475 19780 leader_election.cc:304] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb90a9aba85a4018bf48a8e64992e8f0; no voters:
I20250626 01:58:58.267608 19780 leader_election.cc:290] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250626 01:58:58.268164 19784 raft_consensus.cc:2802] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 01:58:58.271349 19784 raft_consensus.cc:695] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [term 3 LEADER]: Becoming Leader. State: Replica: eb90a9aba85a4018bf48a8e64992e8f0, State: Running, Role: LEADER
I20250626 01:58:58.272187 19784 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb90a9aba85a4018bf48a8e64992e8f0" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 43009 } }
I20250626 01:58:58.272734 19780 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 01:58:58.282923 19785 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "eb90a9aba85a4018bf48a8e64992e8f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb90a9aba85a4018bf48a8e64992e8f0" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 43009 } } }
I20250626 01:58:58.283643 19785 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [sys.catalog]: This master's current role is: LEADER
I20250626 01:58:58.286069 19786 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb90a9aba85a4018bf48a8e64992e8f0. Latest consensus state: current_term: 3 leader_uuid: "eb90a9aba85a4018bf48a8e64992e8f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb90a9aba85a4018bf48a8e64992e8f0" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 43009 } } }
I20250626 01:58:58.286758 19786 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [sys.catalog]: This master's current role is: LEADER
I20250626 01:58:58.293160 19791 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 01:58:58.308252 19791 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=516aa34a5a1842edbcb93963e32cd775]
I20250626 01:58:58.310593 19791 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=53b4ca6fa1494d44822bc977e13bd91f]
I20250626 01:58:58.312716 19791 catalog_manager.cc:671] Loaded metadata for table TestTable [id=b97c84af241c43559e078eebe41fabc0]
I20250626 01:58:58.323681 19791 tablet_loader.cc:96] loaded metadata for tablet 242b3d8e6a774185a554cf2603c1e5a0 (table TestTable [id=b97c84af241c43559e078eebe41fabc0])
I20250626 01:58:58.325398 19791 tablet_loader.cc:96] loaded metadata for tablet bbe200e5cd43440e83047942538c8c1b (table TestTable1 [id=516aa34a5a1842edbcb93963e32cd775])
I20250626 01:58:58.327080 19791 tablet_loader.cc:96] loaded metadata for tablet d5b1c5efbd044314965eb5c7b1bede3c (table TestTable2 [id=53b4ca6fa1494d44822bc977e13bd91f])
I20250626 01:58:58.329244 19791 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 01:58:58.337482 19791 catalog_manager.cc:1261] Loaded cluster ID: b34015b3edf14cc89b090bae5fb94094
I20250626 01:58:58.337826 19791 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 01:58:58.347262 19801 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250626 01:58:58.348062 19791 catalog_manager.cc:1506] Loading token signing keys...
I20250626 01:58:58.351894 19791 catalog_manager.cc:5966] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0: Loaded TSK: 0
I20250626 01:58:58.353173 19791 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250626 01:58:58.509359 19782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:58:58.509920 19782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 01:58:58.510442 19782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:58:58.542985 19782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 01:58:58.543957 19782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.19.193
I20250626 01:58:58.580224 19782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:42273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.19.193:44115
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.19.193
--webserver_port=44591
--tserver_master_addrs=127.16.19.254:43009
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.19.193
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 01:58:58.581768 19782 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 01:58:58.583535 19782 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:58:58.602144 19808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:58:58.602614 19807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:58:58.605612 19810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:58:58.605113 19782 server_base.cc:1048] running on GCE node
I20250626 01:58:59.773960 19782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 01:58:59.776978 19782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 01:58:59.778466 19782 hybrid_clock.cc:648] HybridClock initialized: now 1750903139778403 us; error 81 us; skew 500 ppm
I20250626 01:58:59.779274 19782 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:58:59.786542 19782 webserver.cc:469] Webserver started at http://127.16.19.193:44591/ using document root <none> and password file <none>
I20250626 01:58:59.787462 19782 fs_manager.cc:362] Metadata directory not provided
I20250626 01:58:59.787700 19782 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:58:59.796020 19782 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20250626 01:58:59.801780 19817 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:58:59.802992 19782 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20250626 01:58:59.803375 19782 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f91b0712e19140bd9b9453efa221e865"
format_stamp: "Formatted at 2025-06-26 01:58:46 on dist-test-slave-h4rq"
I20250626 01:58:59.806202 19782 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:58:59.879035 19782 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:58:59.880589 19782 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 01:58:59.881062 19782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:58:59.883708 19782 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:58:59.889605 19824 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250626 01:58:59.900525 19782 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250626 01:58:59.900815 19782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.000s sys 0.002s
I20250626 01:58:59.901098 19782 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250626 01:58:59.905735 19782 ts_tablet_manager.cc:610] Registered 1 tablets
I20250626 01:58:59.905930 19782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.004s sys 0.000s
I20250626 01:58:59.906361 19824 tablet_bootstrap.cc:492] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865: Bootstrap starting.
I20250626 01:58:59.986950 19824 log.cc:826] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865: Log is configured to *not* fsync() on all Append() calls
I20250626 01:59:00.106007 19782 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.193:44115
I20250626 01:59:00.106225 19931 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.193:44115 every 8 connection(s)
I20250626 01:59:00.109598 19782 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250626 01:59:00.112843 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 19782
I20250626 01:59:00.114840 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.19.194:39201
--local_ip_for_outbound_sockets=127.16.19.194
--tserver_master_addrs=127.16.19.254:43009
--webserver_port=38337
--webserver_interface=127.16.19.194
--builtin_ntp_servers=127.16.19.212:42273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 01:59:00.159072 19824 tablet_bootstrap.cc:492] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:59:00.160202 19824 tablet_bootstrap.cc:492] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865: Bootstrap complete.
I20250626 01:59:00.162001 19824 ts_tablet_manager.cc:1397] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865: Time spent bootstrapping tablet: real 0.256s user 0.223s sys 0.020s
I20250626 01:59:00.168324 19932 heartbeater.cc:344] Connected to a master server at 127.16.19.254:43009
I20250626 01:59:00.168843 19932 heartbeater.cc:461] Registering TS with master...
I20250626 01:59:00.170035 19932 heartbeater.cc:507] Master 127.16.19.254:43009 requested a full tablet report, sending...
I20250626 01:59:00.174681 19745 ts_manager.cc:194] Registered new tserver with Master: f91b0712e19140bd9b9453efa221e865 (127.16.19.193:44115)
I20250626 01:59:00.179620 19824 raft_consensus.cc:357] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } }
I20250626 01:59:00.182790 19824 raft_consensus.cc:738] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f91b0712e19140bd9b9453efa221e865, State: Initialized, Role: FOLLOWER
I20250626 01:59:00.183876 19745 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.19.193:40145
I20250626 01:59:00.183796 19824 consensus_queue.cc:260] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } }
I20250626 01:59:00.184605 19824 raft_consensus.cc:397] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 01:59:00.185058 19824 raft_consensus.cc:491] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 01:59:00.185467 19824 raft_consensus.cc:3058] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:59:00.193464 19824 raft_consensus.cc:513] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } }
I20250626 01:59:00.194424 19824 leader_election.cc:304] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f91b0712e19140bd9b9453efa221e865; no voters:
I20250626 01:59:00.196694 19824 leader_election.cc:290] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250626 01:59:00.197517 19937 raft_consensus.cc:2802] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 01:59:00.206905 19824 ts_tablet_manager.cc:1428] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865: Time spent starting tablet: real 0.045s user 0.040s sys 0.000s
I20250626 01:59:00.207732 19932 heartbeater.cc:499] Master 127.16.19.254:43009 was elected leader, sending a full tablet report...
I20250626 01:59:00.209303 19937 raft_consensus.cc:695] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [term 2 LEADER]: Becoming Leader. State: Replica: f91b0712e19140bd9b9453efa221e865, State: Running, Role: LEADER
I20250626 01:59:00.210192 19937 consensus_queue.cc:237] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } }
I20250626 01:59:00.222410 19745 catalog_manager.cc:5582] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 reported cstate change: term changed from 0 to 2, leader changed from <none> to f91b0712e19140bd9b9453efa221e865 (127.16.19.193), VOTER f91b0712e19140bd9b9453efa221e865 (127.16.19.193) added. New cstate: current_term: 2 leader_uuid: "f91b0712e19140bd9b9453efa221e865" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } health_report { overall_health: HEALTHY } } }
W20250626 01:59:00.273072 19745 catalog_manager.cc:5260] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 242b3d8e6a774185a554cf2603c1e5a0 with cas_config_opid_index -1: no extra replica candidate found for tablet 242b3d8e6a774185a554cf2603c1e5a0 (table TestTable [id=b97c84af241c43559e078eebe41fabc0]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (1) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
W20250626 01:59:00.467761 19936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:59:00.468289 19936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 01:59:00.468850 19936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:59:00.500021 19936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 01:59:00.500933 19936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.19.194
I20250626 01:59:00.535697 19936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:42273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.19.194:39201
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.16.19.194
--webserver_port=38337
--tserver_master_addrs=127.16.19.254:43009
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.19.194
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 01:59:00.537052 19936 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 01:59:00.538677 19936 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:59:00.554049 19951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:59:00.554147 19952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:59:00.554090 19954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:59:01.727646 19953 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 01:59:01.727738 19936 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 01:59:01.731729 19936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 01:59:01.734225 19936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 01:59:01.735560 19936 hybrid_clock.cc:648] HybridClock initialized: now 1750903141735538 us; error 46 us; skew 500 ppm
I20250626 01:59:01.736347 19936 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:59:01.743117 19936 webserver.cc:469] Webserver started at http://127.16.19.194:38337/ using document root <none> and password file <none>
I20250626 01:59:01.744021 19936 fs_manager.cc:362] Metadata directory not provided
I20250626 01:59:01.744246 19936 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:59:01.752104 19936 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20250626 01:59:01.756825 19961 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:59:01.757851 19936 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20250626 01:59:01.758183 19936 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "cf4e54b733f84093b1488c82347f0389"
format_stamp: "Formatted at 2025-06-26 01:58:48 on dist-test-slave-h4rq"
I20250626 01:59:01.760118 19936 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:59:01.819737 19936 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:59:01.821324 19936 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 01:59:01.821758 19936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:59:01.824247 19936 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:59:01.829999 19968 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250626 01:59:01.837658 19936 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250626 01:59:01.837941 19936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.002s
I20250626 01:59:01.838229 19936 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250626 01:59:01.843034 19936 ts_tablet_manager.cc:610] Registered 1 tablets
I20250626 01:59:01.843242 19936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.002s sys 0.001s
I20250626 01:59:01.843624 19968 tablet_bootstrap.cc:492] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389: Bootstrap starting.
I20250626 01:59:01.898514 19968 log.cc:826] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389: Log is configured to *not* fsync() on all Append() calls
I20250626 01:59:01.986173 19968 tablet_bootstrap.cc:492] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:59:01.987337 19968 tablet_bootstrap.cc:492] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389: Bootstrap complete.
I20250626 01:59:01.989179 19968 ts_tablet_manager.cc:1397] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389: Time spent bootstrapping tablet: real 0.146s user 0.085s sys 0.059s
I20250626 01:59:02.006805 19968 raft_consensus.cc:357] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } }
I20250626 01:59:02.009569 19968 raft_consensus.cc:738] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf4e54b733f84093b1488c82347f0389, State: Initialized, Role: FOLLOWER
I20250626 01:59:02.010560 19968 consensus_queue.cc:260] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } }
I20250626 01:59:02.011132 19968 raft_consensus.cc:397] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 01:59:02.011557 19968 raft_consensus.cc:491] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 01:59:02.011926 19968 raft_consensus.cc:3058] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389 [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:59:02.018483 19968 raft_consensus.cc:513] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } }
I20250626 01:59:02.019356 19968 leader_election.cc:304] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf4e54b733f84093b1488c82347f0389; no voters:
I20250626 01:59:02.022282 19968 leader_election.cc:290] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250626 01:59:02.022584 20071 raft_consensus.cc:2802] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 01:59:02.025772 20071 raft_consensus.cc:695] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389 [term 2 LEADER]: Becoming Leader. State: Replica: cf4e54b733f84093b1488c82347f0389, State: Running, Role: LEADER
I20250626 01:59:02.026780 20071 consensus_queue.cc:237] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } }
I20250626 01:59:02.030702 19968 ts_tablet_manager.cc:1428] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389: Time spent starting tablet: real 0.041s user 0.031s sys 0.004s
I20250626 01:59:02.034533 19936 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.194:39201
I20250626 01:59:02.035048 20079 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.194:39201 every 8 connection(s)
I20250626 01:59:02.037796 19936 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250626 01:59:02.039198 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 19936
I20250626 01:59:02.041440 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.19.195:40417
--local_ip_for_outbound_sockets=127.16.19.195
--tserver_master_addrs=127.16.19.254:43009
--webserver_port=39177
--webserver_interface=127.16.19.195
--builtin_ntp_servers=127.16.19.212:42273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 01:59:02.061300 20081 heartbeater.cc:344] Connected to a master server at 127.16.19.254:43009
I20250626 01:59:02.061741 20081 heartbeater.cc:461] Registering TS with master...
I20250626 01:59:02.062713 20081 heartbeater.cc:507] Master 127.16.19.254:43009 requested a full tablet report, sending...
I20250626 01:59:02.066157 19745 ts_manager.cc:194] Registered new tserver with Master: cf4e54b733f84093b1488c82347f0389 (127.16.19.194:39201)
I20250626 01:59:02.067209 19745 catalog_manager.cc:5582] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "cf4e54b733f84093b1488c82347f0389" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } health_report { overall_health: HEALTHY } } }
I20250626 01:59:02.079326 19745 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.19.194:50281
I20250626 01:59:02.082803 20081 heartbeater.cc:499] Master 127.16.19.254:43009 was elected leader, sending a full tablet report...
W20250626 01:59:02.373064 20085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:59:02.373574 20085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 01:59:02.374079 20085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:59:02.405877 20085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 01:59:02.406773 20085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.19.195
I20250626 01:59:02.442967 20085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:42273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.19.195:40417
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.16.19.195
--webserver_port=39177
--tserver_master_addrs=127.16.19.254:43009
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.19.195
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 01:59:02.444347 20085 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 01:59:02.446024 20085 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:59:02.463290 20095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:59:02.588791 19887 consensus_queue.cc:237] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: NON_VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: true } }
I20250626 01:59:02.597474 20105 raft_consensus.cc:2953] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index -1 to 10, NON_VOTER cf4e54b733f84093b1488c82347f0389 (127.16.19.194) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: NON_VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: true } } }
I20250626 01:59:02.611838 19729 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 242b3d8e6a774185a554cf2603c1e5a0 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 8)
I20250626 01:59:02.619338 19745 catalog_manager.cc:5582] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 reported cstate change: config changed from index -1 to 10, NON_VOTER cf4e54b733f84093b1488c82347f0389 (127.16.19.194) added. New cstate: current_term: 2 leader_uuid: "f91b0712e19140bd9b9453efa221e865" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: NON_VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250626 01:59:02.627803 19821 consensus_peers.cc:487] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 -> Peer cf4e54b733f84093b1488c82347f0389 (127.16.19.194:39201): Couldn't send request to peer cf4e54b733f84093b1488c82347f0389. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 242b3d8e6a774185a554cf2603c1e5a0. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:59:03.177278 20112 ts_tablet_manager.cc:927] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389: Initiating tablet copy from peer f91b0712e19140bd9b9453efa221e865 (127.16.19.193:44115)
I20250626 01:59:03.179879 20112 tablet_copy_client.cc:323] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389: tablet copy: Beginning tablet copy session from remote peer at address 127.16.19.193:44115
I20250626 01:59:03.211364 19907 tablet_copy_service.cc:140] P f91b0712e19140bd9b9453efa221e865: Received BeginTabletCopySession request for tablet 242b3d8e6a774185a554cf2603c1e5a0 from peer cf4e54b733f84093b1488c82347f0389 ({username='slave'} at 127.16.19.194:35403)
I20250626 01:59:03.212368 19907 tablet_copy_service.cc:161] P f91b0712e19140bd9b9453efa221e865: Beginning new tablet copy session on tablet 242b3d8e6a774185a554cf2603c1e5a0 from peer cf4e54b733f84093b1488c82347f0389 at {username='slave'} at 127.16.19.194:35403: session id = cf4e54b733f84093b1488c82347f0389-242b3d8e6a774185a554cf2603c1e5a0
I20250626 01:59:03.224532 19907 tablet_copy_source_session.cc:215] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 01:59:03.230290 20112 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 242b3d8e6a774185a554cf2603c1e5a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:59:03.240763 20112 tablet_copy_client.cc:806] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389: tablet copy: Starting download of 0 data blocks...
I20250626 01:59:03.241287 20112 tablet_copy_client.cc:670] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389: tablet copy: Starting download of 1 WAL segments...
I20250626 01:59:03.245944 20112 tablet_copy_client.cc:538] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 01:59:03.251408 20112 tablet_bootstrap.cc:492] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389: Bootstrap starting.
I20250626 01:59:03.339751 20112 tablet_bootstrap.cc:492] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:59:03.340425 20112 tablet_bootstrap.cc:492] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389: Bootstrap complete.
I20250626 01:59:03.340910 20112 ts_tablet_manager.cc:1397] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389: Time spent bootstrapping tablet: real 0.090s user 0.076s sys 0.015s
I20250626 01:59:03.342556 20112 raft_consensus.cc:357] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: NON_VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: true } }
I20250626 01:59:03.342979 20112 raft_consensus.cc:738] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: cf4e54b733f84093b1488c82347f0389, State: Initialized, Role: LEARNER
I20250626 01:59:03.343385 20112 consensus_queue.cc:260] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: NON_VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: true } }
I20250626 01:59:03.347530 20112 ts_tablet_manager.cc:1428] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20250626 01:59:03.349049 19907 tablet_copy_service.cc:342] P f91b0712e19140bd9b9453efa221e865: Request end of tablet copy session cf4e54b733f84093b1488c82347f0389-242b3d8e6a774185a554cf2603c1e5a0 received from {username='slave'} at 127.16.19.194:35403
I20250626 01:59:03.349478 19907 tablet_copy_service.cc:434] P f91b0712e19140bd9b9453efa221e865: ending tablet copy session cf4e54b733f84093b1488c82347f0389-242b3d8e6a774185a554cf2603c1e5a0 on tablet 242b3d8e6a774185a554cf2603c1e5a0 with peer cf4e54b733f84093b1488c82347f0389
W20250626 01:59:02.463629 20096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:59:02.466212 20085 server_base.cc:1048] running on GCE node
W20250626 01:59:02.463632 20098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:59:03.670892 20085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 01:59:03.673540 20085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 01:59:03.674935 20085 hybrid_clock.cc:648] HybridClock initialized: now 1750903143674900 us; error 51 us; skew 500 ppm
I20250626 01:59:03.675776 20085 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:59:03.683722 20031 raft_consensus.cc:1215] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20250626 01:59:03.688601 20085 webserver.cc:469] Webserver started at http://127.16.19.195:39177/ using document root <none> and password file <none>
I20250626 01:59:03.690009 20085 fs_manager.cc:362] Metadata directory not provided
I20250626 01:59:03.690371 20085 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:59:03.698164 20085 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20250626 01:59:03.702551 20125 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:59:03.703531 20085 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250626 01:59:03.703843 20085 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "7ab09cbcdaba4e069a36503512251ef7"
format_stamp: "Formatted at 2025-06-26 01:58:49 on dist-test-slave-h4rq"
I20250626 01:59:03.705785 20085 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:59:03.759653 20085 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:59:03.761154 20085 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 01:59:03.761579 20085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:59:03.764056 20085 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:59:03.770291 20132 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250626 01:59:03.781742 20085 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250626 01:59:03.782078 20085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.001s sys 0.000s
I20250626 01:59:03.782433 20085 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250626 01:59:03.790452 20085 ts_tablet_manager.cc:610] Registered 1 tablets
I20250626 01:59:03.790720 20085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.008s user 0.007s sys 0.000s
I20250626 01:59:03.791127 20132 tablet_bootstrap.cc:492] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7: Bootstrap starting.
I20250626 01:59:03.876973 20132 log.cc:826] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7: Log is configured to *not* fsync() on all Append() calls
I20250626 01:59:03.963771 20085 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.195:40417
I20250626 01:59:03.963862 20239 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.195:40417 every 8 connection(s)
I20250626 01:59:03.967609 20085 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250626 01:59:03.970161 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 20085
I20250626 01:59:04.006191 20240 heartbeater.cc:344] Connected to a master server at 127.16.19.254:43009
I20250626 01:59:04.006623 20240 heartbeater.cc:461] Registering TS with master...
I20250626 01:59:04.007606 20240 heartbeater.cc:507] Master 127.16.19.254:43009 requested a full tablet report, sending...
I20250626 01:59:04.010912 19744 ts_manager.cc:194] Registered new tserver with Master: 7ab09cbcdaba4e069a36503512251ef7 (127.16.19.195:40417)
I20250626 01:59:04.013700 20132 tablet_bootstrap.cc:492] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:59:04.013727 19744 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.19.195:60191
I20250626 01:59:04.014542 20132 tablet_bootstrap.cc:492] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7: Bootstrap complete.
I20250626 01:59:04.014925 16463 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 01:59:04.016167 20132 ts_tablet_manager.cc:1397] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7: Time spent bootstrapping tablet: real 0.226s user 0.180s sys 0.036s
I20250626 01:59:04.019097 16463 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
W20250626 01:59:04.022261 16463 ts_itest-base.cc:209] found only 2 out of 3 replicas of tablet 242b3d8e6a774185a554cf2603c1e5a0: tablet_id: "242b3d8e6a774185a554cf2603c1e5a0" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER } interned_replicas { ts_info_idx: 1 role: LEARNER }
I20250626 01:59:04.028257 20132 raft_consensus.cc:357] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } }
I20250626 01:59:04.030262 20132 raft_consensus.cc:738] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ab09cbcdaba4e069a36503512251ef7, State: Initialized, Role: FOLLOWER
I20250626 01:59:04.031087 20132 consensus_queue.cc:260] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } }
I20250626 01:59:04.031518 20132 raft_consensus.cc:397] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 01:59:04.031790 20132 raft_consensus.cc:491] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 01:59:04.032055 20132 raft_consensus.cc:3058] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:59:04.037147 20132 raft_consensus.cc:513] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } }
I20250626 01:59:04.037756 20132 leader_election.cc:304] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ab09cbcdaba4e069a36503512251ef7; no voters:
I20250626 01:59:04.039528 20132 leader_election.cc:290] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250626 01:59:04.040114 20246 raft_consensus.cc:2802] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 01:59:04.042763 20240 heartbeater.cc:499] Master 127.16.19.254:43009 was elected leader, sending a full tablet report...
I20250626 01:59:04.042860 20132 ts_tablet_manager.cc:1428] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7: Time spent starting tablet: real 0.026s user 0.023s sys 0.006s
I20250626 01:59:04.043936 20246 raft_consensus.cc:695] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [term 2 LEADER]: Becoming Leader. State: Replica: 7ab09cbcdaba4e069a36503512251ef7, State: Running, Role: LEADER
I20250626 01:59:04.044874 20246 consensus_queue.cc:237] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } }
I20250626 01:59:04.057453 19744 catalog_manager.cc:5582] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 reported cstate change: term changed from 0 to 2, leader changed from <none> to 7ab09cbcdaba4e069a36503512251ef7 (127.16.19.195), VOTER 7ab09cbcdaba4e069a36503512251ef7 (127.16.19.195) added. New cstate: current_term: 2 leader_uuid: "7ab09cbcdaba4e069a36503512251ef7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } health_report { overall_health: HEALTHY } } }
I20250626 01:59:04.085325 20195 consensus_queue.cc:237] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } } peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: NON_VOTER last_known_addr { host: "127.16.19.193" port: 44115 } attrs { promote: true } }
I20250626 01:59:04.088433 20248 raft_consensus.cc:2953] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index -1 to 9, NON_VOTER f91b0712e19140bd9b9453efa221e865 (127.16.19.193) added. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } } peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: NON_VOTER last_known_addr { host: "127.16.19.193" port: 44115 } attrs { promote: true } } }
I20250626 01:59:04.096374 19730 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bbe200e5cd43440e83047942538c8c1b with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250626 01:59:04.097729 20126 consensus_peers.cc:487] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 -> Peer f91b0712e19140bd9b9453efa221e865 (127.16.19.193:44115): Couldn't send request to peer f91b0712e19140bd9b9453efa221e865. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bbe200e5cd43440e83047942538c8c1b. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:59:04.098872 19744 catalog_manager.cc:5582] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 reported cstate change: config changed from index -1 to 9, NON_VOTER f91b0712e19140bd9b9453efa221e865 (127.16.19.193) added. New cstate: current_term: 2 leader_uuid: "7ab09cbcdaba4e069a36503512251ef7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: NON_VOTER last_known_addr { host: "127.16.19.193" port: 44115 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 01:59:04.108321 20195 consensus_queue.cc:237] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } } peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: NON_VOTER last_known_addr { host: "127.16.19.193" port: 44115 } attrs { promote: true } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: NON_VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: true } }
I20250626 01:59:04.111553 20246 raft_consensus.cc:2953] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, NON_VOTER cf4e54b733f84093b1488c82347f0389 (127.16.19.194) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } } peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: NON_VOTER last_known_addr { host: "127.16.19.193" port: 44115 } attrs { promote: true } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: NON_VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: true } } }
W20250626 01:59:04.113298 20126 consensus_peers.cc:487] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 -> Peer f91b0712e19140bd9b9453efa221e865 (127.16.19.193:44115): Couldn't send request to peer f91b0712e19140bd9b9453efa221e865. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bbe200e5cd43440e83047942538c8c1b. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:59:04.119275 19730 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bbe200e5cd43440e83047942538c8c1b with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250626 01:59:04.122368 19744 catalog_manager.cc:5582] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 reported cstate change: config changed from index 9 to 10, NON_VOTER cf4e54b733f84093b1488c82347f0389 (127.16.19.194) added. New cstate: current_term: 2 leader_uuid: "7ab09cbcdaba4e069a36503512251ef7" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: NON_VOTER last_known_addr { host: "127.16.19.193" port: 44115 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: NON_VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250626 01:59:04.124043 20129 consensus_peers.cc:487] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 -> Peer cf4e54b733f84093b1488c82347f0389 (127.16.19.194:39201): Couldn't send request to peer cf4e54b733f84093b1488c82347f0389. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bbe200e5cd43440e83047942538c8c1b. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:59:04.266199 20259 raft_consensus.cc:1062] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865: attempting to promote NON_VOTER cf4e54b733f84093b1488c82347f0389 to VOTER
I20250626 01:59:04.267601 20259 consensus_queue.cc:237] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } }
I20250626 01:59:04.271620 20031 raft_consensus.cc:1273] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389 [term 2 LEARNER]: Refusing update from remote peer f91b0712e19140bd9b9453efa221e865: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250626 01:59:04.272773 20260 consensus_queue.cc:1035] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250626 01:59:04.279247 20259 raft_consensus.cc:2953] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, cf4e54b733f84093b1488c82347f0389 (127.16.19.194) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } } }
I20250626 01:59:04.280527 20031 raft_consensus.cc:2953] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, cf4e54b733f84093b1488c82347f0389 (127.16.19.194) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } } }
I20250626 01:59:04.288504 19744 catalog_manager.cc:5582] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 reported cstate change: config changed from index 10 to 11, cf4e54b733f84093b1488c82347f0389 (127.16.19.194) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "f91b0712e19140bd9b9453efa221e865" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 01:59:04.298424 19887 consensus_queue.cc:237] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } } peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: NON_VOTER last_known_addr { host: "127.16.19.195" port: 40417 } attrs { promote: true } }
I20250626 01:59:04.303156 20031 raft_consensus.cc:1273] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389 [term 2 FOLLOWER]: Refusing update from remote peer f91b0712e19140bd9b9453efa221e865: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250626 01:59:04.304769 20260 consensus_queue.cc:1035] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250626 01:59:04.310072 20263 raft_consensus.cc:2953] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 7ab09cbcdaba4e069a36503512251ef7 (127.16.19.195) added. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } } peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: NON_VOTER last_known_addr { host: "127.16.19.195" port: 40417 } attrs { promote: true } } }
I20250626 01:59:04.311472 20031 raft_consensus.cc:2953] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 7ab09cbcdaba4e069a36503512251ef7 (127.16.19.195) added. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } } peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: NON_VOTER last_known_addr { host: "127.16.19.195" port: 40417 } attrs { promote: true } } }
W20250626 01:59:04.312788 19819 consensus_peers.cc:487] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 -> Peer 7ab09cbcdaba4e069a36503512251ef7 (127.16.19.195:40417): Couldn't send request to peer 7ab09cbcdaba4e069a36503512251ef7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 242b3d8e6a774185a554cf2603c1e5a0. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:59:04.319603 19729 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 242b3d8e6a774185a554cf2603c1e5a0 with cas_config_opid_index 11: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250626 01:59:04.322571 19744 catalog_manager.cc:5582] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 reported cstate change: config changed from index 11 to 12, NON_VOTER 7ab09cbcdaba4e069a36503512251ef7 (127.16.19.195) added. New cstate: current_term: 2 leader_uuid: "f91b0712e19140bd9b9453efa221e865" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: NON_VOTER last_known_addr { host: "127.16.19.195" port: 40417 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 01:59:04.492937 20266 ts_tablet_manager.cc:927] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389: Initiating tablet copy from peer 7ab09cbcdaba4e069a36503512251ef7 (127.16.19.195:40417)
I20250626 01:59:04.494131 20266 tablet_copy_client.cc:323] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389: tablet copy: Beginning tablet copy session from remote peer at address 127.16.19.195:40417
I20250626 01:59:04.504241 20215 tablet_copy_service.cc:140] P 7ab09cbcdaba4e069a36503512251ef7: Received BeginTabletCopySession request for tablet bbe200e5cd43440e83047942538c8c1b from peer cf4e54b733f84093b1488c82347f0389 ({username='slave'} at 127.16.19.194:46361)
I20250626 01:59:04.504745 20215 tablet_copy_service.cc:161] P 7ab09cbcdaba4e069a36503512251ef7: Beginning new tablet copy session on tablet bbe200e5cd43440e83047942538c8c1b from peer cf4e54b733f84093b1488c82347f0389 at {username='slave'} at 127.16.19.194:46361: session id = cf4e54b733f84093b1488c82347f0389-bbe200e5cd43440e83047942538c8c1b
I20250626 01:59:04.509466 20215 tablet_copy_source_session.cc:215] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 01:59:04.512334 20266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbe200e5cd43440e83047942538c8c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:59:04.521931 20266 tablet_copy_client.cc:806] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389: tablet copy: Starting download of 0 data blocks...
I20250626 01:59:04.522338 20266 tablet_copy_client.cc:670] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389: tablet copy: Starting download of 1 WAL segments...
I20250626 01:59:04.526204 20266 tablet_copy_client.cc:538] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 01:59:04.531579 20266 tablet_bootstrap.cc:492] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389: Bootstrap starting.
I20250626 01:59:04.612003 20266 tablet_bootstrap.cc:492] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:59:04.612761 20266 tablet_bootstrap.cc:492] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389: Bootstrap complete.
I20250626 01:59:04.613178 20266 ts_tablet_manager.cc:1397] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389: Time spent bootstrapping tablet: real 0.082s user 0.071s sys 0.012s
I20250626 01:59:04.614651 20266 raft_consensus.cc:357] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } } peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: NON_VOTER last_known_addr { host: "127.16.19.193" port: 44115 } attrs { promote: true } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: NON_VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: true } }
I20250626 01:59:04.615121 20266 raft_consensus.cc:738] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: cf4e54b733f84093b1488c82347f0389, State: Initialized, Role: LEARNER
I20250626 01:59:04.615459 20266 consensus_queue.cc:260] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } } peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: NON_VOTER last_known_addr { host: "127.16.19.193" port: 44115 } attrs { promote: true } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: NON_VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: true } }
I20250626 01:59:04.617038 20266 ts_tablet_manager.cc:1428] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250626 01:59:04.618613 20215 tablet_copy_service.cc:342] P 7ab09cbcdaba4e069a36503512251ef7: Request end of tablet copy session cf4e54b733f84093b1488c82347f0389-bbe200e5cd43440e83047942538c8c1b received from {username='slave'} at 127.16.19.194:46361
I20250626 01:59:04.618984 20215 tablet_copy_service.cc:434] P 7ab09cbcdaba4e069a36503512251ef7: ending tablet copy session cf4e54b733f84093b1488c82347f0389-bbe200e5cd43440e83047942538c8c1b on tablet bbe200e5cd43440e83047942538c8c1b with peer cf4e54b733f84093b1488c82347f0389
I20250626 01:59:04.649330 20270 ts_tablet_manager.cc:927] T bbe200e5cd43440e83047942538c8c1b P f91b0712e19140bd9b9453efa221e865: Initiating tablet copy from peer 7ab09cbcdaba4e069a36503512251ef7 (127.16.19.195:40417)
I20250626 01:59:04.651381 20270 tablet_copy_client.cc:323] T bbe200e5cd43440e83047942538c8c1b P f91b0712e19140bd9b9453efa221e865: tablet copy: Beginning tablet copy session from remote peer at address 127.16.19.195:40417
I20250626 01:59:04.652832 20215 tablet_copy_service.cc:140] P 7ab09cbcdaba4e069a36503512251ef7: Received BeginTabletCopySession request for tablet bbe200e5cd43440e83047942538c8c1b from peer f91b0712e19140bd9b9453efa221e865 ({username='slave'} at 127.16.19.193:52871)
I20250626 01:59:04.653223 20215 tablet_copy_service.cc:161] P 7ab09cbcdaba4e069a36503512251ef7: Beginning new tablet copy session on tablet bbe200e5cd43440e83047942538c8c1b from peer f91b0712e19140bd9b9453efa221e865 at {username='slave'} at 127.16.19.193:52871: session id = f91b0712e19140bd9b9453efa221e865-bbe200e5cd43440e83047942538c8c1b
I20250626 01:59:04.657337 20215 tablet_copy_source_session.cc:215] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 01:59:04.659790 20270 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbe200e5cd43440e83047942538c8c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:59:04.669633 20270 tablet_copy_client.cc:806] T bbe200e5cd43440e83047942538c8c1b P f91b0712e19140bd9b9453efa221e865: tablet copy: Starting download of 0 data blocks...
I20250626 01:59:04.670061 20270 tablet_copy_client.cc:670] T bbe200e5cd43440e83047942538c8c1b P f91b0712e19140bd9b9453efa221e865: tablet copy: Starting download of 1 WAL segments...
I20250626 01:59:04.673411 20270 tablet_copy_client.cc:538] T bbe200e5cd43440e83047942538c8c1b P f91b0712e19140bd9b9453efa221e865: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 01:59:04.678755 20270 tablet_bootstrap.cc:492] T bbe200e5cd43440e83047942538c8c1b P f91b0712e19140bd9b9453efa221e865: Bootstrap starting.
I20250626 01:59:04.757328 20270 tablet_bootstrap.cc:492] T bbe200e5cd43440e83047942538c8c1b P f91b0712e19140bd9b9453efa221e865: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:59:04.757951 20270 tablet_bootstrap.cc:492] T bbe200e5cd43440e83047942538c8c1b P f91b0712e19140bd9b9453efa221e865: Bootstrap complete.
I20250626 01:59:04.758375 20270 ts_tablet_manager.cc:1397] T bbe200e5cd43440e83047942538c8c1b P f91b0712e19140bd9b9453efa221e865: Time spent bootstrapping tablet: real 0.080s user 0.068s sys 0.012s
I20250626 01:59:04.760011 20270 raft_consensus.cc:357] T bbe200e5cd43440e83047942538c8c1b P f91b0712e19140bd9b9453efa221e865 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } } peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: NON_VOTER last_known_addr { host: "127.16.19.193" port: 44115 } attrs { promote: true } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: NON_VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: true } }
I20250626 01:59:04.760466 20270 raft_consensus.cc:738] T bbe200e5cd43440e83047942538c8c1b P f91b0712e19140bd9b9453efa221e865 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: f91b0712e19140bd9b9453efa221e865, State: Initialized, Role: LEARNER
I20250626 01:59:04.760902 20270 consensus_queue.cc:260] T bbe200e5cd43440e83047942538c8c1b P f91b0712e19140bd9b9453efa221e865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } } peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: NON_VOTER last_known_addr { host: "127.16.19.193" port: 44115 } attrs { promote: true } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: NON_VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: true } }
I20250626 01:59:04.762451 20270 ts_tablet_manager.cc:1428] T bbe200e5cd43440e83047942538c8c1b P f91b0712e19140bd9b9453efa221e865: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250626 01:59:04.764163 20215 tablet_copy_service.cc:342] P 7ab09cbcdaba4e069a36503512251ef7: Request end of tablet copy session f91b0712e19140bd9b9453efa221e865-bbe200e5cd43440e83047942538c8c1b received from {username='slave'} at 127.16.19.193:52871
I20250626 01:59:04.764586 20215 tablet_copy_service.cc:434] P 7ab09cbcdaba4e069a36503512251ef7: ending tablet copy session f91b0712e19140bd9b9453efa221e865-bbe200e5cd43440e83047942538c8c1b on tablet bbe200e5cd43440e83047942538c8c1b with peer f91b0712e19140bd9b9453efa221e865
I20250626 01:59:04.791189 20273 ts_tablet_manager.cc:927] T 242b3d8e6a774185a554cf2603c1e5a0 P 7ab09cbcdaba4e069a36503512251ef7: Initiating tablet copy from peer f91b0712e19140bd9b9453efa221e865 (127.16.19.193:44115)
I20250626 01:59:04.793033 20273 tablet_copy_client.cc:323] T 242b3d8e6a774185a554cf2603c1e5a0 P 7ab09cbcdaba4e069a36503512251ef7: tablet copy: Beginning tablet copy session from remote peer at address 127.16.19.193:44115
I20250626 01:59:04.794494 19907 tablet_copy_service.cc:140] P f91b0712e19140bd9b9453efa221e865: Received BeginTabletCopySession request for tablet 242b3d8e6a774185a554cf2603c1e5a0 from peer 7ab09cbcdaba4e069a36503512251ef7 ({username='slave'} at 127.16.19.195:55257)
I20250626 01:59:04.794885 19907 tablet_copy_service.cc:161] P f91b0712e19140bd9b9453efa221e865: Beginning new tablet copy session on tablet 242b3d8e6a774185a554cf2603c1e5a0 from peer 7ab09cbcdaba4e069a36503512251ef7 at {username='slave'} at 127.16.19.195:55257: session id = 7ab09cbcdaba4e069a36503512251ef7-242b3d8e6a774185a554cf2603c1e5a0
I20250626 01:59:04.799226 19907 tablet_copy_source_session.cc:215] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 01:59:04.801837 20273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 242b3d8e6a774185a554cf2603c1e5a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:59:04.815035 20273 tablet_copy_client.cc:806] T 242b3d8e6a774185a554cf2603c1e5a0 P 7ab09cbcdaba4e069a36503512251ef7: tablet copy: Starting download of 0 data blocks...
I20250626 01:59:04.815529 20273 tablet_copy_client.cc:670] T 242b3d8e6a774185a554cf2603c1e5a0 P 7ab09cbcdaba4e069a36503512251ef7: tablet copy: Starting download of 1 WAL segments...
I20250626 01:59:04.818971 20273 tablet_copy_client.cc:538] T 242b3d8e6a774185a554cf2603c1e5a0 P 7ab09cbcdaba4e069a36503512251ef7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 01:59:04.824275 20273 tablet_bootstrap.cc:492] T 242b3d8e6a774185a554cf2603c1e5a0 P 7ab09cbcdaba4e069a36503512251ef7: Bootstrap starting.
I20250626 01:59:04.853778 19731 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 242b3d8e6a774185a554cf2603c1e5a0 with cas_config_opid_index 10: aborting the task: latest config opid_index 12; task opid_index 10
I20250626 01:59:04.915216 20273 tablet_bootstrap.cc:492] T 242b3d8e6a774185a554cf2603c1e5a0 P 7ab09cbcdaba4e069a36503512251ef7: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:59:04.915870 20273 tablet_bootstrap.cc:492] T 242b3d8e6a774185a554cf2603c1e5a0 P 7ab09cbcdaba4e069a36503512251ef7: Bootstrap complete.
I20250626 01:59:04.916319 20273 ts_tablet_manager.cc:1397] T 242b3d8e6a774185a554cf2603c1e5a0 P 7ab09cbcdaba4e069a36503512251ef7: Time spent bootstrapping tablet: real 0.092s user 0.080s sys 0.012s
I20250626 01:59:04.918035 20273 raft_consensus.cc:357] T 242b3d8e6a774185a554cf2603c1e5a0 P 7ab09cbcdaba4e069a36503512251ef7 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } } peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: NON_VOTER last_known_addr { host: "127.16.19.195" port: 40417 } attrs { promote: true } }
I20250626 01:59:04.918534 20273 raft_consensus.cc:738] T 242b3d8e6a774185a554cf2603c1e5a0 P 7ab09cbcdaba4e069a36503512251ef7 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 7ab09cbcdaba4e069a36503512251ef7, State: Initialized, Role: LEARNER
I20250626 01:59:04.918955 20273 consensus_queue.cc:260] T 242b3d8e6a774185a554cf2603c1e5a0 P 7ab09cbcdaba4e069a36503512251ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } } peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: NON_VOTER last_known_addr { host: "127.16.19.195" port: 40417 } attrs { promote: true } }
I20250626 01:59:04.920346 20273 ts_tablet_manager.cc:1428] T 242b3d8e6a774185a554cf2603c1e5a0 P 7ab09cbcdaba4e069a36503512251ef7: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250626 01:59:04.921782 19907 tablet_copy_service.cc:342] P f91b0712e19140bd9b9453efa221e865: Request end of tablet copy session 7ab09cbcdaba4e069a36503512251ef7-242b3d8e6a774185a554cf2603c1e5a0 received from {username='slave'} at 127.16.19.195:55257
I20250626 01:59:04.922101 19907 tablet_copy_service.cc:434] P f91b0712e19140bd9b9453efa221e865: ending tablet copy session 7ab09cbcdaba4e069a36503512251ef7-242b3d8e6a774185a554cf2603c1e5a0 on tablet 242b3d8e6a774185a554cf2603c1e5a0 with peer 7ab09cbcdaba4e069a36503512251ef7
I20250626 01:59:04.950852 20031 raft_consensus.cc:1215] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20250626 01:59:05.026914 16463 ts_itest-base.cc:246] Waiting for 1 tablets on tserver f91b0712e19140bd9b9453efa221e865 to finish bootstrapping
I20250626 01:59:05.040647 16463 ts_itest-base.cc:246] Waiting for 1 tablets on tserver cf4e54b733f84093b1488c82347f0389 to finish bootstrapping
I20250626 01:59:05.054874 16463 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 7ab09cbcdaba4e069a36503512251ef7 to finish bootstrapping
I20250626 01:59:05.085217 19887 raft_consensus.cc:1215] T bbe200e5cd43440e83047942538c8c1b P f91b0712e19140bd9b9453efa221e865 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20250626 01:59:05.195237 20195 raft_consensus.cc:1215] T 242b3d8e6a774185a554cf2603c1e5a0 P 7ab09cbcdaba4e069a36503512251ef7 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
I20250626 01:59:05.336694 20174 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250626 01:59:05.340215 20011 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250626 01:59:05.343972 19866 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250626 01:59:05.411237 20277 raft_consensus.cc:1062] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7: attempting to promote NON_VOTER cf4e54b733f84093b1488c82347f0389 to VOTER
I20250626 01:59:05.413609 20277 consensus_queue.cc:237] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } } peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: NON_VOTER last_known_addr { host: "127.16.19.193" port: 44115 } attrs { promote: true } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } }
I20250626 01:59:05.429405 20031 raft_consensus.cc:1273] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389 [term 2 LEARNER]: Refusing update from remote peer 7ab09cbcdaba4e069a36503512251ef7: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250626 01:59:05.431161 20248 consensus_queue.cc:1035] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250626 01:59:05.441107 19887 raft_consensus.cc:1273] T bbe200e5cd43440e83047942538c8c1b P f91b0712e19140bd9b9453efa221e865 [term 2 LEARNER]: Refusing update from remote peer 7ab09cbcdaba4e069a36503512251ef7: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250626 01:59:05.442870 20248 consensus_queue.cc:1035] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: NON_VOTER last_known_addr { host: "127.16.19.193" port: 44115 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250626 01:59:05.460587 20248 raft_consensus.cc:2953] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, cf4e54b733f84093b1488c82347f0389 (127.16.19.194) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } } peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: NON_VOTER last_known_addr { host: "127.16.19.193" port: 44115 } attrs { promote: true } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } } }
I20250626 01:59:05.473212 19887 raft_consensus.cc:2953] T bbe200e5cd43440e83047942538c8c1b P f91b0712e19140bd9b9453efa221e865 [term 2 LEARNER]: Committing config change with OpId 2.11: config changed from index 10 to 11, cf4e54b733f84093b1488c82347f0389 (127.16.19.194) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } } peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: NON_VOTER last_known_addr { host: "127.16.19.193" port: 44115 } attrs { promote: true } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } } }
I20250626 01:59:05.483433 20031 raft_consensus.cc:2953] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, cf4e54b733f84093b1488c82347f0389 (127.16.19.194) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } } peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: NON_VOTER last_known_addr { host: "127.16.19.193" port: 44115 } attrs { promote: true } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } } }
I20250626 01:59:05.496951 20277 raft_consensus.cc:1062] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7: attempting to promote NON_VOTER f91b0712e19140bd9b9453efa221e865 to VOTER
I20250626 01:59:05.498677 20277 consensus_queue.cc:237] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } } peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } attrs { promote: false } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } }
I20250626 01:59:05.498541 19743 catalog_manager.cc:5582] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 reported cstate change: config changed from index 10 to 11, cf4e54b733f84093b1488c82347f0389 (127.16.19.194) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "7ab09cbcdaba4e069a36503512251ef7" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: NON_VOTER last_known_addr { host: "127.16.19.193" port: 44115 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 01:59:05.510006 19886 raft_consensus.cc:1273] T bbe200e5cd43440e83047942538c8c1b P f91b0712e19140bd9b9453efa221e865 [term 2 LEARNER]: Refusing update from remote peer 7ab09cbcdaba4e069a36503512251ef7: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250626 01:59:05.517664 20030 raft_consensus.cc:1273] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389 [term 2 FOLLOWER]: Refusing update from remote peer 7ab09cbcdaba4e069a36503512251ef7: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250626 01:59:05.518213 20277 consensus_queue.cc:1035] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250626 01:59:05.522146 20248 consensus_queue.cc:1035] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20250626 01:59:05.524794 20276 raft_consensus.cc:2953] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, f91b0712e19140bd9b9453efa221e865 (127.16.19.193) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } } peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } attrs { promote: false } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } } }
I20250626 01:59:05.532601 20031 raft_consensus.cc:2953] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, f91b0712e19140bd9b9453efa221e865 (127.16.19.193) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } } peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } attrs { promote: false } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } } }
I20250626 01:59:05.534562 19886 raft_consensus.cc:2953] T bbe200e5cd43440e83047942538c8c1b P f91b0712e19140bd9b9453efa221e865 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, f91b0712e19140bd9b9453efa221e865 (127.16.19.193) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } } peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } attrs { promote: false } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } } }
I20250626 01:59:05.553443 19745 catalog_manager.cc:5582] T bbe200e5cd43440e83047942538c8c1b P f91b0712e19140bd9b9453efa221e865 reported cstate change: config changed from index 11 to 12, f91b0712e19140bd9b9453efa221e865 (127.16.19.193) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "7ab09cbcdaba4e069a36503512251ef7" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } } peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } attrs { promote: false } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } } }
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
eb90a9aba85a4018bf48a8e64992e8f0 | 127.16.19.254:43009 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.16.19.212:42273 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
7ab09cbcdaba4e069a36503512251ef7 | 127.16.19.195:40417 | HEALTHY | <none> | 1 | 0
cf4e54b733f84093b1488c82347f0389 | 127.16.19.194:39201 | HEALTHY | <none> | 1 | 0
f91b0712e19140bd9b9453efa221e865 | 127.16.19.193:44115 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.16.19.193 | experimental | 127.16.19.193:44115
local_ip_for_outbound_sockets | 127.16.19.194 | experimental | 127.16.19.194:39201
local_ip_for_outbound_sockets | 127.16.19.195 | experimental | 127.16.19.195:40417
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.16.19.193:44115
server_dump_info_path | /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.16.19.194:39201
server_dump_info_path | /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.16.19.195:40417
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.16.19.212:42273 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable1 | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable2 | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 3
Tablets | 3
Replicas | 7
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250626 01:59:05.619449 16463 log_verifier.cc:126] Checking tablet 242b3d8e6a774185a554cf2603c1e5a0
I20250626 01:59:05.719117 16463 log_verifier.cc:177] Verified matching terms for 12 ops in tablet 242b3d8e6a774185a554cf2603c1e5a0
I20250626 01:59:05.719470 16463 log_verifier.cc:126] Checking tablet bbe200e5cd43440e83047942538c8c1b
I20250626 01:59:05.721470 20289 raft_consensus.cc:1062] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865: attempting to promote NON_VOTER 7ab09cbcdaba4e069a36503512251ef7 to VOTER
I20250626 01:59:05.722888 20289 consensus_queue.cc:237] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } } peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } attrs { promote: false } }
I20250626 01:59:05.728404 20195 raft_consensus.cc:1273] T 242b3d8e6a774185a554cf2603c1e5a0 P 7ab09cbcdaba4e069a36503512251ef7 [term 2 LEARNER]: Refusing update from remote peer f91b0712e19140bd9b9453efa221e865: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250626 01:59:05.729763 20031 raft_consensus.cc:1273] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389 [term 2 FOLLOWER]: Refusing update from remote peer f91b0712e19140bd9b9453efa221e865: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250626 01:59:05.729745 20289 consensus_queue.cc:1035] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250626 01:59:05.731118 20317 consensus_queue.cc:1035] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20250626 01:59:05.736681 20289 raft_consensus.cc:2953] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 7ab09cbcdaba4e069a36503512251ef7 (127.16.19.195) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } } peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } attrs { promote: false } } }
I20250626 01:59:05.739042 20195 raft_consensus.cc:2953] T 242b3d8e6a774185a554cf2603c1e5a0 P 7ab09cbcdaba4e069a36503512251ef7 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 7ab09cbcdaba4e069a36503512251ef7 (127.16.19.195) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } } peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } attrs { promote: false } } }
I20250626 01:59:05.743346 20031 raft_consensus.cc:2953] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 7ab09cbcdaba4e069a36503512251ef7 (127.16.19.195) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } } peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } attrs { promote: false } } }
I20250626 01:59:05.750188 19744 catalog_manager.cc:5582] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 reported cstate change: config changed from index 12 to 13, 7ab09cbcdaba4e069a36503512251ef7 (127.16.19.195) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "f91b0712e19140bd9b9453efa221e865" committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 01:59:05.817112 16463 log_verifier.cc:177] Verified matching terms for 12 ops in tablet bbe200e5cd43440e83047942538c8c1b
I20250626 01:59:05.817356 16463 log_verifier.cc:126] Checking tablet d5b1c5efbd044314965eb5c7b1bede3c
I20250626 01:59:05.843359 16463 log_verifier.cc:177] Verified matching terms for 7 ops in tablet d5b1c5efbd044314965eb5c7b1bede3c
I20250626 01:59:05.843724 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 19711
I20250626 01:59:05.873351 16463 minidump.cc:252] Setting minidump size limit to 20M
I20250626 01:59:05.874547 16463 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 01:59:05.875538 16463 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:59:05.886231 20323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:59:05.886271 20324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:59:05.887831 20326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:59:05.985419 16463 server_base.cc:1048] running on GCE node
I20250626 01:59:05.986562 16463 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:59:05.986779 16463 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:59:05.986963 16463 hybrid_clock.cc:648] HybridClock initialized: now 1750903145986938 us; error 0 us; skew 500 ppm
I20250626 01:59:05.987596 16463 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:59:05.991142 16463 webserver.cc:469] Webserver started at http://0.0.0.0:33279/ using document root <none> and password file <none>
I20250626 01:59:05.991945 16463 fs_manager.cc:362] Metadata directory not provided
I20250626 01:59:05.992156 16463 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:59:05.997519 16463 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20250626 01:59:06.001183 20331 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:59:06.002135 16463 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250626 01:59:06.002430 16463 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
uuid: "eb90a9aba85a4018bf48a8e64992e8f0"
format_stamp: "Formatted at 2025-06-26 01:58:44 on dist-test-slave-h4rq"
I20250626 01:59:06.004022 16463 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:59:06.030313 16463 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:59:06.031884 16463 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 01:59:06.032313 16463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:59:06.041502 16463 sys_catalog.cc:263] Verifying existing consensus state
W20250626 01:59:06.045817 16463 sys_catalog.cc:243] For a single master config, on-disk Raft master: 127.16.19.254:43009 exists but no master address supplied!
I20250626 01:59:06.047737 16463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0: Bootstrap starting.
I20250626 01:59:06.098917 16463 log.cc:826] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0: Log is configured to *not* fsync() on all Append() calls
I20250626 01:59:06.159536 16463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=13 ignored=0} mutations{seen=21 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:59:06.160320 16463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0: Bootstrap complete.
I20250626 01:59:06.173290 16463 raft_consensus.cc:357] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb90a9aba85a4018bf48a8e64992e8f0" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 43009 } }
I20250626 01:59:06.173858 16463 raft_consensus.cc:738] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb90a9aba85a4018bf48a8e64992e8f0, State: Initialized, Role: FOLLOWER
I20250626 01:59:06.174557 16463 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb90a9aba85a4018bf48a8e64992e8f0" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 43009 } }
I20250626 01:59:06.175017 16463 raft_consensus.cc:397] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [term 3 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 01:59:06.175259 16463 raft_consensus.cc:491] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [term 3 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 01:59:06.175563 16463 raft_consensus.cc:3058] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [term 3 FOLLOWER]: Advancing to term 4
I20250626 01:59:06.180783 16463 raft_consensus.cc:513] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb90a9aba85a4018bf48a8e64992e8f0" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 43009 } }
I20250626 01:59:06.181455 16463 leader_election.cc:304] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb90a9aba85a4018bf48a8e64992e8f0; no voters:
I20250626 01:59:06.182536 16463 leader_election.cc:290] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [CANDIDATE]: Term 4 election: Requested vote from peers
I20250626 01:59:06.182826 20338 raft_consensus.cc:2802] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 01:59:06.184087 20338 raft_consensus.cc:695] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [term 4 LEADER]: Becoming Leader. State: Replica: eb90a9aba85a4018bf48a8e64992e8f0, State: Running, Role: LEADER
I20250626 01:59:06.184823 20338 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb90a9aba85a4018bf48a8e64992e8f0" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 43009 } }
I20250626 01:59:06.191818 20340 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb90a9aba85a4018bf48a8e64992e8f0. Latest consensus state: current_term: 4 leader_uuid: "eb90a9aba85a4018bf48a8e64992e8f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb90a9aba85a4018bf48a8e64992e8f0" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 43009 } } }
I20250626 01:59:06.192325 20340 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [sys.catalog]: This master's current role is: LEADER
I20250626 01:59:06.192224 20339 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 leader_uuid: "eb90a9aba85a4018bf48a8e64992e8f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb90a9aba85a4018bf48a8e64992e8f0" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 43009 } } }
I20250626 01:59:06.192775 20339 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [sys.catalog]: This master's current role is: LEADER
I20250626 01:59:06.219444 16463 tablet_replica.cc:331] stopping tablet replica
I20250626 01:59:06.220273 16463 raft_consensus.cc:2241] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [term 4 LEADER]: Raft consensus shutting down.
I20250626 01:59:06.220827 16463 raft_consensus.cc:2270] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250626 01:59:06.223156 16463 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250626 01:59:06.223664 16463 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250626 01:59:06.253815 16463 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
W20250626 01:59:06.778985 20081 heartbeater.cc:646] Failed to heartbeat to 127.16.19.254:43009 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.19.254:43009: connect: Connection refused (error 111)
W20250626 01:59:06.801810 19932 heartbeater.cc:646] Failed to heartbeat to 127.16.19.254:43009 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.19.254:43009: connect: Connection refused (error 111)
W20250626 01:59:06.804831 20240 heartbeater.cc:646] Failed to heartbeat to 127.16.19.254:43009 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.19.254:43009: connect: Connection refused (error 111)
I20250626 01:59:11.793813 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 19782
I20250626 01:59:11.820293 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 19936
I20250626 01:59:11.849606 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 20085
I20250626 01:59:11.879179 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.19.254:43009
--webserver_interface=127.16.19.254
--webserver_port=40755
--builtin_ntp_servers=127.16.19.212:42273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.19.254:43009 with env {}
W20250626 01:59:12.181421 20421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:59:12.182018 20421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 01:59:12.182490 20421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:59:12.213721 20421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 01:59:12.214056 20421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 01:59:12.214313 20421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 01:59:12.214557 20421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 01:59:12.250597 20421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:42273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.19.254:43009
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.19.254:43009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.19.254
--webserver_port=40755
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 01:59:12.251917 20421 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 01:59:12.253551 20421 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:59:12.267155 20427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:59:12.267413 20428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:59:13.445792 20429 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250626 01:59:13.447619 20430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:59:13.447670 20421 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 01:59:13.450106 20421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 01:59:13.452584 20421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 01:59:13.453958 20421 hybrid_clock.cc:648] HybridClock initialized: now 1750903153453922 us; error 48 us; skew 500 ppm
I20250626 01:59:13.454767 20421 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:59:13.461120 20421 webserver.cc:469] Webserver started at http://127.16.19.254:40755/ using document root <none> and password file <none>
I20250626 01:59:13.462009 20421 fs_manager.cc:362] Metadata directory not provided
I20250626 01:59:13.462219 20421 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:59:13.470696 20421 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.000s sys 0.005s
I20250626 01:59:13.476835 20437 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:59:13.477810 20421 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250626 01:59:13.478140 20421 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
uuid: "eb90a9aba85a4018bf48a8e64992e8f0"
format_stamp: "Formatted at 2025-06-26 01:58:44 on dist-test-slave-h4rq"
I20250626 01:59:13.480118 20421 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:59:13.528249 20421 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:59:13.529716 20421 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 01:59:13.530156 20421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:59:13.601539 20421 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.254:43009
I20250626 01:59:13.601610 20488 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.254:43009 every 8 connection(s)
I20250626 01:59:13.604369 20421 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250626 01:59:13.613013 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 20421
I20250626 01:59:13.614207 20489 sys_catalog.cc:263] Verifying existing consensus state
I20250626 01:59:13.614589 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.19.193:44115
--local_ip_for_outbound_sockets=127.16.19.193
--tserver_master_addrs=127.16.19.254:43009
--webserver_port=44591
--webserver_interface=127.16.19.193
--builtin_ntp_servers=127.16.19.212:42273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 01:59:13.618884 20489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0: Bootstrap starting.
I20250626 01:59:13.628862 20489 log.cc:826] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0: Log is configured to *not* fsync() on all Append() calls
I20250626 01:59:13.705222 20489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=15 ignored=0} mutations{seen=23 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:59:13.705976 20489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0: Bootstrap complete.
I20250626 01:59:13.724884 20489 raft_consensus.cc:357] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb90a9aba85a4018bf48a8e64992e8f0" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 43009 } }
I20250626 01:59:13.726858 20489 raft_consensus.cc:738] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb90a9aba85a4018bf48a8e64992e8f0, State: Initialized, Role: FOLLOWER
I20250626 01:59:13.727591 20489 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb90a9aba85a4018bf48a8e64992e8f0" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 43009 } }
I20250626 01:59:13.728065 20489 raft_consensus.cc:397] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [term 5 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 01:59:13.728315 20489 raft_consensus.cc:491] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [term 5 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 01:59:13.728587 20489 raft_consensus.cc:3058] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [term 5 FOLLOWER]: Advancing to term 6
I20250626 01:59:13.733597 20489 raft_consensus.cc:513] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb90a9aba85a4018bf48a8e64992e8f0" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 43009 } }
I20250626 01:59:13.734246 20489 leader_election.cc:304] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb90a9aba85a4018bf48a8e64992e8f0; no voters:
I20250626 01:59:13.736325 20489 leader_election.cc:290] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [CANDIDATE]: Term 6 election: Requested vote from peers
I20250626 01:59:13.736764 20493 raft_consensus.cc:2802] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [term 6 FOLLOWER]: Leader election won for term 6
I20250626 01:59:13.739882 20493 raft_consensus.cc:695] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [term 6 LEADER]: Becoming Leader. State: Replica: eb90a9aba85a4018bf48a8e64992e8f0, State: Running, Role: LEADER
I20250626 01:59:13.740832 20493 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 6, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb90a9aba85a4018bf48a8e64992e8f0" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 43009 } }
I20250626 01:59:13.741209 20489 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 01:59:13.751470 20495 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb90a9aba85a4018bf48a8e64992e8f0. Latest consensus state: current_term: 6 leader_uuid: "eb90a9aba85a4018bf48a8e64992e8f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb90a9aba85a4018bf48a8e64992e8f0" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 43009 } } }
I20250626 01:59:13.752863 20494 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 leader_uuid: "eb90a9aba85a4018bf48a8e64992e8f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb90a9aba85a4018bf48a8e64992e8f0" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 43009 } } }
I20250626 01:59:13.753561 20494 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [sys.catalog]: This master's current role is: LEADER
I20250626 01:59:13.756769 20495 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0 [sys.catalog]: This master's current role is: LEADER
I20250626 01:59:13.766937 20500 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 01:59:13.781836 20500 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=516aa34a5a1842edbcb93963e32cd775]
I20250626 01:59:13.783960 20500 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=53b4ca6fa1494d44822bc977e13bd91f]
I20250626 01:59:13.785836 20500 catalog_manager.cc:671] Loaded metadata for table TestTable [id=653cf9b21d7c405295d3deef89704bc2]
I20250626 01:59:13.794764 20500 tablet_loader.cc:96] loaded metadata for tablet 242b3d8e6a774185a554cf2603c1e5a0 (table TestTable [id=653cf9b21d7c405295d3deef89704bc2])
I20250626 01:59:13.796450 20500 tablet_loader.cc:96] loaded metadata for tablet bbe200e5cd43440e83047942538c8c1b (table TestTable1 [id=516aa34a5a1842edbcb93963e32cd775])
I20250626 01:59:13.799191 20500 tablet_loader.cc:96] loaded metadata for tablet d5b1c5efbd044314965eb5c7b1bede3c (table TestTable2 [id=53b4ca6fa1494d44822bc977e13bd91f])
I20250626 01:59:13.800791 20500 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 01:59:13.806826 20500 catalog_manager.cc:1261] Loaded cluster ID: b34015b3edf14cc89b090bae5fb94094
I20250626 01:59:13.807181 20500 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 01:59:13.816424 20500 catalog_manager.cc:1506] Loading token signing keys...
I20250626 01:59:13.816982 20510 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250626 01:59:13.822809 20500 catalog_manager.cc:5966] T 00000000000000000000000000000000 P eb90a9aba85a4018bf48a8e64992e8f0: Loaded TSK: 0
I20250626 01:59:13.824406 20500 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250626 01:59:13.977370 20491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:59:13.977900 20491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 01:59:13.978430 20491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:59:14.010632 20491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 01:59:14.011539 20491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.19.193
I20250626 01:59:14.047286 20491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:42273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.19.193:44115
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.19.193
--webserver_port=44591
--tserver_master_addrs=127.16.19.254:43009
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.19.193
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 01:59:14.048599 20491 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 01:59:14.050316 20491 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:59:14.072402 20517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:59:15.468452 20515 debug-util.cc:398] Leaking SignalData structure 0x7b08000184e0 after lost signal to thread 20491
W20250626 01:59:14.073412 20516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:59:15.680490 20491 thread.cc:640] OpenStack (cloud detector) Time spent creating pthread: real 1.613s user 0.637s sys 0.966s
W20250626 01:59:15.681245 20491 thread.cc:606] OpenStack (cloud detector) Time spent starting thread: real 1.614s user 0.637s sys 0.966s
W20250626 01:59:15.682909 20519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:59:15.685964 20518 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 01:59:15.685983 20491 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 01:59:15.687285 20491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 01:59:15.689620 20491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 01:59:15.690994 20491 hybrid_clock.cc:648] HybridClock initialized: now 1750903155690940 us; error 75 us; skew 500 ppm
I20250626 01:59:15.691800 20491 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:59:15.702598 20491 webserver.cc:469] Webserver started at http://127.16.19.193:44591/ using document root <none> and password file <none>
I20250626 01:59:15.703600 20491 fs_manager.cc:362] Metadata directory not provided
I20250626 01:59:15.703850 20491 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:59:15.712441 20491 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20250626 01:59:15.718376 20526 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:59:15.719568 20491 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.003s
I20250626 01:59:15.719889 20491 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f91b0712e19140bd9b9453efa221e865"
format_stamp: "Formatted at 2025-06-26 01:58:46 on dist-test-slave-h4rq"
I20250626 01:59:15.721916 20491 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:59:15.770165 20491 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:59:15.771628 20491 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 01:59:15.772047 20491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:59:15.775183 20491 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:59:15.781782 20533 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250626 01:59:15.793437 20491 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250626 01:59:15.793661 20491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.003s sys 0.000s
I20250626 01:59:15.793964 20491 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250626 01:59:15.799289 20533 tablet_bootstrap.cc:492] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865: Bootstrap starting.
I20250626 01:59:15.804163 20491 ts_tablet_manager.cc:610] Registered 2 tablets
I20250626 01:59:15.804476 20491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.011s user 0.004s sys 0.003s
I20250626 01:59:15.881012 20533 log.cc:826] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865: Log is configured to *not* fsync() on all Append() calls
I20250626 01:59:15.971436 20491 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.193:44115
I20250626 01:59:15.971505 20640 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.193:44115 every 8 connection(s)
I20250626 01:59:15.975193 20491 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250626 01:59:15.977916 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 20491
I20250626 01:59:15.979871 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.19.194:39201
--local_ip_for_outbound_sockets=127.16.19.194
--tserver_master_addrs=127.16.19.254:43009
--webserver_port=38337
--webserver_interface=127.16.19.194
--builtin_ntp_servers=127.16.19.212:42273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 01:59:16.014802 20641 heartbeater.cc:344] Connected to a master server at 127.16.19.254:43009
I20250626 01:59:16.015313 20641 heartbeater.cc:461] Registering TS with master...
I20250626 01:59:16.016537 20641 heartbeater.cc:507] Master 127.16.19.254:43009 requested a full tablet report, sending...
I20250626 01:59:16.021780 20454 ts_manager.cc:194] Registered new tserver with Master: f91b0712e19140bd9b9453efa221e865 (127.16.19.193:44115)
I20250626 01:59:16.026113 20454 catalog_manager.cc:5582] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 reported cstate change: config changed from index -1 to 13, term changed from 0 to 2, VOTER 7ab09cbcdaba4e069a36503512251ef7 (127.16.19.195) added, VOTER cf4e54b733f84093b1488c82347f0389 (127.16.19.194) added, VOTER f91b0712e19140bd9b9453efa221e865 (127.16.19.193) added. New cstate: current_term: 2 committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } } peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } attrs { promote: false } } }
I20250626 01:59:16.039155 20533 tablet_bootstrap.cc:492] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:59:16.040156 20533 tablet_bootstrap.cc:492] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865: Bootstrap complete.
I20250626 01:59:16.041507 20533 ts_tablet_manager.cc:1397] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865: Time spent bootstrapping tablet: real 0.243s user 0.193s sys 0.036s
I20250626 01:59:16.055102 20533 raft_consensus.cc:357] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } } peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } attrs { promote: false } }
I20250626 01:59:16.058492 20533 raft_consensus.cc:738] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f91b0712e19140bd9b9453efa221e865, State: Initialized, Role: FOLLOWER
I20250626 01:59:16.059535 20533 consensus_queue.cc:260] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } } peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } attrs { promote: false } }
I20250626 01:59:16.069412 20533 ts_tablet_manager.cc:1428] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865: Time spent starting tablet: real 0.028s user 0.022s sys 0.004s
I20250626 01:59:16.070293 20533 tablet_bootstrap.cc:492] T bbe200e5cd43440e83047942538c8c1b P f91b0712e19140bd9b9453efa221e865: Bootstrap starting.
I20250626 01:59:16.082218 20454 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.19.193:35859
I20250626 01:59:16.085932 20641 heartbeater.cc:499] Master 127.16.19.254:43009 was elected leader, sending a full tablet report...
I20250626 01:59:16.187438 20533 tablet_bootstrap.cc:492] T bbe200e5cd43440e83047942538c8c1b P f91b0712e19140bd9b9453efa221e865: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:59:16.188133 20533 tablet_bootstrap.cc:492] T bbe200e5cd43440e83047942538c8c1b P f91b0712e19140bd9b9453efa221e865: Bootstrap complete.
I20250626 01:59:16.189329 20533 ts_tablet_manager.cc:1397] T bbe200e5cd43440e83047942538c8c1b P f91b0712e19140bd9b9453efa221e865: Time spent bootstrapping tablet: real 0.119s user 0.107s sys 0.008s
I20250626 01:59:16.190865 20533 raft_consensus.cc:357] T bbe200e5cd43440e83047942538c8c1b P f91b0712e19140bd9b9453efa221e865 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } } peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } attrs { promote: false } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } }
I20250626 01:59:16.191313 20533 raft_consensus.cc:738] T bbe200e5cd43440e83047942538c8c1b P f91b0712e19140bd9b9453efa221e865 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f91b0712e19140bd9b9453efa221e865, State: Initialized, Role: FOLLOWER
I20250626 01:59:16.191838 20533 consensus_queue.cc:260] T bbe200e5cd43440e83047942538c8c1b P f91b0712e19140bd9b9453efa221e865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } } peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } attrs { promote: false } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } }
I20250626 01:59:16.193192 20533 ts_tablet_manager.cc:1428] T bbe200e5cd43440e83047942538c8c1b P f91b0712e19140bd9b9453efa221e865: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20250626 01:59:16.344547 20645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:59:16.345101 20645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 01:59:16.345616 20645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:59:16.377140 20645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 01:59:16.378096 20645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.19.194
I20250626 01:59:16.413576 20645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:42273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.19.194:39201
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.16.19.194
--webserver_port=38337
--tserver_master_addrs=127.16.19.254:43009
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.19.194
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 01:59:16.414990 20645 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 01:59:16.416654 20645 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:59:16.434197 20658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:59:17.429232 20663 raft_consensus.cc:491] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:59:17.429718 20663 raft_consensus.cc:513] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } } peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } attrs { promote: false } }
I20250626 01:59:17.431916 20663 leader_election.cc:290] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cf4e54b733f84093b1488c82347f0389 (127.16.19.194:39201), 7ab09cbcdaba4e069a36503512251ef7 (127.16.19.195:40417)
W20250626 01:59:17.435806 20530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.19.194:39201: connect: Connection refused (error 111)
W20250626 01:59:17.439829 20530 leader_election.cc:336] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer cf4e54b733f84093b1488c82347f0389 (127.16.19.194:39201): Network error: Client connection negotiation failed: client connection to 127.16.19.194:39201: connect: Connection refused (error 111)
W20250626 01:59:17.440527 20528 leader_election.cc:336] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7ab09cbcdaba4e069a36503512251ef7 (127.16.19.195:40417): Network error: Client connection negotiation failed: client connection to 127.16.19.195:40417: connect: Connection refused (error 111)
I20250626 01:59:17.440994 20528 leader_election.cc:304] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f91b0712e19140bd9b9453efa221e865; no voters: 7ab09cbcdaba4e069a36503512251ef7, cf4e54b733f84093b1488c82347f0389
I20250626 01:59:17.441778 20663 raft_consensus.cc:2747] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250626 01:59:16.434230 20660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:59:16.435910 20645 server_base.cc:1048] running on GCE node
W20250626 01:59:16.434188 20657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:59:17.612113 20645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 01:59:17.614913 20645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 01:59:17.616355 20645 hybrid_clock.cc:648] HybridClock initialized: now 1750903157616306 us; error 47 us; skew 500 ppm
I20250626 01:59:17.617391 20645 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:59:17.624864 20645 webserver.cc:469] Webserver started at http://127.16.19.194:38337/ using document root <none> and password file <none>
I20250626 01:59:17.626048 20645 fs_manager.cc:362] Metadata directory not provided
I20250626 01:59:17.626313 20645 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:59:17.635200 20645 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20250626 01:59:17.639801 20671 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:59:17.640899 20645 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250626 01:59:17.641201 20645 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "cf4e54b733f84093b1488c82347f0389"
format_stamp: "Formatted at 2025-06-26 01:58:48 on dist-test-slave-h4rq"
I20250626 01:59:17.643155 20645 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:59:17.691690 20645 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:59:17.693236 20645 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 01:59:17.693681 20645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:59:17.696180 20645 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:59:17.702041 20678 ts_tablet_manager.cc:542] Loading tablet metadata (0/3 complete)
I20250626 01:59:17.717844 20645 ts_tablet_manager.cc:579] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20250626 01:59:17.718142 20645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.018s user 0.000s sys 0.002s
I20250626 01:59:17.718461 20645 ts_tablet_manager.cc:594] Registering tablets (0/3 complete)
I20250626 01:59:17.724131 20678 tablet_bootstrap.cc:492] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389: Bootstrap starting.
I20250626 01:59:17.729995 20645 ts_tablet_manager.cc:610] Registered 3 tablets
I20250626 01:59:17.730238 20645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.012s user 0.010s sys 0.001s
I20250626 01:59:17.778301 20678 log.cc:826] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389: Log is configured to *not* fsync() on all Append() calls
I20250626 01:59:17.875016 20678 tablet_bootstrap.cc:492] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:59:17.875885 20678 tablet_bootstrap.cc:492] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389: Bootstrap complete.
I20250626 01:59:17.877349 20678 ts_tablet_manager.cc:1397] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389: Time spent bootstrapping tablet: real 0.154s user 0.093s sys 0.058s
I20250626 01:59:17.894480 20678 raft_consensus.cc:357] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } } peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } attrs { promote: false } }
I20250626 01:59:17.896539 20678 raft_consensus.cc:738] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf4e54b733f84093b1488c82347f0389, State: Initialized, Role: FOLLOWER
I20250626 01:59:17.897435 20678 consensus_queue.cc:260] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } } peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } attrs { promote: false } }
I20250626 01:59:17.902242 20678 ts_tablet_manager.cc:1428] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389: Time spent starting tablet: real 0.025s user 0.018s sys 0.005s
I20250626 01:59:17.903088 20678 tablet_bootstrap.cc:492] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389: Bootstrap starting.
I20250626 01:59:17.932907 20645 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.194:39201
I20250626 01:59:17.933079 20786 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.194:39201 every 8 connection(s)
I20250626 01:59:17.935496 20645 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250626 01:59:17.944485 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 20645
I20250626 01:59:17.946605 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.19.195:40417
--local_ip_for_outbound_sockets=127.16.19.195
--tserver_master_addrs=127.16.19.254:43009
--webserver_port=39177
--webserver_interface=127.16.19.195
--builtin_ntp_servers=127.16.19.212:42273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250626 01:59:17.955706 20787 heartbeater.cc:344] Connected to a master server at 127.16.19.254:43009
I20250626 01:59:17.956135 20787 heartbeater.cc:461] Registering TS with master...
I20250626 01:59:17.957235 20787 heartbeater.cc:507] Master 127.16.19.254:43009 requested a full tablet report, sending...
I20250626 01:59:17.960983 20454 ts_manager.cc:194] Registered new tserver with Master: cf4e54b733f84093b1488c82347f0389 (127.16.19.194:39201)
I20250626 01:59:17.964565 20454 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.19.194:33551
I20250626 01:59:18.032333 20678 tablet_bootstrap.cc:492] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:59:18.033339 20678 tablet_bootstrap.cc:492] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389: Bootstrap complete.
I20250626 01:59:18.034842 20678 ts_tablet_manager.cc:1397] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389: Time spent bootstrapping tablet: real 0.132s user 0.114s sys 0.015s
I20250626 01:59:18.037163 20678 raft_consensus.cc:357] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } }
I20250626 01:59:18.037691 20678 raft_consensus.cc:738] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf4e54b733f84093b1488c82347f0389, State: Initialized, Role: FOLLOWER
I20250626 01:59:18.038293 20678 consensus_queue.cc:260] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } }
I20250626 01:59:18.038828 20678 raft_consensus.cc:397] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 01:59:18.039165 20678 raft_consensus.cc:491] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 01:59:18.039543 20678 raft_consensus.cc:3058] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389 [term 2 FOLLOWER]: Advancing to term 3
I20250626 01:59:18.048084 20678 raft_consensus.cc:513] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } }
I20250626 01:59:18.048960 20678 leader_election.cc:304] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf4e54b733f84093b1488c82347f0389; no voters:
I20250626 01:59:18.049600 20678 leader_election.cc:290] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250626 01:59:18.049858 20768 raft_consensus.cc:2802] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 01:59:18.056254 20787 heartbeater.cc:499] Master 127.16.19.254:43009 was elected leader, sending a full tablet report...
I20250626 01:59:18.059058 20678 ts_tablet_manager.cc:1428] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389: Time spent starting tablet: real 0.024s user 0.018s sys 0.005s
I20250626 01:59:18.059890 20678 tablet_bootstrap.cc:492] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389: Bootstrap starting.
I20250626 01:59:18.060606 20768 raft_consensus.cc:695] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389 [term 3 LEADER]: Becoming Leader. State: Replica: cf4e54b733f84093b1488c82347f0389, State: Running, Role: LEADER
I20250626 01:59:18.065433 20796 raft_consensus.cc:491] T bbe200e5cd43440e83047942538c8c1b P f91b0712e19140bd9b9453efa221e865 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:59:18.065927 20796 raft_consensus.cc:513] T bbe200e5cd43440e83047942538c8c1b P f91b0712e19140bd9b9453efa221e865 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } } peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } attrs { promote: false } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } }
I20250626 01:59:18.065683 20768 consensus_queue.cc:237] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } }
I20250626 01:59:18.078119 20796 leader_election.cc:290] T bbe200e5cd43440e83047942538c8c1b P f91b0712e19140bd9b9453efa221e865 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7ab09cbcdaba4e069a36503512251ef7 (127.16.19.195:40417), cf4e54b733f84093b1488c82347f0389 (127.16.19.194:39201)
I20250626 01:59:18.098448 20454 catalog_manager.cc:5582] T d5b1c5efbd044314965eb5c7b1bede3c P cf4e54b733f84093b1488c82347f0389 reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "cf4e54b733f84093b1488c82347f0389" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } health_report { overall_health: HEALTHY } } }
W20250626 01:59:18.112672 20528 leader_election.cc:336] T bbe200e5cd43440e83047942538c8c1b P f91b0712e19140bd9b9453efa221e865 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7ab09cbcdaba4e069a36503512251ef7 (127.16.19.195:40417): Network error: Client connection negotiation failed: client connection to 127.16.19.195:40417: connect: Connection refused (error 111)
I20250626 01:59:18.118112 20741 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbe200e5cd43440e83047942538c8c1b" candidate_uuid: "f91b0712e19140bd9b9453efa221e865" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "cf4e54b733f84093b1488c82347f0389" is_pre_election: true
W20250626 01:59:18.121529 20530 leader_election.cc:343] T bbe200e5cd43440e83047942538c8c1b P f91b0712e19140bd9b9453efa221e865 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer cf4e54b733f84093b1488c82347f0389 (127.16.19.194:39201): Illegal state: must be running to vote when last-logged opid is not known
I20250626 01:59:18.122193 20530 leader_election.cc:304] T bbe200e5cd43440e83047942538c8c1b P f91b0712e19140bd9b9453efa221e865 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f91b0712e19140bd9b9453efa221e865; no voters: 7ab09cbcdaba4e069a36503512251ef7, cf4e54b733f84093b1488c82347f0389
I20250626 01:59:18.123701 20796 raft_consensus.cc:2747] T bbe200e5cd43440e83047942538c8c1b P f91b0712e19140bd9b9453efa221e865 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 01:59:18.201160 20678 tablet_bootstrap.cc:492] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:59:18.201882 20678 tablet_bootstrap.cc:492] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389: Bootstrap complete.
I20250626 01:59:18.203017 20678 ts_tablet_manager.cc:1397] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389: Time spent bootstrapping tablet: real 0.143s user 0.100s sys 0.019s
I20250626 01:59:18.204524 20678 raft_consensus.cc:357] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } } peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } attrs { promote: false } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } }
I20250626 01:59:18.205034 20678 raft_consensus.cc:738] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf4e54b733f84093b1488c82347f0389, State: Initialized, Role: FOLLOWER
I20250626 01:59:18.205516 20678 consensus_queue.cc:260] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } } peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } attrs { promote: false } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } }
I20250626 01:59:18.207113 20678 ts_tablet_manager.cc:1428] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20250626 01:59:18.325488 20792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:59:18.326022 20792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 01:59:18.326535 20792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:59:18.358086 20792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 01:59:18.358960 20792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.19.195
I20250626 01:59:18.393872 20792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:42273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.19.195:40417
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.16.19.195
--webserver_port=39177
--tserver_master_addrs=127.16.19.254:43009
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.19.195
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 01:59:18.395251 20792 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 01:59:18.396940 20792 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:59:18.414012 20813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:59:19.293346 20819 raft_consensus.cc:491] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:59:19.293730 20819 raft_consensus.cc:513] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } } peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } attrs { promote: false } }
I20250626 01:59:19.295172 20819 leader_election.cc:290] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cf4e54b733f84093b1488c82347f0389 (127.16.19.194:39201), 7ab09cbcdaba4e069a36503512251ef7 (127.16.19.195:40417)
I20250626 01:59:19.296761 20741 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "242b3d8e6a774185a554cf2603c1e5a0" candidate_uuid: "f91b0712e19140bd9b9453efa221e865" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "cf4e54b733f84093b1488c82347f0389" is_pre_election: true
I20250626 01:59:19.297397 20741 raft_consensus.cc:2466] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f91b0712e19140bd9b9453efa221e865 in term 2.
I20250626 01:59:19.298385 20530 leader_election.cc:304] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf4e54b733f84093b1488c82347f0389, f91b0712e19140bd9b9453efa221e865; no voters:
I20250626 01:59:19.299142 20819 raft_consensus.cc:2802] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 01:59:19.299471 20819 raft_consensus.cc:491] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:59:19.299777 20819 raft_consensus.cc:3058] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [term 2 FOLLOWER]: Advancing to term 3
W20250626 01:59:19.303620 20528 leader_election.cc:336] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7ab09cbcdaba4e069a36503512251ef7 (127.16.19.195:40417): Network error: Client connection negotiation failed: client connection to 127.16.19.195:40417: connect: Connection refused (error 111)
I20250626 01:59:19.307152 20819 raft_consensus.cc:513] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } } peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } attrs { promote: false } }
I20250626 01:59:19.308720 20819 leader_election.cc:290] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [CANDIDATE]: Term 3 election: Requested vote from peers cf4e54b733f84093b1488c82347f0389 (127.16.19.194:39201), 7ab09cbcdaba4e069a36503512251ef7 (127.16.19.195:40417)
I20250626 01:59:19.309883 20741 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "242b3d8e6a774185a554cf2603c1e5a0" candidate_uuid: "f91b0712e19140bd9b9453efa221e865" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "cf4e54b733f84093b1488c82347f0389"
I20250626 01:59:19.310703 20741 raft_consensus.cc:3058] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389 [term 2 FOLLOWER]: Advancing to term 3
W20250626 01:59:19.311875 20528 leader_election.cc:336] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 7ab09cbcdaba4e069a36503512251ef7 (127.16.19.195:40417): Network error: Client connection negotiation failed: client connection to 127.16.19.195:40417: connect: Connection refused (error 111)
I20250626 01:59:19.320947 20741 raft_consensus.cc:2466] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f91b0712e19140bd9b9453efa221e865 in term 3.
I20250626 01:59:19.322494 20530 leader_election.cc:304] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf4e54b733f84093b1488c82347f0389, f91b0712e19140bd9b9453efa221e865; no voters: 7ab09cbcdaba4e069a36503512251ef7
I20250626 01:59:19.323107 20819 raft_consensus.cc:2802] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 01:59:19.324232 20819 raft_consensus.cc:695] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [term 3 LEADER]: Becoming Leader. State: Replica: f91b0712e19140bd9b9453efa221e865, State: Running, Role: LEADER
I20250626 01:59:19.325166 20819 consensus_queue.cc:237] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } } peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } attrs { promote: false } }
I20250626 01:59:19.337610 20454 catalog_manager.cc:5582] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 reported cstate change: term changed from 2 to 3, leader changed from <none> to f91b0712e19140bd9b9453efa221e865 (127.16.19.193). New cstate: current_term: 3 leader_uuid: "f91b0712e19140bd9b9453efa221e865" committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250626 01:59:18.414134 20816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:59:18.414574 20814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:59:18.416354 20792 server_base.cc:1048] running on GCE node
I20250626 01:59:19.603168 20792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 01:59:19.605984 20792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 01:59:19.607508 20792 hybrid_clock.cc:648] HybridClock initialized: now 1750903159607453 us; error 54 us; skew 500 ppm
I20250626 01:59:19.608551 20792 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:59:19.616336 20792 webserver.cc:469] Webserver started at http://127.16.19.195:39177/ using document root <none> and password file <none>
I20250626 01:59:19.617723 20792 fs_manager.cc:362] Metadata directory not provided
I20250626 01:59:19.618016 20792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:59:19.628345 20792 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.006s sys 0.001s
I20250626 01:59:19.634225 20832 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:59:19.635385 20792 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250626 01:59:19.635764 20792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "7ab09cbcdaba4e069a36503512251ef7"
format_stamp: "Formatted at 2025-06-26 01:58:49 on dist-test-slave-h4rq"
I20250626 01:59:19.638463 20792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:59:19.697031 20792 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:59:19.698545 20792 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 01:59:19.698952 20792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:59:19.701467 20792 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:59:19.707334 20839 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250626 01:59:19.718928 20792 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250626 01:59:19.719206 20792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.002s sys 0.000s
I20250626 01:59:19.719471 20792 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250626 01:59:19.724604 20839 tablet_bootstrap.cc:492] T 242b3d8e6a774185a554cf2603c1e5a0 P 7ab09cbcdaba4e069a36503512251ef7: Bootstrap starting.
I20250626 01:59:19.727269 20792 ts_tablet_manager.cc:610] Registered 2 tablets
I20250626 01:59:19.727478 20792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.008s user 0.002s sys 0.003s
W20250626 01:59:19.733284 20528 consensus_peers.cc:487] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 -> Peer 7ab09cbcdaba4e069a36503512251ef7 (127.16.19.195:40417): Couldn't send request to peer 7ab09cbcdaba4e069a36503512251ef7. Status: Network error: Client connection negotiation failed: client connection to 127.16.19.195:40417: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 01:59:19.738452 20741 raft_consensus.cc:1273] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389 [term 3 FOLLOWER]: Refusing update from remote peer f91b0712e19140bd9b9453efa221e865: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20250626 01:59:19.740208 20819 consensus_queue.cc:1035] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20250626 01:59:19.787396 20596 consensus_queue.cc:237] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 13, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } }
I20250626 01:59:19.793040 20741 raft_consensus.cc:1273] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389 [term 3 FOLLOWER]: Refusing update from remote peer f91b0712e19140bd9b9453efa221e865: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20250626 01:59:19.794903 20822 consensus_queue.cc:1035] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20250626 01:59:19.800664 20839 log.cc:826] T 242b3d8e6a774185a554cf2603c1e5a0 P 7ab09cbcdaba4e069a36503512251ef7: Log is configured to *not* fsync() on all Append() calls
I20250626 01:59:19.802981 20819 raft_consensus.cc:2953] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [term 3 LEADER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER 7ab09cbcdaba4e069a36503512251ef7 (127.16.19.195) evicted. New config: { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } } }
I20250626 01:59:19.805265 20740 raft_consensus.cc:2953] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389 [term 3 FOLLOWER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER 7ab09cbcdaba4e069a36503512251ef7 (127.16.19.195) evicted. New config: { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } } }
I20250626 01:59:19.820451 20438 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 242b3d8e6a774185a554cf2603c1e5a0 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 01:59:19.825254 20454 catalog_manager.cc:5582] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389 reported cstate change: config changed from index 13 to 15, VOTER 7ab09cbcdaba4e069a36503512251ef7 (127.16.19.195) evicted. New cstate: current_term: 3 leader_uuid: "f91b0712e19140bd9b9453efa221e865" committed_config { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } } }
W20250626 01:59:19.836890 20454 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet 242b3d8e6a774185a554cf2603c1e5a0 on TS 7ab09cbcdaba4e069a36503512251ef7: Not found: failed to reset TS proxy: Could not find TS for UUID 7ab09cbcdaba4e069a36503512251ef7
I20250626 01:59:19.842664 20596 consensus_queue.cc:237] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 3.15, Last appended by leader: 13, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } }
I20250626 01:59:19.846571 20819 raft_consensus.cc:2953] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 [term 3 LEADER]: Committing config change with OpId 3.16: config changed from index 15 to 16, VOTER cf4e54b733f84093b1488c82347f0389 (127.16.19.194) evicted. New config: { opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } } }
I20250626 01:59:19.855085 20438 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 242b3d8e6a774185a554cf2603c1e5a0 with cas_config_opid_index 15: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 01:59:19.859977 20453 catalog_manager.cc:5582] T 242b3d8e6a774185a554cf2603c1e5a0 P f91b0712e19140bd9b9453efa221e865 reported cstate change: config changed from index 15 to 16, VOTER cf4e54b733f84093b1488c82347f0389 (127.16.19.194) evicted. New cstate: current_term: 3 leader_uuid: "f91b0712e19140bd9b9453efa221e865" committed_config { opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } health_report { overall_health: HEALTHY } } }
I20250626 01:59:19.886314 20721 tablet_service.cc:1515] Processing DeleteTablet for tablet 242b3d8e6a774185a554cf2603c1e5a0 with delete_type TABLET_DATA_TOMBSTONED (TS cf4e54b733f84093b1488c82347f0389 not found in new config with opid_index 16) from {username='slave'} at 127.0.0.1:50100
I20250626 01:59:19.889572 20909 tablet_replica.cc:331] stopping tablet replica
W20250626 01:59:19.890120 20439 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 242b3d8e6a774185a554cf2603c1e5a0 on TS 7ab09cbcdaba4e069a36503512251ef7 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 7ab09cbcdaba4e069a36503512251ef7
I20250626 01:59:19.890552 20909 raft_consensus.cc:2241] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250626 01:59:19.891248 20909 raft_consensus.cc:2270] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 01:59:19.895740 20909 ts_tablet_manager.cc:1905] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 01:59:19.911389 20843 raft_consensus.cc:491] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:59:19.911944 20843 raft_consensus.cc:513] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } } peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } attrs { promote: false } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } }
I20250626 01:59:19.918092 20909 ts_tablet_manager.cc:1918] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.15
I20250626 01:59:19.918604 20909 log.cc:1199] T 242b3d8e6a774185a554cf2603c1e5a0 P cf4e54b733f84093b1488c82347f0389: Deleting WAL directory at /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal/wals/242b3d8e6a774185a554cf2603c1e5a0
I20250626 01:59:19.920464 20843 leader_election.cc:290] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7ab09cbcdaba4e069a36503512251ef7 (127.16.19.195:40417), f91b0712e19140bd9b9453efa221e865 (127.16.19.193:44115)
I20250626 01:59:19.922186 20441 catalog_manager.cc:4928] TS cf4e54b733f84093b1488c82347f0389 (127.16.19.194:39201): tablet 242b3d8e6a774185a554cf2603c1e5a0 (table TestTable [id=653cf9b21d7c405295d3deef89704bc2]) successfully deleted
W20250626 01:59:19.925016 20673 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.19.195:40417: connect: Connection refused (error 111)
I20250626 01:59:19.936599 20596 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbe200e5cd43440e83047942538c8c1b" candidate_uuid: "cf4e54b733f84093b1488c82347f0389" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "f91b0712e19140bd9b9453efa221e865" is_pre_election: true
I20250626 01:59:19.937678 20596 raft_consensus.cc:2466] T bbe200e5cd43440e83047942538c8c1b P f91b0712e19140bd9b9453efa221e865 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf4e54b733f84093b1488c82347f0389 in term 2.
I20250626 01:59:19.939206 20672 leader_election.cc:304] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf4e54b733f84093b1488c82347f0389, f91b0712e19140bd9b9453efa221e865; no voters:
I20250626 01:59:19.939894 20843 raft_consensus.cc:2802] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 01:59:19.940300 20843 raft_consensus.cc:491] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:59:19.940732 20843 raft_consensus.cc:3058] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389 [term 2 FOLLOWER]: Advancing to term 3
I20250626 01:59:19.950831 20843 raft_consensus.cc:513] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } } peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } attrs { promote: false } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } }
I20250626 01:59:19.953807 20843 leader_election.cc:290] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389 [CANDIDATE]: Term 3 election: Requested vote from peers 7ab09cbcdaba4e069a36503512251ef7 (127.16.19.195:40417), f91b0712e19140bd9b9453efa221e865 (127.16.19.193:44115)
I20250626 01:59:19.955008 20596 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbe200e5cd43440e83047942538c8c1b" candidate_uuid: "cf4e54b733f84093b1488c82347f0389" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "f91b0712e19140bd9b9453efa221e865"
I20250626 01:59:19.955770 20596 raft_consensus.cc:3058] T bbe200e5cd43440e83047942538c8c1b P f91b0712e19140bd9b9453efa221e865 [term 2 FOLLOWER]: Advancing to term 3
W20250626 01:59:19.959663 20673 leader_election.cc:336] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7ab09cbcdaba4e069a36503512251ef7 (127.16.19.195:40417): Network error: Client connection negotiation failed: client connection to 127.16.19.195:40417: connect: Connection refused (error 111)
I20250626 01:59:19.962330 20596 raft_consensus.cc:2466] T bbe200e5cd43440e83047942538c8c1b P f91b0712e19140bd9b9453efa221e865 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf4e54b733f84093b1488c82347f0389 in term 3.
I20250626 01:59:19.963210 20672 leader_election.cc:304] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf4e54b733f84093b1488c82347f0389, f91b0712e19140bd9b9453efa221e865; no voters:
I20250626 01:59:19.963807 20843 raft_consensus.cc:2802] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 01:59:19.965763 20843 raft_consensus.cc:695] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389 [term 3 LEADER]: Becoming Leader. State: Replica: cf4e54b733f84093b1488c82347f0389, State: Running, Role: LEADER
W20250626 01:59:19.967056 20673 leader_election.cc:336] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 7ab09cbcdaba4e069a36503512251ef7 (127.16.19.195:40417): Network error: Client connection negotiation failed: client connection to 127.16.19.195:40417: connect: Connection refused (error 111)
I20250626 01:59:19.967573 20843 consensus_queue.cc:237] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } } peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } attrs { promote: false } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } }
I20250626 01:59:19.980224 20453 catalog_manager.cc:5582] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389 reported cstate change: term changed from 2 to 3, leader changed from 7ab09cbcdaba4e069a36503512251ef7 (127.16.19.195) to cf4e54b733f84093b1488c82347f0389 (127.16.19.194). New cstate: current_term: 3 leader_uuid: "cf4e54b733f84093b1488c82347f0389" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 01:59:20.031678 20839 tablet_bootstrap.cc:492] T 242b3d8e6a774185a554cf2603c1e5a0 P 7ab09cbcdaba4e069a36503512251ef7: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:59:20.032943 20839 tablet_bootstrap.cc:492] T 242b3d8e6a774185a554cf2603c1e5a0 P 7ab09cbcdaba4e069a36503512251ef7: Bootstrap complete.
I20250626 01:59:20.035163 20839 ts_tablet_manager.cc:1397] T 242b3d8e6a774185a554cf2603c1e5a0 P 7ab09cbcdaba4e069a36503512251ef7: Time spent bootstrapping tablet: real 0.311s user 0.210s sys 0.041s
I20250626 01:59:20.047096 20792 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.195:40417
I20250626 01:59:20.047281 20960 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.195:40417 every 8 connection(s)
I20250626 01:59:20.050668 20792 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250626 01:59:20.055349 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 20792
I20250626 01:59:20.065807 20839 raft_consensus.cc:357] T 242b3d8e6a774185a554cf2603c1e5a0 P 7ab09cbcdaba4e069a36503512251ef7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } } peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } attrs { promote: false } }
I20250626 01:59:20.075683 20839 raft_consensus.cc:738] T 242b3d8e6a774185a554cf2603c1e5a0 P 7ab09cbcdaba4e069a36503512251ef7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ab09cbcdaba4e069a36503512251ef7, State: Initialized, Role: FOLLOWER
I20250626 01:59:20.079146 20839 consensus_queue.cc:260] T 242b3d8e6a774185a554cf2603c1e5a0 P 7ab09cbcdaba4e069a36503512251ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } } peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } attrs { promote: false } }
I20250626 01:59:20.091133 20839 ts_tablet_manager.cc:1428] T 242b3d8e6a774185a554cf2603c1e5a0 P 7ab09cbcdaba4e069a36503512251ef7: Time spent starting tablet: real 0.056s user 0.034s sys 0.001s
I20250626 01:59:20.091818 20839 tablet_bootstrap.cc:492] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7: Bootstrap starting.
I20250626 01:59:20.101351 20961 heartbeater.cc:344] Connected to a master server at 127.16.19.254:43009
I20250626 01:59:20.101883 20961 heartbeater.cc:461] Registering TS with master...
I20250626 01:59:20.103150 20961 heartbeater.cc:507] Master 127.16.19.254:43009 requested a full tablet report, sending...
I20250626 01:59:20.107515 20453 ts_manager.cc:194] Registered new tserver with Master: 7ab09cbcdaba4e069a36503512251ef7 (127.16.19.195:40417)
I20250626 01:59:20.112656 20453 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.19.195:47619
I20250626 01:59:20.116843 20961 heartbeater.cc:499] Master 127.16.19.254:43009 was elected leader, sending a full tablet report...
I20250626 01:59:20.118103 16463 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 01:59:20.124542 16463 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
W20250626 01:59:20.129069 16463 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 242b3d8e6a774185a554cf2603c1e5a0: tablet_id: "242b3d8e6a774185a554cf2603c1e5a0" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
I20250626 01:59:20.194836 20839 tablet_bootstrap.cc:492] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:59:20.195549 20839 tablet_bootstrap.cc:492] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7: Bootstrap complete.
I20250626 01:59:20.196731 20839 ts_tablet_manager.cc:1397] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7: Time spent bootstrapping tablet: real 0.105s user 0.097s sys 0.008s
I20250626 01:59:20.198289 20839 raft_consensus.cc:357] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } } peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } attrs { promote: false } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } }
I20250626 01:59:20.198740 20839 raft_consensus.cc:738] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ab09cbcdaba4e069a36503512251ef7, State: Initialized, Role: FOLLOWER
I20250626 01:59:20.199183 20839 consensus_queue.cc:260] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 } } peers { permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } attrs { promote: false } } peers { permanent_uuid: "cf4e54b733f84093b1488c82347f0389" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 39201 } attrs { promote: false } }
I20250626 01:59:20.200511 20839 ts_tablet_manager.cc:1428] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7: Time spent starting tablet: real 0.004s user 0.000s sys 0.000s
I20250626 01:59:20.223564 20889 tablet_service.cc:1515] Processing DeleteTablet for tablet 242b3d8e6a774185a554cf2603c1e5a0 with delete_type TABLET_DATA_TOMBSTONED (TS 7ab09cbcdaba4e069a36503512251ef7 not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:36330
I20250626 01:59:20.228552 20970 tablet_replica.cc:331] stopping tablet replica
I20250626 01:59:20.229194 20970 raft_consensus.cc:2241] T 242b3d8e6a774185a554cf2603c1e5a0 P 7ab09cbcdaba4e069a36503512251ef7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 01:59:20.229588 20970 raft_consensus.cc:2270] T 242b3d8e6a774185a554cf2603c1e5a0 P 7ab09cbcdaba4e069a36503512251ef7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:59:20.231962 20970 ts_tablet_manager.cc:1905] T 242b3d8e6a774185a554cf2603c1e5a0 P 7ab09cbcdaba4e069a36503512251ef7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 01:59:20.243815 20970 ts_tablet_manager.cc:1918] T 242b3d8e6a774185a554cf2603c1e5a0 P 7ab09cbcdaba4e069a36503512251ef7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.13
I20250626 01:59:20.244153 20970 log.cc:1199] T 242b3d8e6a774185a554cf2603c1e5a0 P 7ab09cbcdaba4e069a36503512251ef7: Deleting WAL directory at /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal/wals/242b3d8e6a774185a554cf2603c1e5a0
I20250626 01:59:20.245839 20439 catalog_manager.cc:4928] TS 7ab09cbcdaba4e069a36503512251ef7 (127.16.19.195:40417): tablet 242b3d8e6a774185a554cf2603c1e5a0 (table TestTable [id=653cf9b21d7c405295d3deef89704bc2]) successfully deleted
I20250626 01:59:20.372817 20596 raft_consensus.cc:1273] T bbe200e5cd43440e83047942538c8c1b P f91b0712e19140bd9b9453efa221e865 [term 3 FOLLOWER]: Refusing update from remote peer cf4e54b733f84093b1488c82347f0389: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20250626 01:59:20.374478 20843 consensus_queue.cc:1035] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f91b0712e19140bd9b9453efa221e865" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 44115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250626 01:59:20.395154 20911 raft_consensus.cc:3058] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [term 2 FOLLOWER]: Advancing to term 3
I20250626 01:59:20.401057 20911 raft_consensus.cc:1273] T bbe200e5cd43440e83047942538c8c1b P 7ab09cbcdaba4e069a36503512251ef7 [term 3 FOLLOWER]: Refusing update from remote peer cf4e54b733f84093b1488c82347f0389: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20250626 01:59:20.402513 20843 consensus_queue.cc:1035] T bbe200e5cd43440e83047942538c8c1b P cf4e54b733f84093b1488c82347f0389 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ab09cbcdaba4e069a36503512251ef7" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 40417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20250626 01:59:21.133275 16463 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 242b3d8e6a774185a554cf2603c1e5a0: tablet_id: "242b3d8e6a774185a554cf2603c1e5a0" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 01:59:22.136822 16463 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 242b3d8e6a774185a554cf2603c1e5a0: tablet_id: "242b3d8e6a774185a554cf2603c1e5a0" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 01:59:23.140652 16463 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 242b3d8e6a774185a554cf2603c1e5a0: tablet_id: "242b3d8e6a774185a554cf2603c1e5a0" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 01:59:24.144107 16463 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 242b3d8e6a774185a554cf2603c1e5a0: tablet_id: "242b3d8e6a774185a554cf2603c1e5a0" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 01:59:25.147804 16463 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 242b3d8e6a774185a554cf2603c1e5a0: tablet_id: "242b3d8e6a774185a554cf2603c1e5a0" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 01:59:26.151384 16463 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 242b3d8e6a774185a554cf2603c1e5a0: tablet_id: "242b3d8e6a774185a554cf2603c1e5a0" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 01:59:27.155025 16463 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 242b3d8e6a774185a554cf2603c1e5a0: tablet_id: "242b3d8e6a774185a554cf2603c1e5a0" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 01:59:28.159144 16463 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 242b3d8e6a774185a554cf2603c1e5a0: tablet_id: "242b3d8e6a774185a554cf2603c1e5a0" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 01:59:29.163074 16463 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 242b3d8e6a774185a554cf2603c1e5a0: tablet_id: "242b3d8e6a774185a554cf2603c1e5a0" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 01:59:30.166743 16463 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 242b3d8e6a774185a554cf2603c1e5a0: tablet_id: "242b3d8e6a774185a554cf2603c1e5a0" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 01:59:31.170431 16463 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 242b3d8e6a774185a554cf2603c1e5a0: tablet_id: "242b3d8e6a774185a554cf2603c1e5a0" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 01:59:32.173938 16463 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 242b3d8e6a774185a554cf2603c1e5a0: tablet_id: "242b3d8e6a774185a554cf2603c1e5a0" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 01:59:33.177951 16463 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 242b3d8e6a774185a554cf2603c1e5a0: tablet_id: "242b3d8e6a774185a554cf2603c1e5a0" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 01:59:34.181885 16463 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 242b3d8e6a774185a554cf2603c1e5a0: tablet_id: "242b3d8e6a774185a554cf2603c1e5a0" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 01:59:35.185861 16463 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 242b3d8e6a774185a554cf2603c1e5a0: tablet_id: "242b3d8e6a774185a554cf2603c1e5a0" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 01:59:36.190905 16463 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 242b3d8e6a774185a554cf2603c1e5a0: tablet_id: "242b3d8e6a774185a554cf2603c1e5a0" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 01:59:37.166889 20783 debug-util.cc:398] Leaking SignalData structure 0x7b0800022380 after lost signal to thread 20652
W20250626 01:59:37.168058 20783 debug-util.cc:398] Leaking SignalData structure 0x7b08000aeaa0 after lost signal to thread 20786
W20250626 01:59:37.195392 16463 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 242b3d8e6a774185a554cf2603c1e5a0: tablet_id: "242b3d8e6a774185a554cf2603c1e5a0" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 01:59:38.200675 16463 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 242b3d8e6a774185a554cf2603c1e5a0: tablet_id: "242b3d8e6a774185a554cf2603c1e5a0" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250626 01:59:39.204535 16463 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 242b3d8e6a774185a554cf2603c1e5a0: tablet_id: "242b3d8e6a774185a554cf2603c1e5a0" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/kudu-admin-test.cc:3914: Failure
Failed
Bad status: Not found: not all replicas of tablets comprising table TestTable are registered yet
I20250626 01:59:40.207599 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 20491
I20250626 01:59:40.234792 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 20645
I20250626 01:59:40.265210 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 20792
I20250626 01:59:40.293733 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 20421
2025-06-26T01:59:40Z chronyd exiting
I20250626 01:59:40.345443 16463 test_util.cc:183] -----------------------------------------------
I20250626 01:59:40.345662 16463 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750903057700330-16463-0
[ FAILED ] AdminCliTest.TestRebuildTables (57874 ms)
[----------] 5 tests from AdminCliTest (122572 ms total)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest
[ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4
I20250626 01:59:40.349635 16463 test_util.cc:276] Using random seed: -367406935
I20250626 01:59:40.355968 16463 ts_itest-base.cc:115] Starting cluster with:
I20250626 01:59:40.356151 16463 ts_itest-base.cc:116] --------------
I20250626 01:59:40.356261 16463 ts_itest-base.cc:117] 5 tablet servers
I20250626 01:59:40.356361 16463 ts_itest-base.cc:118] 3 replicas per TS
I20250626 01:59:40.356469 16463 ts_itest-base.cc:119] --------------
2025-06-26T01:59:40Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-26T01:59:40Z Disabled control of system clock
I20250626 01:59:40.401938 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.19.254:42069
--webserver_interface=127.16.19.254
--webserver_port=0
--builtin_ntp_servers=127.16.19.212:39835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.19.254:42069
--raft_prepare_replacement_before_eviction=true with env {}
W20250626 01:59:40.712898 21005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:59:40.713506 21005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 01:59:40.713987 21005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:59:40.745705 21005 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250626 01:59:40.746130 21005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 01:59:40.746387 21005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 01:59:40.746644 21005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 01:59:40.746882 21005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 01:59:40.786648 21005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:39835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.19.254:42069
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.19.254:42069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.19.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 01:59:40.788084 21005 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 01:59:40.790351 21005 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:59:40.806154 21011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:59:40.806978 21014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:59:40.808930 21005 server_base.cc:1048] running on GCE node
W20250626 01:59:40.807076 21012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:59:41.997414 21005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 01:59:42.001276 21005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 01:59:42.002871 21005 hybrid_clock.cc:648] HybridClock initialized: now 1750903182002820 us; error 48 us; skew 500 ppm
I20250626 01:59:42.004202 21005 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:59:42.018210 21005 webserver.cc:469] Webserver started at http://127.16.19.254:34299/ using document root <none> and password file <none>
I20250626 01:59:42.019564 21005 fs_manager.cc:362] Metadata directory not provided
I20250626 01:59:42.019775 21005 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:59:42.020205 21005 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:59:42.024757 21005 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "f6a32dbd0c714410bf49f779cf82ef04"
format_stamp: "Formatted at 2025-06-26 01:59:42 on dist-test-slave-h4rq"
I20250626 01:59:42.025897 21005 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "f6a32dbd0c714410bf49f779cf82ef04"
format_stamp: "Formatted at 2025-06-26 01:59:42 on dist-test-slave-h4rq"
I20250626 01:59:42.033274 21005 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250626 01:59:42.038909 21021 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:59:42.040055 21005 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.002s
I20250626 01:59:42.040380 21005 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f6a32dbd0c714410bf49f779cf82ef04"
format_stamp: "Formatted at 2025-06-26 01:59:42 on dist-test-slave-h4rq"
I20250626 01:59:42.040788 21005 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:59:42.108244 21005 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:59:42.109866 21005 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 01:59:42.110332 21005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:59:42.182389 21005 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.254:42069
I20250626 01:59:42.182482 21072 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.254:42069 every 8 connection(s)
I20250626 01:59:42.185142 21005 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250626 01:59:42.188889 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 21005
I20250626 01:59:42.189438 16463 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250626 01:59:42.191514 21073 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:59:42.219748 21073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6a32dbd0c714410bf49f779cf82ef04: Bootstrap starting.
I20250626 01:59:42.225452 21073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6a32dbd0c714410bf49f779cf82ef04: Neither blocks nor log segments found. Creating new log.
I20250626 01:59:42.227523 21073 log.cc:826] T 00000000000000000000000000000000 P f6a32dbd0c714410bf49f779cf82ef04: Log is configured to *not* fsync() on all Append() calls
I20250626 01:59:42.232061 21073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6a32dbd0c714410bf49f779cf82ef04: No bootstrap required, opened a new log
I20250626 01:59:42.250041 21073 raft_consensus.cc:357] T 00000000000000000000000000000000 P f6a32dbd0c714410bf49f779cf82ef04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6a32dbd0c714410bf49f779cf82ef04" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 42069 } }
I20250626 01:59:42.250747 21073 raft_consensus.cc:383] T 00000000000000000000000000000000 P f6a32dbd0c714410bf49f779cf82ef04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:59:42.251008 21073 raft_consensus.cc:738] T 00000000000000000000000000000000 P f6a32dbd0c714410bf49f779cf82ef04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6a32dbd0c714410bf49f779cf82ef04, State: Initialized, Role: FOLLOWER
I20250626 01:59:42.251711 21073 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6a32dbd0c714410bf49f779cf82ef04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6a32dbd0c714410bf49f779cf82ef04" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 42069 } }
I20250626 01:59:42.252197 21073 raft_consensus.cc:397] T 00000000000000000000000000000000 P f6a32dbd0c714410bf49f779cf82ef04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 01:59:42.252473 21073 raft_consensus.cc:491] T 00000000000000000000000000000000 P f6a32dbd0c714410bf49f779cf82ef04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 01:59:42.252787 21073 raft_consensus.cc:3058] T 00000000000000000000000000000000 P f6a32dbd0c714410bf49f779cf82ef04 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:59:42.257421 21073 raft_consensus.cc:513] T 00000000000000000000000000000000 P f6a32dbd0c714410bf49f779cf82ef04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6a32dbd0c714410bf49f779cf82ef04" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 42069 } }
I20250626 01:59:42.258185 21073 leader_election.cc:304] T 00000000000000000000000000000000 P f6a32dbd0c714410bf49f779cf82ef04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6a32dbd0c714410bf49f779cf82ef04; no voters:
I20250626 01:59:42.259819 21073 leader_election.cc:290] T 00000000000000000000000000000000 P f6a32dbd0c714410bf49f779cf82ef04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 01:59:42.260608 21078 raft_consensus.cc:2802] T 00000000000000000000000000000000 P f6a32dbd0c714410bf49f779cf82ef04 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:59:42.262981 21078 raft_consensus.cc:695] T 00000000000000000000000000000000 P f6a32dbd0c714410bf49f779cf82ef04 [term 1 LEADER]: Becoming Leader. State: Replica: f6a32dbd0c714410bf49f779cf82ef04, State: Running, Role: LEADER
I20250626 01:59:42.263972 21073 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6a32dbd0c714410bf49f779cf82ef04 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 01:59:42.263904 21078 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6a32dbd0c714410bf49f779cf82ef04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6a32dbd0c714410bf49f779cf82ef04" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 42069 } }
I20250626 01:59:42.275547 21079 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6a32dbd0c714410bf49f779cf82ef04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6a32dbd0c714410bf49f779cf82ef04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6a32dbd0c714410bf49f779cf82ef04" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 42069 } } }
I20250626 01:59:42.275533 21080 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6a32dbd0c714410bf49f779cf82ef04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6a32dbd0c714410bf49f779cf82ef04. Latest consensus state: current_term: 1 leader_uuid: "f6a32dbd0c714410bf49f779cf82ef04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6a32dbd0c714410bf49f779cf82ef04" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 42069 } } }
I20250626 01:59:42.276280 21079 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6a32dbd0c714410bf49f779cf82ef04 [sys.catalog]: This master's current role is: LEADER
I20250626 01:59:42.276427 21080 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6a32dbd0c714410bf49f779cf82ef04 [sys.catalog]: This master's current role is: LEADER
I20250626 01:59:42.279269 21088 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 01:59:42.292479 21088 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 01:59:42.309423 21088 catalog_manager.cc:1349] Generated new cluster ID: e0f45de269cc46ce8001faa792517fd9
I20250626 01:59:42.309707 21088 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 01:59:42.347975 21088 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 01:59:42.349485 21088 catalog_manager.cc:1506] Loading token signing keys...
I20250626 01:59:42.366585 21088 catalog_manager.cc:5955] T 00000000000000000000000000000000 P f6a32dbd0c714410bf49f779cf82ef04: Generated new TSK 0
I20250626 01:59:42.367452 21088 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 01:59:42.376423 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.19.193:0
--local_ip_for_outbound_sockets=127.16.19.193
--webserver_interface=127.16.19.193
--webserver_port=0
--tserver_master_addrs=127.16.19.254:42069
--builtin_ntp_servers=127.16.19.212:39835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250626 01:59:42.695672 21097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:59:42.696192 21097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 01:59:42.696768 21097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:59:42.729804 21097 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250626 01:59:42.730290 21097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 01:59:42.731202 21097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.19.193
I20250626 01:59:42.767448 21097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:39835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.19.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.19.193
--webserver_port=0
--tserver_master_addrs=127.16.19.254:42069
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.19.193
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 01:59:42.768901 21097 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 01:59:42.770627 21097 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:59:42.789816 21103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:59:42.789927 21104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:59:42.792416 21097 server_base.cc:1048] running on GCE node
W20250626 01:59:42.791309 21106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:59:43.972684 21097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 01:59:43.975507 21097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 01:59:43.977025 21097 hybrid_clock.cc:648] HybridClock initialized: now 1750903183976968 us; error 56 us; skew 500 ppm
I20250626 01:59:43.978101 21097 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:59:43.986846 21097 webserver.cc:469] Webserver started at http://127.16.19.193:32999/ using document root <none> and password file <none>
I20250626 01:59:43.987840 21097 fs_manager.cc:362] Metadata directory not provided
I20250626 01:59:43.988067 21097 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:59:43.988519 21097 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:59:43.994163 21097 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "05c9f80da92d47d88a526a913bba5387"
format_stamp: "Formatted at 2025-06-26 01:59:43 on dist-test-slave-h4rq"
I20250626 01:59:43.995846 21097 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "05c9f80da92d47d88a526a913bba5387"
format_stamp: "Formatted at 2025-06-26 01:59:43 on dist-test-slave-h4rq"
I20250626 01:59:44.006325 21097 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.007s sys 0.001s
I20250626 01:59:44.013046 21113 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:59:44.014467 21097 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250626 01:59:44.014995 21097 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "05c9f80da92d47d88a526a913bba5387"
format_stamp: "Formatted at 2025-06-26 01:59:43 on dist-test-slave-h4rq"
I20250626 01:59:44.015580 21097 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:59:44.093592 21097 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:59:44.095121 21097 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 01:59:44.095575 21097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:59:44.098129 21097 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:59:44.102301 21097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:59:44.102535 21097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:59:44.102806 21097 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:59:44.102967 21097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:59:44.246197 21097 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.193:40551
I20250626 01:59:44.246315 21225 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.193:40551 every 8 connection(s)
I20250626 01:59:44.248765 21097 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250626 01:59:44.253696 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 21097
I20250626 01:59:44.254247 16463 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250626 01:59:44.261849 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.19.194:0
--local_ip_for_outbound_sockets=127.16.19.194
--webserver_interface=127.16.19.194
--webserver_port=0
--tserver_master_addrs=127.16.19.254:42069
--builtin_ntp_servers=127.16.19.212:39835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250626 01:59:44.273442 21226 heartbeater.cc:344] Connected to a master server at 127.16.19.254:42069
I20250626 01:59:44.273906 21226 heartbeater.cc:461] Registering TS with master...
I20250626 01:59:44.274933 21226 heartbeater.cc:507] Master 127.16.19.254:42069 requested a full tablet report, sending...
I20250626 01:59:44.277711 21038 ts_manager.cc:194] Registered new tserver with Master: 05c9f80da92d47d88a526a913bba5387 (127.16.19.193:40551)
I20250626 01:59:44.280721 21038 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.19.193:51231
W20250626 01:59:44.572180 21230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:59:44.572712 21230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 01:59:44.573266 21230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:59:44.604591 21230 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250626 01:59:44.605033 21230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 01:59:44.605835 21230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.19.194
I20250626 01:59:44.641579 21230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:39835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.19.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.16.19.194
--webserver_port=0
--tserver_master_addrs=127.16.19.254:42069
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.19.194
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 01:59:44.642917 21230 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 01:59:44.644587 21230 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:59:44.662236 21236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:59:45.284785 21226 heartbeater.cc:499] Master 127.16.19.254:42069 was elected leader, sending a full tablet report...
W20250626 01:59:44.662386 21237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:59:44.663304 21239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:59:44.664911 21230 server_base.cc:1048] running on GCE node
I20250626 01:59:45.828882 21230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 01:59:45.831192 21230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 01:59:45.832542 21230 hybrid_clock.cc:648] HybridClock initialized: now 1750903185832505 us; error 65 us; skew 500 ppm
I20250626 01:59:45.833398 21230 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:59:45.839638 21230 webserver.cc:469] Webserver started at http://127.16.19.194:34749/ using document root <none> and password file <none>
I20250626 01:59:45.840561 21230 fs_manager.cc:362] Metadata directory not provided
I20250626 01:59:45.840818 21230 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:59:45.841267 21230 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:59:45.845804 21230 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "df2d94c744ba4a2085d8f37cda8b5472"
format_stamp: "Formatted at 2025-06-26 01:59:45 on dist-test-slave-h4rq"
I20250626 01:59:45.846905 21230 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "df2d94c744ba4a2085d8f37cda8b5472"
format_stamp: "Formatted at 2025-06-26 01:59:45 on dist-test-slave-h4rq"
I20250626 01:59:45.854609 21230 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.000s sys 0.008s
I20250626 01:59:45.860935 21246 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:59:45.862092 21230 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250626 01:59:45.862413 21230 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "df2d94c744ba4a2085d8f37cda8b5472"
format_stamp: "Formatted at 2025-06-26 01:59:45 on dist-test-slave-h4rq"
I20250626 01:59:45.862740 21230 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:59:45.919109 21230 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:59:45.920606 21230 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 01:59:45.921077 21230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:59:45.923698 21230 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:59:45.927806 21230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:59:45.928002 21230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:59:45.928256 21230 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:59:45.928404 21230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:59:46.064776 21230 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.194:35583
I20250626 01:59:46.064869 21358 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.194:35583 every 8 connection(s)
I20250626 01:59:46.067386 21230 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250626 01:59:46.069507 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 21230
I20250626 01:59:46.070048 16463 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250626 01:59:46.077368 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.19.195:0
--local_ip_for_outbound_sockets=127.16.19.195
--webserver_interface=127.16.19.195
--webserver_port=0
--tserver_master_addrs=127.16.19.254:42069
--builtin_ntp_servers=127.16.19.212:39835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250626 01:59:46.088963 21359 heartbeater.cc:344] Connected to a master server at 127.16.19.254:42069
I20250626 01:59:46.089386 21359 heartbeater.cc:461] Registering TS with master...
I20250626 01:59:46.090399 21359 heartbeater.cc:507] Master 127.16.19.254:42069 requested a full tablet report, sending...
I20250626 01:59:46.092572 21038 ts_manager.cc:194] Registered new tserver with Master: df2d94c744ba4a2085d8f37cda8b5472 (127.16.19.194:35583)
I20250626 01:59:46.093832 21038 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.19.194:44157
W20250626 01:59:46.381780 21363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:59:46.382292 21363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 01:59:46.382797 21363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:59:46.413817 21363 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250626 01:59:46.414239 21363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 01:59:46.415055 21363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.19.195
I20250626 01:59:46.449911 21363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:39835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.19.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.16.19.195
--webserver_port=0
--tserver_master_addrs=127.16.19.254:42069
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.19.195
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 01:59:46.451349 21363 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 01:59:46.453040 21363 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:59:46.470481 21369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:59:47.097013 21359 heartbeater.cc:499] Master 127.16.19.254:42069 was elected leader, sending a full tablet report...
W20250626 01:59:46.470532 21372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:59:46.470760 21370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:59:46.473207 21363 server_base.cc:1048] running on GCE node
I20250626 01:59:47.636237 21363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 01:59:47.639093 21363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 01:59:47.640601 21363 hybrid_clock.cc:648] HybridClock initialized: now 1750903187640553 us; error 63 us; skew 500 ppm
I20250626 01:59:47.641467 21363 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:59:47.652987 21363 webserver.cc:469] Webserver started at http://127.16.19.195:44539/ using document root <none> and password file <none>
I20250626 01:59:47.653960 21363 fs_manager.cc:362] Metadata directory not provided
I20250626 01:59:47.654170 21363 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:59:47.654626 21363 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:59:47.659078 21363 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "b6f341f2c2824f1698d332d083f39a2a"
format_stamp: "Formatted at 2025-06-26 01:59:47 on dist-test-slave-h4rq"
I20250626 01:59:47.660171 21363 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "b6f341f2c2824f1698d332d083f39a2a"
format_stamp: "Formatted at 2025-06-26 01:59:47 on dist-test-slave-h4rq"
I20250626 01:59:47.667505 21363 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s
I20250626 01:59:47.673475 21379 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:59:47.674615 21363 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250626 01:59:47.674966 21363 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "b6f341f2c2824f1698d332d083f39a2a"
format_stamp: "Formatted at 2025-06-26 01:59:47 on dist-test-slave-h4rq"
I20250626 01:59:47.675287 21363 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:59:47.750516 21363 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:59:47.752054 21363 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 01:59:47.752497 21363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:59:47.755249 21363 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:59:47.759402 21363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:59:47.759616 21363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:59:47.759876 21363 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:59:47.760035 21363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:59:47.897518 21363 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.195:38649
I20250626 01:59:47.897629 21491 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.195:38649 every 8 connection(s)
I20250626 01:59:47.900058 21363 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250626 01:59:47.909019 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 21363
I20250626 01:59:47.909581 16463 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250626 01:59:47.917363 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.19.196:0
--local_ip_for_outbound_sockets=127.16.19.196
--webserver_interface=127.16.19.196
--webserver_port=0
--tserver_master_addrs=127.16.19.254:42069
--builtin_ntp_servers=127.16.19.212:39835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250626 01:59:47.925032 21492 heartbeater.cc:344] Connected to a master server at 127.16.19.254:42069
I20250626 01:59:47.925621 21492 heartbeater.cc:461] Registering TS with master...
I20250626 01:59:47.926986 21492 heartbeater.cc:507] Master 127.16.19.254:42069 requested a full tablet report, sending...
I20250626 01:59:47.929141 21038 ts_manager.cc:194] Registered new tserver with Master: b6f341f2c2824f1698d332d083f39a2a (127.16.19.195:38649)
I20250626 01:59:47.930426 21038 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.19.195:47631
W20250626 01:59:48.227670 21496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:59:48.228186 21496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 01:59:48.228746 21496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:59:48.261804 21496 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250626 01:59:48.262228 21496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 01:59:48.263031 21496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.19.196
I20250626 01:59:48.299132 21496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:39835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.19.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.16.19.196
--webserver_port=0
--tserver_master_addrs=127.16.19.254:42069
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.19.196
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 01:59:48.300460 21496 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 01:59:48.302124 21496 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:59:48.318162 21503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:59:48.934157 21492 heartbeater.cc:499] Master 127.16.19.254:42069 was elected leader, sending a full tablet report...
W20250626 01:59:48.318239 21504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:59:48.320454 21496 server_base.cc:1048] running on GCE node
W20250626 01:59:48.319448 21506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:59:49.481395 21496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 01:59:49.484153 21496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 01:59:49.485590 21496 hybrid_clock.cc:648] HybridClock initialized: now 1750903189485545 us; error 65 us; skew 500 ppm
I20250626 01:59:49.486387 21496 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:59:49.492686 21496 webserver.cc:469] Webserver started at http://127.16.19.196:35609/ using document root <none> and password file <none>
I20250626 01:59:49.493610 21496 fs_manager.cc:362] Metadata directory not provided
I20250626 01:59:49.493805 21496 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:59:49.494221 21496 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:59:49.498739 21496 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "7d9f03440ea3491eb366aaf761a00773"
format_stamp: "Formatted at 2025-06-26 01:59:49 on dist-test-slave-h4rq"
I20250626 01:59:49.499809 21496 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "7d9f03440ea3491eb366aaf761a00773"
format_stamp: "Formatted at 2025-06-26 01:59:49 on dist-test-slave-h4rq"
I20250626 01:59:49.506950 21496 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20250626 01:59:49.512732 21513 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:59:49.513749 21496 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250626 01:59:49.514072 21496 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "7d9f03440ea3491eb366aaf761a00773"
format_stamp: "Formatted at 2025-06-26 01:59:49 on dist-test-slave-h4rq"
I20250626 01:59:49.514407 21496 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:59:49.567111 21496 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:59:49.568575 21496 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 01:59:49.569046 21496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:59:49.571560 21496 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:59:49.575688 21496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:59:49.575949 21496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:59:49.576256 21496 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:59:49.576536 21496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250626 01:59:49.712778 21496 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.196:39875
I20250626 01:59:49.712884 21625 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.196:39875 every 8 connection(s)
I20250626 01:59:49.715327 21496 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250626 01:59:49.717267 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 21496
I20250626 01:59:49.718184 16463 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250626 01:59:49.727272 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.19.197:0
--local_ip_for_outbound_sockets=127.16.19.197
--webserver_interface=127.16.19.197
--webserver_port=0
--tserver_master_addrs=127.16.19.254:42069
--builtin_ntp_servers=127.16.19.212:39835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250626 01:59:49.740070 21626 heartbeater.cc:344] Connected to a master server at 127.16.19.254:42069
I20250626 01:59:49.740665 21626 heartbeater.cc:461] Registering TS with master...
I20250626 01:59:49.742010 21626 heartbeater.cc:507] Master 127.16.19.254:42069 requested a full tablet report, sending...
I20250626 01:59:49.744688 21038 ts_manager.cc:194] Registered new tserver with Master: 7d9f03440ea3491eb366aaf761a00773 (127.16.19.196:39875)
I20250626 01:59:49.746563 21038 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.19.196:56701
W20250626 01:59:50.038172 21630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:59:50.038640 21630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 01:59:50.039085 21630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:59:50.073968 21630 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250626 01:59:50.074324 21630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 01:59:50.075074 21630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.19.197
I20250626 01:59:50.111302 21630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:39835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.19.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.16.19.197
--webserver_port=0
--tserver_master_addrs=127.16.19.254:42069
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.19.197
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 01:59:50.112572 21630 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 01:59:50.114144 21630 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:59:50.131156 21636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:59:50.750515 21626 heartbeater.cc:499] Master 127.16.19.254:42069 was elected leader, sending a full tablet report...
W20250626 01:59:50.132217 21637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:59:51.306008 21638 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 01:59:51.306069 21630 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250626 01:59:50.131264 21639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:59:51.311046 21630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 01:59:51.313846 21630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 01:59:51.315284 21630 hybrid_clock.cc:648] HybridClock initialized: now 1750903191315260 us; error 41 us; skew 500 ppm
I20250626 01:59:51.316077 21630 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:59:51.322579 21630 webserver.cc:469] Webserver started at http://127.16.19.197:44143/ using document root <none> and password file <none>
I20250626 01:59:51.323454 21630 fs_manager.cc:362] Metadata directory not provided
I20250626 01:59:51.323650 21630 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:59:51.324056 21630 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:59:51.328878 21630 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "33cb3d159e8e496696852d376e3120a9"
format_stamp: "Formatted at 2025-06-26 01:59:51 on dist-test-slave-h4rq"
I20250626 01:59:51.329962 21630 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "33cb3d159e8e496696852d376e3120a9"
format_stamp: "Formatted at 2025-06-26 01:59:51 on dist-test-slave-h4rq"
I20250626 01:59:51.337021 21630 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250626 01:59:51.342561 21647 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:59:51.343577 21630 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250626 01:59:51.343878 21630 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "33cb3d159e8e496696852d376e3120a9"
format_stamp: "Formatted at 2025-06-26 01:59:51 on dist-test-slave-h4rq"
I20250626 01:59:51.344202 21630 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:59:51.400774 21630 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:59:51.402261 21630 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 01:59:51.402706 21630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:59:51.405236 21630 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:59:51.409436 21630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:59:51.409643 21630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:59:51.409904 21630 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:59:51.410066 21630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:59:51.549023 21630 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.197:45591
I20250626 01:59:51.549155 21759 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.197:45591 every 8 connection(s)
I20250626 01:59:51.551723 21630 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250626 01:59:51.559490 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 21630
I20250626 01:59:51.559904 16463 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250626 01:59:51.572882 21760 heartbeater.cc:344] Connected to a master server at 127.16.19.254:42069
I20250626 01:59:51.573310 21760 heartbeater.cc:461] Registering TS with master...
I20250626 01:59:51.574288 21760 heartbeater.cc:507] Master 127.16.19.254:42069 requested a full tablet report, sending...
I20250626 01:59:51.576393 21038 ts_manager.cc:194] Registered new tserver with Master: 33cb3d159e8e496696852d376e3120a9 (127.16.19.197:45591)
I20250626 01:59:51.577941 21038 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.19.197:52937
I20250626 01:59:51.581458 16463 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250626 01:59:51.618175 21037 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35170:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250626 01:59:51.721174 21161 tablet_service.cc:1468] Processing CreateTablet for tablet d761c4d87b38429e84d423cd5b633bfa (DEFAULT_TABLE table=TestTable [id=127974bef24b46859d27920ace5aaa57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 01:59:51.721592 21294 tablet_service.cc:1468] Processing CreateTablet for tablet d761c4d87b38429e84d423cd5b633bfa (DEFAULT_TABLE table=TestTable [id=127974bef24b46859d27920ace5aaa57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 01:59:51.721309 21427 tablet_service.cc:1468] Processing CreateTablet for tablet d761c4d87b38429e84d423cd5b633bfa (DEFAULT_TABLE table=TestTable [id=127974bef24b46859d27920ace5aaa57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 01:59:51.723292 21161 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d761c4d87b38429e84d423cd5b633bfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:59:51.723413 21427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d761c4d87b38429e84d423cd5b633bfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:59:51.723460 21294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d761c4d87b38429e84d423cd5b633bfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:59:51.748121 21779 tablet_bootstrap.cc:492] T d761c4d87b38429e84d423cd5b633bfa P 05c9f80da92d47d88a526a913bba5387: Bootstrap starting.
I20250626 01:59:51.753722 21780 tablet_bootstrap.cc:492] T d761c4d87b38429e84d423cd5b633bfa P b6f341f2c2824f1698d332d083f39a2a: Bootstrap starting.
I20250626 01:59:51.754285 21779 tablet_bootstrap.cc:654] T d761c4d87b38429e84d423cd5b633bfa P 05c9f80da92d47d88a526a913bba5387: Neither blocks nor log segments found. Creating new log.
I20250626 01:59:51.756803 21779 log.cc:826] T d761c4d87b38429e84d423cd5b633bfa P 05c9f80da92d47d88a526a913bba5387: Log is configured to *not* fsync() on all Append() calls
I20250626 01:59:51.762234 21780 tablet_bootstrap.cc:654] T d761c4d87b38429e84d423cd5b633bfa P b6f341f2c2824f1698d332d083f39a2a: Neither blocks nor log segments found. Creating new log.
I20250626 01:59:51.763350 21781 tablet_bootstrap.cc:492] T d761c4d87b38429e84d423cd5b633bfa P df2d94c744ba4a2085d8f37cda8b5472: Bootstrap starting.
I20250626 01:59:51.765369 21780 log.cc:826] T d761c4d87b38429e84d423cd5b633bfa P b6f341f2c2824f1698d332d083f39a2a: Log is configured to *not* fsync() on all Append() calls
I20250626 01:59:51.768832 21779 tablet_bootstrap.cc:492] T d761c4d87b38429e84d423cd5b633bfa P 05c9f80da92d47d88a526a913bba5387: No bootstrap required, opened a new log
I20250626 01:59:51.769364 21779 ts_tablet_manager.cc:1397] T d761c4d87b38429e84d423cd5b633bfa P 05c9f80da92d47d88a526a913bba5387: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.011s
I20250626 01:59:51.770265 21781 tablet_bootstrap.cc:654] T d761c4d87b38429e84d423cd5b633bfa P df2d94c744ba4a2085d8f37cda8b5472: Neither blocks nor log segments found. Creating new log.
I20250626 01:59:51.770418 21780 tablet_bootstrap.cc:492] T d761c4d87b38429e84d423cd5b633bfa P b6f341f2c2824f1698d332d083f39a2a: No bootstrap required, opened a new log
I20250626 01:59:51.770856 21780 ts_tablet_manager.cc:1397] T d761c4d87b38429e84d423cd5b633bfa P b6f341f2c2824f1698d332d083f39a2a: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.004s
I20250626 01:59:51.772734 21781 log.cc:826] T d761c4d87b38429e84d423cd5b633bfa P df2d94c744ba4a2085d8f37cda8b5472: Log is configured to *not* fsync() on all Append() calls
I20250626 01:59:51.780280 21781 tablet_bootstrap.cc:492] T d761c4d87b38429e84d423cd5b633bfa P df2d94c744ba4a2085d8f37cda8b5472: No bootstrap required, opened a new log
I20250626 01:59:51.780862 21781 ts_tablet_manager.cc:1397] T d761c4d87b38429e84d423cd5b633bfa P df2d94c744ba4a2085d8f37cda8b5472: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.007s
I20250626 01:59:51.789928 21780 raft_consensus.cc:357] T d761c4d87b38429e84d423cd5b633bfa P b6f341f2c2824f1698d332d083f39a2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6f341f2c2824f1698d332d083f39a2a" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 38649 } } peers { permanent_uuid: "05c9f80da92d47d88a526a913bba5387" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 40551 } } peers { permanent_uuid: "df2d94c744ba4a2085d8f37cda8b5472" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 35583 } }
I20250626 01:59:51.790776 21780 raft_consensus.cc:383] T d761c4d87b38429e84d423cd5b633bfa P b6f341f2c2824f1698d332d083f39a2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:59:51.791121 21780 raft_consensus.cc:738] T d761c4d87b38429e84d423cd5b633bfa P b6f341f2c2824f1698d332d083f39a2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6f341f2c2824f1698d332d083f39a2a, State: Initialized, Role: FOLLOWER
I20250626 01:59:51.792093 21780 consensus_queue.cc:260] T d761c4d87b38429e84d423cd5b633bfa P b6f341f2c2824f1698d332d083f39a2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6f341f2c2824f1698d332d083f39a2a" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 38649 } } peers { permanent_uuid: "05c9f80da92d47d88a526a913bba5387" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 40551 } } peers { permanent_uuid: "df2d94c744ba4a2085d8f37cda8b5472" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 35583 } }
I20250626 01:59:51.796027 21780 ts_tablet_manager.cc:1428] T d761c4d87b38429e84d423cd5b633bfa P b6f341f2c2824f1698d332d083f39a2a: Time spent starting tablet: real 0.025s user 0.014s sys 0.009s
I20250626 01:59:51.796550 21779 raft_consensus.cc:357] T d761c4d87b38429e84d423cd5b633bfa P 05c9f80da92d47d88a526a913bba5387 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6f341f2c2824f1698d332d083f39a2a" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 38649 } } peers { permanent_uuid: "05c9f80da92d47d88a526a913bba5387" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 40551 } } peers { permanent_uuid: "df2d94c744ba4a2085d8f37cda8b5472" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 35583 } }
I20250626 01:59:51.797430 21779 raft_consensus.cc:383] T d761c4d87b38429e84d423cd5b633bfa P 05c9f80da92d47d88a526a913bba5387 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:59:51.797665 21779 raft_consensus.cc:738] T d761c4d87b38429e84d423cd5b633bfa P 05c9f80da92d47d88a526a913bba5387 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05c9f80da92d47d88a526a913bba5387, State: Initialized, Role: FOLLOWER
I20250626 01:59:51.798548 21779 consensus_queue.cc:260] T d761c4d87b38429e84d423cd5b633bfa P 05c9f80da92d47d88a526a913bba5387 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6f341f2c2824f1698d332d083f39a2a" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 38649 } } peers { permanent_uuid: "05c9f80da92d47d88a526a913bba5387" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 40551 } } peers { permanent_uuid: "df2d94c744ba4a2085d8f37cda8b5472" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 35583 } }
I20250626 01:59:51.803128 21779 ts_tablet_manager.cc:1428] T d761c4d87b38429e84d423cd5b633bfa P 05c9f80da92d47d88a526a913bba5387: Time spent starting tablet: real 0.033s user 0.028s sys 0.004s
I20250626 01:59:51.808039 21781 raft_consensus.cc:357] T d761c4d87b38429e84d423cd5b633bfa P df2d94c744ba4a2085d8f37cda8b5472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6f341f2c2824f1698d332d083f39a2a" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 38649 } } peers { permanent_uuid: "05c9f80da92d47d88a526a913bba5387" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 40551 } } peers { permanent_uuid: "df2d94c744ba4a2085d8f37cda8b5472" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 35583 } }
I20250626 01:59:51.808765 21781 raft_consensus.cc:383] T d761c4d87b38429e84d423cd5b633bfa P df2d94c744ba4a2085d8f37cda8b5472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:59:51.809047 21781 raft_consensus.cc:738] T d761c4d87b38429e84d423cd5b633bfa P df2d94c744ba4a2085d8f37cda8b5472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2d94c744ba4a2085d8f37cda8b5472, State: Initialized, Role: FOLLOWER
I20250626 01:59:51.809895 21781 consensus_queue.cc:260] T d761c4d87b38429e84d423cd5b633bfa P df2d94c744ba4a2085d8f37cda8b5472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6f341f2c2824f1698d332d083f39a2a" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 38649 } } peers { permanent_uuid: "05c9f80da92d47d88a526a913bba5387" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 40551 } } peers { permanent_uuid: "df2d94c744ba4a2085d8f37cda8b5472" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 35583 } }
I20250626 01:59:51.814174 21781 ts_tablet_manager.cc:1428] T d761c4d87b38429e84d423cd5b633bfa P df2d94c744ba4a2085d8f37cda8b5472: Time spent starting tablet: real 0.033s user 0.028s sys 0.002s
W20250626 01:59:51.828466 21360 tablet.cc:2378] T d761c4d87b38429e84d423cd5b633bfa P df2d94c744ba4a2085d8f37cda8b5472: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 01:59:51.865454 21785 raft_consensus.cc:491] T d761c4d87b38429e84d423cd5b633bfa P b6f341f2c2824f1698d332d083f39a2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:59:51.865954 21785 raft_consensus.cc:513] T d761c4d87b38429e84d423cd5b633bfa P b6f341f2c2824f1698d332d083f39a2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6f341f2c2824f1698d332d083f39a2a" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 38649 } } peers { permanent_uuid: "05c9f80da92d47d88a526a913bba5387" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 40551 } } peers { permanent_uuid: "df2d94c744ba4a2085d8f37cda8b5472" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 35583 } }
I20250626 01:59:51.868294 21785 leader_election.cc:290] T d761c4d87b38429e84d423cd5b633bfa P b6f341f2c2824f1698d332d083f39a2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05c9f80da92d47d88a526a913bba5387 (127.16.19.193:40551), df2d94c744ba4a2085d8f37cda8b5472 (127.16.19.194:35583)
I20250626 01:59:51.881469 21314 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d761c4d87b38429e84d423cd5b633bfa" candidate_uuid: "b6f341f2c2824f1698d332d083f39a2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2d94c744ba4a2085d8f37cda8b5472" is_pre_election: true
I20250626 01:59:51.881628 21181 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d761c4d87b38429e84d423cd5b633bfa" candidate_uuid: "b6f341f2c2824f1698d332d083f39a2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05c9f80da92d47d88a526a913bba5387" is_pre_election: true
I20250626 01:59:51.882427 21181 raft_consensus.cc:2466] T d761c4d87b38429e84d423cd5b633bfa P 05c9f80da92d47d88a526a913bba5387 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6f341f2c2824f1698d332d083f39a2a in term 0.
I20250626 01:59:51.882428 21314 raft_consensus.cc:2466] T d761c4d87b38429e84d423cd5b633bfa P df2d94c744ba4a2085d8f37cda8b5472 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6f341f2c2824f1698d332d083f39a2a in term 0.
I20250626 01:59:51.883694 21383 leader_election.cc:304] T d761c4d87b38429e84d423cd5b633bfa P b6f341f2c2824f1698d332d083f39a2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6f341f2c2824f1698d332d083f39a2a, df2d94c744ba4a2085d8f37cda8b5472; no voters:
I20250626 01:59:51.884357 21785 raft_consensus.cc:2802] T d761c4d87b38429e84d423cd5b633bfa P b6f341f2c2824f1698d332d083f39a2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:59:51.884650 21785 raft_consensus.cc:491] T d761c4d87b38429e84d423cd5b633bfa P b6f341f2c2824f1698d332d083f39a2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:59:51.884899 21785 raft_consensus.cc:3058] T d761c4d87b38429e84d423cd5b633bfa P b6f341f2c2824f1698d332d083f39a2a [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:59:51.889231 21785 raft_consensus.cc:513] T d761c4d87b38429e84d423cd5b633bfa P b6f341f2c2824f1698d332d083f39a2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6f341f2c2824f1698d332d083f39a2a" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 38649 } } peers { permanent_uuid: "05c9f80da92d47d88a526a913bba5387" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 40551 } } peers { permanent_uuid: "df2d94c744ba4a2085d8f37cda8b5472" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 35583 } }
I20250626 01:59:51.890708 21785 leader_election.cc:290] T d761c4d87b38429e84d423cd5b633bfa P b6f341f2c2824f1698d332d083f39a2a [CANDIDATE]: Term 1 election: Requested vote from peers 05c9f80da92d47d88a526a913bba5387 (127.16.19.193:40551), df2d94c744ba4a2085d8f37cda8b5472 (127.16.19.194:35583)
I20250626 01:59:51.891417 21181 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d761c4d87b38429e84d423cd5b633bfa" candidate_uuid: "b6f341f2c2824f1698d332d083f39a2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05c9f80da92d47d88a526a913bba5387"
I20250626 01:59:51.891690 21314 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d761c4d87b38429e84d423cd5b633bfa" candidate_uuid: "b6f341f2c2824f1698d332d083f39a2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2d94c744ba4a2085d8f37cda8b5472"
I20250626 01:59:51.891958 21181 raft_consensus.cc:3058] T d761c4d87b38429e84d423cd5b633bfa P 05c9f80da92d47d88a526a913bba5387 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:59:51.892206 21314 raft_consensus.cc:3058] T d761c4d87b38429e84d423cd5b633bfa P df2d94c744ba4a2085d8f37cda8b5472 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:59:51.899094 21181 raft_consensus.cc:2466] T d761c4d87b38429e84d423cd5b633bfa P 05c9f80da92d47d88a526a913bba5387 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6f341f2c2824f1698d332d083f39a2a in term 1.
I20250626 01:59:51.899104 21314 raft_consensus.cc:2466] T d761c4d87b38429e84d423cd5b633bfa P df2d94c744ba4a2085d8f37cda8b5472 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6f341f2c2824f1698d332d083f39a2a in term 1.
I20250626 01:59:51.900293 21382 leader_election.cc:304] T d761c4d87b38429e84d423cd5b633bfa P b6f341f2c2824f1698d332d083f39a2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05c9f80da92d47d88a526a913bba5387, b6f341f2c2824f1698d332d083f39a2a; no voters:
I20250626 01:59:51.901049 21785 raft_consensus.cc:2802] T d761c4d87b38429e84d423cd5b633bfa P b6f341f2c2824f1698d332d083f39a2a [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:59:51.902859 21785 raft_consensus.cc:695] T d761c4d87b38429e84d423cd5b633bfa P b6f341f2c2824f1698d332d083f39a2a [term 1 LEADER]: Becoming Leader. State: Replica: b6f341f2c2824f1698d332d083f39a2a, State: Running, Role: LEADER
I20250626 01:59:51.903856 21785 consensus_queue.cc:237] T d761c4d87b38429e84d423cd5b633bfa P b6f341f2c2824f1698d332d083f39a2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6f341f2c2824f1698d332d083f39a2a" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 38649 } } peers { permanent_uuid: "05c9f80da92d47d88a526a913bba5387" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 40551 } } peers { permanent_uuid: "df2d94c744ba4a2085d8f37cda8b5472" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 35583 } }
W20250626 01:59:51.908293 21493 tablet.cc:2378] T d761c4d87b38429e84d423cd5b633bfa P b6f341f2c2824f1698d332d083f39a2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 01:59:51.918254 21037 catalog_manager.cc:5582] T d761c4d87b38429e84d423cd5b633bfa P b6f341f2c2824f1698d332d083f39a2a reported cstate change: term changed from 0 to 1, leader changed from <none> to b6f341f2c2824f1698d332d083f39a2a (127.16.19.195). New cstate: current_term: 1 leader_uuid: "b6f341f2c2824f1698d332d083f39a2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6f341f2c2824f1698d332d083f39a2a" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 38649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05c9f80da92d47d88a526a913bba5387" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 40551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df2d94c744ba4a2085d8f37cda8b5472" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 35583 } health_report { overall_health: UNKNOWN } } }
I20250626 01:59:51.969854 16463 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250626 01:59:51.973297 16463 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 05c9f80da92d47d88a526a913bba5387 to finish bootstrapping
I20250626 01:59:51.986843 16463 ts_itest-base.cc:246] Waiting for 1 tablets on tserver df2d94c744ba4a2085d8f37cda8b5472 to finish bootstrapping
I20250626 01:59:51.998335 16463 ts_itest-base.cc:246] Waiting for 1 tablets on tserver b6f341f2c2824f1698d332d083f39a2a to finish bootstrapping
I20250626 01:59:52.009459 16463 test_util.cc:276] Using random seed: -355747109
W20250626 01:59:52.012449 21227 tablet.cc:2378] T d761c4d87b38429e84d423cd5b633bfa P 05c9f80da92d47d88a526a913bba5387: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 01:59:52.035250 16463 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20250626 01:59:52.036177 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 21230
W20250626 01:59:52.074185 21383 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.19.194:35583: connect: Connection refused (error 111)
I20250626 01:59:52.075608 21181 raft_consensus.cc:1273] T d761c4d87b38429e84d423cd5b633bfa P 05c9f80da92d47d88a526a913bba5387 [term 1 FOLLOWER]: Refusing update from remote peer b6f341f2c2824f1698d332d083f39a2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 01:59:52.077370 21785 consensus_queue.cc:1035] T d761c4d87b38429e84d423cd5b633bfa P b6f341f2c2824f1698d332d083f39a2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "05c9f80da92d47d88a526a913bba5387" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 40551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 01:59:52.078874 21383 consensus_peers.cc:487] T d761c4d87b38429e84d423cd5b633bfa P b6f341f2c2824f1698d332d083f39a2a -> Peer df2d94c744ba4a2085d8f37cda8b5472 (127.16.19.194:35583): Couldn't send request to peer df2d94c744ba4a2085d8f37cda8b5472. Status: Network error: Client connection negotiation failed: client connection to 127.16.19.194:35583: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 01:59:52.101938 21803 mvcc.cc:204] Tried to move back new op lower bound from 7171699474718539776 to 7171699474048262144. Current Snapshot: MvccSnapshot[applied={T|T < 7171699474718539776}]
I20250626 01:59:52.581523 21760 heartbeater.cc:499] Master 127.16.19.254:42069 was elected leader, sending a full tablet report...
W20250626 01:59:54.542455 21383 consensus_peers.cc:487] T d761c4d87b38429e84d423cd5b633bfa P b6f341f2c2824f1698d332d083f39a2a -> Peer df2d94c744ba4a2085d8f37cda8b5472 (127.16.19.194:35583): Couldn't send request to peer df2d94c744ba4a2085d8f37cda8b5472. Status: Network error: Client connection negotiation failed: client connection to 127.16.19.194:35583: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250626 01:59:54.733366 21695 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 01:59:54.756035 21161 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 01:59:54.758176 21561 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 01:59:54.780328 21427 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250626 01:59:56.535362 21161 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 01:59:56.547389 21695 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 01:59:56.558320 21427 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250626 01:59:56.569723 21561 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250626 01:59:57.293107 21383 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.19.194:35583: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250626 01:59:57.295490 21383 consensus_peers.cc:487] T d761c4d87b38429e84d423cd5b633bfa P b6f341f2c2824f1698d332d083f39a2a -> Peer df2d94c744ba4a2085d8f37cda8b5472 (127.16.19.194:35583): Couldn't send request to peer df2d94c744ba4a2085d8f37cda8b5472. Status: Network error: Client connection negotiation failed: client connection to 127.16.19.194:35583: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250626 01:59:59.044986 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 21097
I20250626 01:59:59.080886 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 21363
I20250626 01:59:59.121445 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 21496
I20250626 01:59:59.146306 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 21630
I20250626 01:59:59.171638 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 21005
2025-06-26T01:59:59Z chronyd exiting
[ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4 (18885 ms)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest (18885 ms total)
[----------] 1 test from ListTableCliSimpleParamTest
[ RUN ] ListTableCliSimpleParamTest.TestListTables/2
I20250626 01:59:59.234823 16463 test_util.cc:276] Using random seed: -348521749
I20250626 01:59:59.239041 16463 ts_itest-base.cc:115] Starting cluster with:
I20250626 01:59:59.239212 16463 ts_itest-base.cc:116] --------------
I20250626 01:59:59.239329 16463 ts_itest-base.cc:117] 1 tablet servers
I20250626 01:59:59.239437 16463 ts_itest-base.cc:118] 1 replicas per TS
I20250626 01:59:59.239547 16463 ts_itest-base.cc:119] --------------
2025-06-26T01:59:59Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-26T01:59:59Z Disabled control of system clock
I20250626 01:59:59.285203 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.19.254:40513
--webserver_interface=127.16.19.254
--webserver_port=0
--builtin_ntp_servers=127.16.19.212:33073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.19.254:40513 with env {}
W20250626 01:59:59.589187 21909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 01:59:59.589740 21909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 01:59:59.590167 21909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 01:59:59.621277 21909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 01:59:59.621568 21909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 01:59:59.621774 21909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 01:59:59.621978 21909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 01:59:59.657562 21909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:33073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.19.254:40513
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.19.254:40513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.19.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 01:59:59.659015 21909 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 01:59:59.660629 21909 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:59:59.675647 21918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:59:59.675690 21916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:59:59.675994 21915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:00:00.880148 21917 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1202 milliseconds
I20250626 02:00:00.880278 21909 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 02:00:00.881558 21909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 02:00:00.884202 21909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 02:00:00.885567 21909 hybrid_clock.cc:648] HybridClock initialized: now 1750903200885505 us; error 70 us; skew 500 ppm
I20250626 02:00:00.886363 21909 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 02:00:00.896472 21909 webserver.cc:469] Webserver started at http://127.16.19.254:39425/ using document root <none> and password file <none>
I20250626 02:00:00.897485 21909 fs_manager.cc:362] Metadata directory not provided
I20250626 02:00:00.897711 21909 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:00:00.898193 21909 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:00:00.902637 21909 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "8bacdcd079624e828387a7919c585426"
format_stamp: "Formatted at 2025-06-26 02:00:00 on dist-test-slave-h4rq"
I20250626 02:00:00.903721 21909 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "8bacdcd079624e828387a7919c585426"
format_stamp: "Formatted at 2025-06-26 02:00:00 on dist-test-slave-h4rq"
I20250626 02:00:00.910925 21909 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20250626 02:00:00.916453 21925 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:00:00.917470 21909 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250626 02:00:00.917790 21909 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
uuid: "8bacdcd079624e828387a7919c585426"
format_stamp: "Formatted at 2025-06-26 02:00:00 on dist-test-slave-h4rq"
I20250626 02:00:00.918135 21909 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:00:00.967741 21909 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:00:00.969276 21909 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 02:00:00.969717 21909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:00:01.040724 21909 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.254:40513
I20250626 02:00:01.040763 21976 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.254:40513 every 8 connection(s)
I20250626 02:00:01.043520 21909 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250626 02:00:01.048816 21977 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:00:01.049043 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 21909
I20250626 02:00:01.049546 16463 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750903057700330-16463-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250626 02:00:01.074470 21977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bacdcd079624e828387a7919c585426: Bootstrap starting.
I20250626 02:00:01.080181 21977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bacdcd079624e828387a7919c585426: Neither blocks nor log segments found. Creating new log.
I20250626 02:00:01.082015 21977 log.cc:826] T 00000000000000000000000000000000 P 8bacdcd079624e828387a7919c585426: Log is configured to *not* fsync() on all Append() calls
I20250626 02:00:01.086618 21977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bacdcd079624e828387a7919c585426: No bootstrap required, opened a new log
I20250626 02:00:01.104125 21977 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8bacdcd079624e828387a7919c585426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bacdcd079624e828387a7919c585426" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 40513 } }
I20250626 02:00:01.104851 21977 raft_consensus.cc:383] T 00000000000000000000000000000000 P 8bacdcd079624e828387a7919c585426 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:00:01.105065 21977 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8bacdcd079624e828387a7919c585426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bacdcd079624e828387a7919c585426, State: Initialized, Role: FOLLOWER
I20250626 02:00:01.105701 21977 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bacdcd079624e828387a7919c585426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bacdcd079624e828387a7919c585426" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 40513 } }
I20250626 02:00:01.106177 21977 raft_consensus.cc:397] T 00000000000000000000000000000000 P 8bacdcd079624e828387a7919c585426 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 02:00:01.106426 21977 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8bacdcd079624e828387a7919c585426 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 02:00:01.106796 21977 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 8bacdcd079624e828387a7919c585426 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:00:01.111328 21977 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8bacdcd079624e828387a7919c585426 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bacdcd079624e828387a7919c585426" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 40513 } }
I20250626 02:00:01.112015 21977 leader_election.cc:304] T 00000000000000000000000000000000 P 8bacdcd079624e828387a7919c585426 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bacdcd079624e828387a7919c585426; no voters:
I20250626 02:00:01.113714 21977 leader_election.cc:290] T 00000000000000000000000000000000 P 8bacdcd079624e828387a7919c585426 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 02:00:01.114425 21982 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 8bacdcd079624e828387a7919c585426 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:00:01.117033 21982 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8bacdcd079624e828387a7919c585426 [term 1 LEADER]: Becoming Leader. State: Replica: 8bacdcd079624e828387a7919c585426, State: Running, Role: LEADER
I20250626 02:00:01.117969 21977 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bacdcd079624e828387a7919c585426 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 02:00:01.117924 21982 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bacdcd079624e828387a7919c585426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bacdcd079624e828387a7919c585426" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 40513 } }
I20250626 02:00:01.129396 21983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bacdcd079624e828387a7919c585426 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bacdcd079624e828387a7919c585426" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bacdcd079624e828387a7919c585426" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 40513 } } }
I20250626 02:00:01.129905 21984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bacdcd079624e828387a7919c585426 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bacdcd079624e828387a7919c585426. Latest consensus state: current_term: 1 leader_uuid: "8bacdcd079624e828387a7919c585426" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bacdcd079624e828387a7919c585426" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 40513 } } }
I20250626 02:00:01.130494 21983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bacdcd079624e828387a7919c585426 [sys.catalog]: This master's current role is: LEADER
I20250626 02:00:01.130841 21984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bacdcd079624e828387a7919c585426 [sys.catalog]: This master's current role is: LEADER
I20250626 02:00:01.142087 21991 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 02:00:01.154039 21991 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 02:00:01.170902 21991 catalog_manager.cc:1349] Generated new cluster ID: 1b591ffa291b4f569e3cb537c67e5d5c
I20250626 02:00:01.171241 21991 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 02:00:01.197405 21991 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 02:00:01.198918 21991 catalog_manager.cc:1506] Loading token signing keys...
I20250626 02:00:01.211624 21991 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 8bacdcd079624e828387a7919c585426: Generated new TSK 0
I20250626 02:00:01.212544 21991 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 02:00:01.220356 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.19.193:0
--local_ip_for_outbound_sockets=127.16.19.193
--webserver_interface=127.16.19.193
--webserver_port=0
--tserver_master_addrs=127.16.19.254:40513
--builtin_ntp_servers=127.16.19.212:33073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250626 02:00:01.543161 22001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 02:00:01.543686 22001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 02:00:01.544185 22001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 02:00:01.576143 22001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 02:00:01.577083 22001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.19.193
I20250626 02:00:01.612243 22001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:33073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.19.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.19.193
--webserver_port=0
--tserver_master_addrs=127.16.19.254:40513
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.19.193
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 02:00:01.613596 22001 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 02:00:01.615226 22001 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:00:01.633545 22008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:00:01.636417 22007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:00:01.635207 22010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:00:02.841812 22009 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 02:00:02.842417 22001 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 02:00:02.846233 22001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 02:00:02.848549 22001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 02:00:02.849973 22001 hybrid_clock.cc:648] HybridClock initialized: now 1750903202849919 us; error 70 us; skew 500 ppm
I20250626 02:00:02.850787 22001 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 02:00:02.858059 22001 webserver.cc:469] Webserver started at http://127.16.19.193:34465/ using document root <none> and password file <none>
I20250626 02:00:02.859009 22001 fs_manager.cc:362] Metadata directory not provided
I20250626 02:00:02.859234 22001 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:00:02.859750 22001 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:00:02.864529 22001 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b6cbda4fd7384d039d0e75935be4d7ac"
format_stamp: "Formatted at 2025-06-26 02:00:02 on dist-test-slave-h4rq"
I20250626 02:00:02.865825 22001 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b6cbda4fd7384d039d0e75935be4d7ac"
format_stamp: "Formatted at 2025-06-26 02:00:02 on dist-test-slave-h4rq"
I20250626 02:00:02.873461 22001 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20250626 02:00:02.879853 22017 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:00:02.881038 22001 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20250626 02:00:02.881357 22001 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b6cbda4fd7384d039d0e75935be4d7ac"
format_stamp: "Formatted at 2025-06-26 02:00:02 on dist-test-slave-h4rq"
I20250626 02:00:02.881688 22001 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:00:02.930781 22001 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:00:02.932261 22001 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 02:00:02.932760 22001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:00:02.935926 22001 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:00:02.941052 22001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:00:02.941283 22001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:00:02.941530 22001 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:00:02.941692 22001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:00:03.086165 22001 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.193:38059
I20250626 02:00:03.086273 22129 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.193:38059 every 8 connection(s)
I20250626 02:00:03.088810 22001 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250626 02:00:03.092659 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 22001
I20250626 02:00:03.093191 16463 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750903057700330-16463-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250626 02:00:03.110020 22130 heartbeater.cc:344] Connected to a master server at 127.16.19.254:40513
I20250626 02:00:03.110481 22130 heartbeater.cc:461] Registering TS with master...
I20250626 02:00:03.111557 22130 heartbeater.cc:507] Master 127.16.19.254:40513 requested a full tablet report, sending...
I20250626 02:00:03.114043 21942 ts_manager.cc:194] Registered new tserver with Master: b6cbda4fd7384d039d0e75935be4d7ac (127.16.19.193:38059)
I20250626 02:00:03.116006 21942 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.19.193:40509
I20250626 02:00:03.126689 16463 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250626 02:00:03.160024 21942 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58700:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250626 02:00:03.217480 22065 tablet_service.cc:1468] Processing CreateTablet for tablet 5c8d58810c8f4c9a9514bf1ac6ca3e7e (DEFAULT_TABLE table=TestTable [id=fae455bba429489e88d5916264396181]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 02:00:03.219130 22065 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c8d58810c8f4c9a9514bf1ac6ca3e7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:00:03.239071 22145 tablet_bootstrap.cc:492] T 5c8d58810c8f4c9a9514bf1ac6ca3e7e P b6cbda4fd7384d039d0e75935be4d7ac: Bootstrap starting.
I20250626 02:00:03.245162 22145 tablet_bootstrap.cc:654] T 5c8d58810c8f4c9a9514bf1ac6ca3e7e P b6cbda4fd7384d039d0e75935be4d7ac: Neither blocks nor log segments found. Creating new log.
I20250626 02:00:03.247025 22145 log.cc:826] T 5c8d58810c8f4c9a9514bf1ac6ca3e7e P b6cbda4fd7384d039d0e75935be4d7ac: Log is configured to *not* fsync() on all Append() calls
I20250626 02:00:03.251758 22145 tablet_bootstrap.cc:492] T 5c8d58810c8f4c9a9514bf1ac6ca3e7e P b6cbda4fd7384d039d0e75935be4d7ac: No bootstrap required, opened a new log
I20250626 02:00:03.252189 22145 ts_tablet_manager.cc:1397] T 5c8d58810c8f4c9a9514bf1ac6ca3e7e P b6cbda4fd7384d039d0e75935be4d7ac: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20250626 02:00:03.270192 22145 raft_consensus.cc:357] T 5c8d58810c8f4c9a9514bf1ac6ca3e7e P b6cbda4fd7384d039d0e75935be4d7ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6cbda4fd7384d039d0e75935be4d7ac" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 38059 } }
I20250626 02:00:03.270828 22145 raft_consensus.cc:383] T 5c8d58810c8f4c9a9514bf1ac6ca3e7e P b6cbda4fd7384d039d0e75935be4d7ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:00:03.271075 22145 raft_consensus.cc:738] T 5c8d58810c8f4c9a9514bf1ac6ca3e7e P b6cbda4fd7384d039d0e75935be4d7ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6cbda4fd7384d039d0e75935be4d7ac, State: Initialized, Role: FOLLOWER
I20250626 02:00:03.271801 22145 consensus_queue.cc:260] T 5c8d58810c8f4c9a9514bf1ac6ca3e7e P b6cbda4fd7384d039d0e75935be4d7ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6cbda4fd7384d039d0e75935be4d7ac" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 38059 } }
I20250626 02:00:03.272344 22145 raft_consensus.cc:397] T 5c8d58810c8f4c9a9514bf1ac6ca3e7e P b6cbda4fd7384d039d0e75935be4d7ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 02:00:03.272662 22145 raft_consensus.cc:491] T 5c8d58810c8f4c9a9514bf1ac6ca3e7e P b6cbda4fd7384d039d0e75935be4d7ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 02:00:03.272989 22145 raft_consensus.cc:3058] T 5c8d58810c8f4c9a9514bf1ac6ca3e7e P b6cbda4fd7384d039d0e75935be4d7ac [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:00:03.278254 22145 raft_consensus.cc:513] T 5c8d58810c8f4c9a9514bf1ac6ca3e7e P b6cbda4fd7384d039d0e75935be4d7ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6cbda4fd7384d039d0e75935be4d7ac" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 38059 } }
I20250626 02:00:03.279067 22145 leader_election.cc:304] T 5c8d58810c8f4c9a9514bf1ac6ca3e7e P b6cbda4fd7384d039d0e75935be4d7ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6cbda4fd7384d039d0e75935be4d7ac; no voters:
I20250626 02:00:03.280858 22145 leader_election.cc:290] T 5c8d58810c8f4c9a9514bf1ac6ca3e7e P b6cbda4fd7384d039d0e75935be4d7ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 02:00:03.281250 22147 raft_consensus.cc:2802] T 5c8d58810c8f4c9a9514bf1ac6ca3e7e P b6cbda4fd7384d039d0e75935be4d7ac [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:00:03.283891 22147 raft_consensus.cc:695] T 5c8d58810c8f4c9a9514bf1ac6ca3e7e P b6cbda4fd7384d039d0e75935be4d7ac [term 1 LEADER]: Becoming Leader. State: Replica: b6cbda4fd7384d039d0e75935be4d7ac, State: Running, Role: LEADER
I20250626 02:00:03.284912 22130 heartbeater.cc:499] Master 127.16.19.254:40513 was elected leader, sending a full tablet report...
I20250626 02:00:03.285447 22147 consensus_queue.cc:237] T 5c8d58810c8f4c9a9514bf1ac6ca3e7e P b6cbda4fd7384d039d0e75935be4d7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6cbda4fd7384d039d0e75935be4d7ac" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 38059 } }
I20250626 02:00:03.285925 22145 ts_tablet_manager.cc:1428] T 5c8d58810c8f4c9a9514bf1ac6ca3e7e P b6cbda4fd7384d039d0e75935be4d7ac: Time spent starting tablet: real 0.033s user 0.022s sys 0.013s
I20250626 02:00:03.300357 21942 catalog_manager.cc:5582] T 5c8d58810c8f4c9a9514bf1ac6ca3e7e P b6cbda4fd7384d039d0e75935be4d7ac reported cstate change: term changed from 0 to 1, leader changed from <none> to b6cbda4fd7384d039d0e75935be4d7ac (127.16.19.193). New cstate: current_term: 1 leader_uuid: "b6cbda4fd7384d039d0e75935be4d7ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6cbda4fd7384d039d0e75935be4d7ac" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 38059 } health_report { overall_health: HEALTHY } } }
I20250626 02:00:03.322986 16463 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250626 02:00:03.325974 16463 ts_itest-base.cc:246] Waiting for 1 tablets on tserver b6cbda4fd7384d039d0e75935be4d7ac to finish bootstrapping
I20250626 02:00:06.025863 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 22001
I20250626 02:00:06.053511 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 21909
2025-06-26T02:00:06Z chronyd exiting
[ OK ] ListTableCliSimpleParamTest.TestListTables/2 (6872 ms)
[----------] 1 test from ListTableCliSimpleParamTest (6872 ms total)
[----------] 1 test from ListTableCliParamTest
[ RUN ] ListTableCliParamTest.ListTabletWithPartitionInfo/4
I20250626 02:00:06.107640 16463 test_util.cc:276] Using random seed: -341648934
[ OK ] ListTableCliParamTest.ListTabletWithPartitionInfo/4 (12 ms)
[----------] 1 test from ListTableCliParamTest (12 ms total)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest
[ RUN ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0
2025-06-26T02:00:06Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-26T02:00:06Z Disabled control of system clock
I20250626 02:00:06.160728 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.19.254:32875
--webserver_interface=127.16.19.254
--webserver_port=0
--builtin_ntp_servers=127.16.19.212:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.19.254:32875 with env {}
W20250626 02:00:06.469107 22173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 02:00:06.469715 22173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 02:00:06.470192 22173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 02:00:06.501248 22173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 02:00:06.501579 22173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 02:00:06.501855 22173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 02:00:06.502099 22173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 02:00:06.537458 22173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.19.254:32875
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.19.254:32875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.19.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 02:00:06.538801 22173 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 02:00:06.540452 22173 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:00:06.554744 22180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:00:06.555541 22182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:00:06.554968 22179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:00:07.731688 22181 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 02:00:07.731815 22173 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 02:00:07.736423 22173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 02:00:07.739493 22173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 02:00:07.740914 22173 hybrid_clock.cc:648] HybridClock initialized: now 1750903207740876 us; error 58 us; skew 500 ppm
I20250626 02:00:07.741668 22173 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 02:00:07.752914 22173 webserver.cc:469] Webserver started at http://127.16.19.254:34315/ using document root <none> and password file <none>
I20250626 02:00:07.753809 22173 fs_manager.cc:362] Metadata directory not provided
I20250626 02:00:07.754004 22173 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:00:07.754406 22173 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:00:07.758787 22173 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/master-0/data/instance:
uuid: "6c857fe3072642ba9478fbd1435880f7"
format_stamp: "Formatted at 2025-06-26 02:00:07 on dist-test-slave-h4rq"
I20250626 02:00:07.759862 22173 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/master-0/wal/instance:
uuid: "6c857fe3072642ba9478fbd1435880f7"
format_stamp: "Formatted at 2025-06-26 02:00:07 on dist-test-slave-h4rq"
I20250626 02:00:07.767040 22173 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250626 02:00:07.772590 22189 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:00:07.773682 22173 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250626 02:00:07.774026 22173 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/master-0/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/master-0/wal
uuid: "6c857fe3072642ba9478fbd1435880f7"
format_stamp: "Formatted at 2025-06-26 02:00:07 on dist-test-slave-h4rq"
I20250626 02:00:07.774363 22173 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:00:07.830253 22173 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:00:07.831743 22173 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 02:00:07.832188 22173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:00:07.903226 22173 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.254:32875
I20250626 02:00:07.903301 22240 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.254:32875 every 8 connection(s)
I20250626 02:00:07.905956 22173 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/master-0/data/info.pb
I20250626 02:00:07.907573 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 22173
I20250626 02:00:07.908102 16463 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/master-0/wal/instance
I20250626 02:00:07.912818 22241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:00:07.937041 22241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c857fe3072642ba9478fbd1435880f7: Bootstrap starting.
I20250626 02:00:07.942735 22241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c857fe3072642ba9478fbd1435880f7: Neither blocks nor log segments found. Creating new log.
I20250626 02:00:07.944541 22241 log.cc:826] T 00000000000000000000000000000000 P 6c857fe3072642ba9478fbd1435880f7: Log is configured to *not* fsync() on all Append() calls
I20250626 02:00:07.948891 22241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c857fe3072642ba9478fbd1435880f7: No bootstrap required, opened a new log
I20250626 02:00:07.966965 22241 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6c857fe3072642ba9478fbd1435880f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c857fe3072642ba9478fbd1435880f7" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 32875 } }
I20250626 02:00:07.967638 22241 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6c857fe3072642ba9478fbd1435880f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:00:07.967892 22241 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6c857fe3072642ba9478fbd1435880f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c857fe3072642ba9478fbd1435880f7, State: Initialized, Role: FOLLOWER
I20250626 02:00:07.968564 22241 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c857fe3072642ba9478fbd1435880f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c857fe3072642ba9478fbd1435880f7" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 32875 } }
I20250626 02:00:07.969117 22241 raft_consensus.cc:397] T 00000000000000000000000000000000 P 6c857fe3072642ba9478fbd1435880f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 02:00:07.969383 22241 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6c857fe3072642ba9478fbd1435880f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 02:00:07.969669 22241 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6c857fe3072642ba9478fbd1435880f7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:00:07.973893 22241 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6c857fe3072642ba9478fbd1435880f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c857fe3072642ba9478fbd1435880f7" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 32875 } }
I20250626 02:00:07.974704 22241 leader_election.cc:304] T 00000000000000000000000000000000 P 6c857fe3072642ba9478fbd1435880f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c857fe3072642ba9478fbd1435880f7; no voters:
I20250626 02:00:07.976348 22241 leader_election.cc:290] T 00000000000000000000000000000000 P 6c857fe3072642ba9478fbd1435880f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 02:00:07.977043 22246 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6c857fe3072642ba9478fbd1435880f7 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:00:07.979418 22246 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6c857fe3072642ba9478fbd1435880f7 [term 1 LEADER]: Becoming Leader. State: Replica: 6c857fe3072642ba9478fbd1435880f7, State: Running, Role: LEADER
I20250626 02:00:07.980187 22246 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c857fe3072642ba9478fbd1435880f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c857fe3072642ba9478fbd1435880f7" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 32875 } }
I20250626 02:00:07.981187 22241 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c857fe3072642ba9478fbd1435880f7 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 02:00:07.991097 22247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c857fe3072642ba9478fbd1435880f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c857fe3072642ba9478fbd1435880f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c857fe3072642ba9478fbd1435880f7" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 32875 } } }
I20250626 02:00:07.991369 22248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c857fe3072642ba9478fbd1435880f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c857fe3072642ba9478fbd1435880f7. Latest consensus state: current_term: 1 leader_uuid: "6c857fe3072642ba9478fbd1435880f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c857fe3072642ba9478fbd1435880f7" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 32875 } } }
I20250626 02:00:07.992123 22247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c857fe3072642ba9478fbd1435880f7 [sys.catalog]: This master's current role is: LEADER
I20250626 02:00:07.992137 22248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c857fe3072642ba9478fbd1435880f7 [sys.catalog]: This master's current role is: LEADER
I20250626 02:00:07.996188 22255 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 02:00:08.009286 22255 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 02:00:08.029029 22255 catalog_manager.cc:1349] Generated new cluster ID: a2c6c62f2d09488296b3d9bf0e484070
I20250626 02:00:08.029361 22255 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 02:00:08.061872 22255 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 02:00:08.063654 22255 catalog_manager.cc:1506] Loading token signing keys...
I20250626 02:00:08.081189 22255 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 6c857fe3072642ba9478fbd1435880f7: Generated new TSK 0
I20250626 02:00:08.082408 22255 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 02:00:08.099712 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.19.193:0
--local_ip_for_outbound_sockets=127.16.19.193
--webserver_interface=127.16.19.193
--webserver_port=0
--tserver_master_addrs=127.16.19.254:32875
--builtin_ntp_servers=127.16.19.212:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20250626 02:00:08.429847 22266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 02:00:08.430377 22266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 02:00:08.430893 22266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 02:00:08.462884 22266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 02:00:08.463783 22266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.19.193
I20250626 02:00:08.499189 22266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.19.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.19.193
--webserver_port=0
--tserver_master_addrs=127.16.19.254:32875
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.19.193
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 02:00:08.500500 22266 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 02:00:08.502193 22266 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:00:08.521559 22273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:00:08.525849 22275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:00:08.525969 22266 server_base.cc:1048] running on GCE node
W20250626 02:00:08.524502 22272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:00:09.722537 22266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 02:00:09.725394 22266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 02:00:09.726933 22266 hybrid_clock.cc:648] HybridClock initialized: now 1750903209726901 us; error 36 us; skew 500 ppm
I20250626 02:00:09.728030 22266 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 02:00:09.736053 22266 webserver.cc:469] Webserver started at http://127.16.19.193:34435/ using document root <none> and password file <none>
I20250626 02:00:09.737393 22266 fs_manager.cc:362] Metadata directory not provided
I20250626 02:00:09.737670 22266 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:00:09.738286 22266 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:00:09.745361 22266 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-0/data/instance:
uuid: "8c554ad3849d45fda4e11514db7b2ae4"
format_stamp: "Formatted at 2025-06-26 02:00:09 on dist-test-slave-h4rq"
I20250626 02:00:09.746944 22266 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-0/wal/instance:
uuid: "8c554ad3849d45fda4e11514db7b2ae4"
format_stamp: "Formatted at 2025-06-26 02:00:09 on dist-test-slave-h4rq"
I20250626 02:00:09.756314 22266 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.006s sys 0.005s
I20250626 02:00:09.763831 22282 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:00:09.765060 22266 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250626 02:00:09.765497 22266 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-0/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-0/wal
uuid: "8c554ad3849d45fda4e11514db7b2ae4"
format_stamp: "Formatted at 2025-06-26 02:00:09 on dist-test-slave-h4rq"
I20250626 02:00:09.766010 22266 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:00:09.823936 22266 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:00:09.825438 22266 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 02:00:09.825896 22266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:00:09.828958 22266 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:00:09.833189 22266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:00:09.833410 22266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:00:09.833653 22266 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:00:09.833827 22266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:00:09.984755 22266 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.193:43387
I20250626 02:00:09.984860 22394 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.193:43387 every 8 connection(s)
I20250626 02:00:09.987362 22266 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-0/data/info.pb
I20250626 02:00:09.993321 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 22266
I20250626 02:00:09.993769 16463 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-0/wal/instance
I20250626 02:00:09.999747 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.19.194:0
--local_ip_for_outbound_sockets=127.16.19.194
--webserver_interface=127.16.19.194
--webserver_port=0
--tserver_master_addrs=127.16.19.254:32875
--builtin_ntp_servers=127.16.19.212:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250626 02:00:10.009716 22395 heartbeater.cc:344] Connected to a master server at 127.16.19.254:32875
I20250626 02:00:10.010275 22395 heartbeater.cc:461] Registering TS with master...
I20250626 02:00:10.011633 22395 heartbeater.cc:507] Master 127.16.19.254:32875 requested a full tablet report, sending...
I20250626 02:00:10.014703 22206 ts_manager.cc:194] Registered new tserver with Master: 8c554ad3849d45fda4e11514db7b2ae4 (127.16.19.193:43387)
I20250626 02:00:10.016604 22206 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.19.193:44949
W20250626 02:00:10.309803 22399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 02:00:10.310334 22399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 02:00:10.310875 22399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 02:00:10.343727 22399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 02:00:10.344657 22399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.19.194
I20250626 02:00:10.380033 22399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.19.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.19.194
--webserver_port=0
--tserver_master_addrs=127.16.19.254:32875
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.19.194
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 02:00:10.381392 22399 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 02:00:10.383121 22399 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:00:10.399602 22408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:00:11.020772 22395 heartbeater.cc:499] Master 127.16.19.254:32875 was elected leader, sending a full tablet report...
W20250626 02:00:10.399681 22405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:00:10.402415 22399 server_base.cc:1048] running on GCE node
W20250626 02:00:10.399945 22406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:00:11.572992 22399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 02:00:11.575229 22399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 02:00:11.576701 22399 hybrid_clock.cc:648] HybridClock initialized: now 1750903211576662 us; error 50 us; skew 500 ppm
I20250626 02:00:11.577504 22399 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 02:00:11.583561 22399 webserver.cc:469] Webserver started at http://127.16.19.194:39303/ using document root <none> and password file <none>
I20250626 02:00:11.584498 22399 fs_manager.cc:362] Metadata directory not provided
I20250626 02:00:11.584756 22399 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:00:11.585222 22399 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:00:11.591342 22399 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-1/data/instance:
uuid: "129a1a235aea4afd8bb9a66c64645879"
format_stamp: "Formatted at 2025-06-26 02:00:11 on dist-test-slave-h4rq"
I20250626 02:00:11.592447 22399 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-1/wal/instance:
uuid: "129a1a235aea4afd8bb9a66c64645879"
format_stamp: "Formatted at 2025-06-26 02:00:11 on dist-test-slave-h4rq"
I20250626 02:00:11.600318 22399 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20250626 02:00:11.606396 22415 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:00:11.607437 22399 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250626 02:00:11.607769 22399 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-1/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-1/wal
uuid: "129a1a235aea4afd8bb9a66c64645879"
format_stamp: "Formatted at 2025-06-26 02:00:11 on dist-test-slave-h4rq"
I20250626 02:00:11.608206 22399 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:00:11.665920 22399 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:00:11.667343 22399 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 02:00:11.667742 22399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:00:11.670202 22399 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:00:11.674255 22399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:00:11.674461 22399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:00:11.674665 22399 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:00:11.674809 22399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:00:11.807375 22399 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.194:34401
I20250626 02:00:11.807543 22527 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.194:34401 every 8 connection(s)
I20250626 02:00:11.810076 22399 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-1/data/info.pb
I20250626 02:00:11.811555 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 22399
I20250626 02:00:11.812162 16463 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-1/wal/instance
I20250626 02:00:11.819736 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.19.195:0
--local_ip_for_outbound_sockets=127.16.19.195
--webserver_interface=127.16.19.195
--webserver_port=0
--tserver_master_addrs=127.16.19.254:32875
--builtin_ntp_servers=127.16.19.212:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250626 02:00:11.832839 22528 heartbeater.cc:344] Connected to a master server at 127.16.19.254:32875
I20250626 02:00:11.833283 22528 heartbeater.cc:461] Registering TS with master...
I20250626 02:00:11.834334 22528 heartbeater.cc:507] Master 127.16.19.254:32875 requested a full tablet report, sending...
I20250626 02:00:11.836589 22206 ts_manager.cc:194] Registered new tserver with Master: 129a1a235aea4afd8bb9a66c64645879 (127.16.19.194:34401)
I20250626 02:00:11.837945 22206 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.19.194:54555
W20250626 02:00:12.149786 22532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 02:00:12.150290 22532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 02:00:12.150801 22532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 02:00:12.182374 22532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 02:00:12.183274 22532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.19.195
I20250626 02:00:12.218406 22532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.19.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.19.195
--webserver_port=0
--tserver_master_addrs=127.16.19.254:32875
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.19.195
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 02:00:12.219836 22532 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 02:00:12.221487 22532 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:00:12.238952 22539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:00:12.841221 22528 heartbeater.cc:499] Master 127.16.19.254:32875 was elected leader, sending a full tablet report...
W20250626 02:00:12.239493 22541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:00:12.240087 22538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:00:12.242154 22532 server_base.cc:1048] running on GCE node
I20250626 02:00:13.399843 22532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 02:00:13.402652 22532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 02:00:13.404088 22532 hybrid_clock.cc:648] HybridClock initialized: now 1750903213404049 us; error 60 us; skew 500 ppm
I20250626 02:00:13.405004 22532 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 02:00:13.415623 22532 webserver.cc:469] Webserver started at http://127.16.19.195:41993/ using document root <none> and password file <none>
I20250626 02:00:13.416595 22532 fs_manager.cc:362] Metadata directory not provided
I20250626 02:00:13.416870 22532 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:00:13.417343 22532 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:00:13.421790 22532 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-2/data/instance:
uuid: "6bb7e1dd53b344f49e42d1587eb3061d"
format_stamp: "Formatted at 2025-06-26 02:00:13 on dist-test-slave-h4rq"
I20250626 02:00:13.422984 22532 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-2/wal/instance:
uuid: "6bb7e1dd53b344f49e42d1587eb3061d"
format_stamp: "Formatted at 2025-06-26 02:00:13 on dist-test-slave-h4rq"
I20250626 02:00:13.430140 22532 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20250626 02:00:13.436041 22548 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:00:13.437199 22532 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20250626 02:00:13.437521 22532 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-2/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-2/wal
uuid: "6bb7e1dd53b344f49e42d1587eb3061d"
format_stamp: "Formatted at 2025-06-26 02:00:13 on dist-test-slave-h4rq"
I20250626 02:00:13.437873 22532 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:00:13.490245 22532 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:00:13.491680 22532 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 02:00:13.492121 22532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:00:13.494630 22532 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:00:13.498879 22532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 02:00:13.499090 22532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 02:00:13.499341 22532 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 02:00:13.499509 22532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 02:00:13.637213 22532 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.195:46681
I20250626 02:00:13.637326 22660 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.195:46681 every 8 connection(s)
I20250626 02:00:13.639832 22532 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-2/data/info.pb
I20250626 02:00:13.650513 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 22532
I20250626 02:00:13.651038 16463 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-2/wal/instance
I20250626 02:00:13.661934 22661 heartbeater.cc:344] Connected to a master server at 127.16.19.254:32875
I20250626 02:00:13.662338 22661 heartbeater.cc:461] Registering TS with master...
I20250626 02:00:13.663290 22661 heartbeater.cc:507] Master 127.16.19.254:32875 requested a full tablet report, sending...
I20250626 02:00:13.665292 22206 ts_manager.cc:194] Registered new tserver with Master: 6bb7e1dd53b344f49e42d1587eb3061d (127.16.19.195:46681)
I20250626 02:00:13.666513 22206 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.19.195:60027
I20250626 02:00:13.671371 16463 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 02:00:13.701202 16463 test_util.cc:276] Using random seed: -334055366
I20250626 02:00:13.740185 22206 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58722:
name: "pre_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 02:00:13.742602 22206 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table pre_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 02:00:13.792160 22330 tablet_service.cc:1468] Processing CreateTablet for tablet 42a75e8727ba4c2b95126d27808996a5 (DEFAULT_TABLE table=pre_rebuild [id=92d8b8710b384feb92ded1d588f7b841]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 02:00:13.792115 22596 tablet_service.cc:1468] Processing CreateTablet for tablet 42a75e8727ba4c2b95126d27808996a5 (DEFAULT_TABLE table=pre_rebuild [id=92d8b8710b384feb92ded1d588f7b841]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 02:00:13.794076 22596 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42a75e8727ba4c2b95126d27808996a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:00:13.794047 22330 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42a75e8727ba4c2b95126d27808996a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:00:13.801210 22463 tablet_service.cc:1468] Processing CreateTablet for tablet 42a75e8727ba4c2b95126d27808996a5 (DEFAULT_TABLE table=pre_rebuild [id=92d8b8710b384feb92ded1d588f7b841]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 02:00:13.802713 22463 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42a75e8727ba4c2b95126d27808996a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:00:13.827912 22686 tablet_bootstrap.cc:492] T 42a75e8727ba4c2b95126d27808996a5 P 6bb7e1dd53b344f49e42d1587eb3061d: Bootstrap starting.
I20250626 02:00:13.838078 22685 tablet_bootstrap.cc:492] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4: Bootstrap starting.
I20250626 02:00:13.840242 22686 tablet_bootstrap.cc:654] T 42a75e8727ba4c2b95126d27808996a5 P 6bb7e1dd53b344f49e42d1587eb3061d: Neither blocks nor log segments found. Creating new log.
I20250626 02:00:13.842849 22686 log.cc:826] T 42a75e8727ba4c2b95126d27808996a5 P 6bb7e1dd53b344f49e42d1587eb3061d: Log is configured to *not* fsync() on all Append() calls
I20250626 02:00:13.845584 22687 tablet_bootstrap.cc:492] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879: Bootstrap starting.
I20250626 02:00:13.846223 22685 tablet_bootstrap.cc:654] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4: Neither blocks nor log segments found. Creating new log.
I20250626 02:00:13.853540 22687 tablet_bootstrap.cc:654] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879: Neither blocks nor log segments found. Creating new log.
I20250626 02:00:13.853546 22685 log.cc:826] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4: Log is configured to *not* fsync() on all Append() calls
I20250626 02:00:13.856256 22687 log.cc:826] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879: Log is configured to *not* fsync() on all Append() calls
I20250626 02:00:13.861043 22686 tablet_bootstrap.cc:492] T 42a75e8727ba4c2b95126d27808996a5 P 6bb7e1dd53b344f49e42d1587eb3061d: No bootstrap required, opened a new log
I20250626 02:00:13.861645 22686 ts_tablet_manager.cc:1397] T 42a75e8727ba4c2b95126d27808996a5 P 6bb7e1dd53b344f49e42d1587eb3061d: Time spent bootstrapping tablet: real 0.034s user 0.019s sys 0.010s
I20250626 02:00:13.862327 22685 tablet_bootstrap.cc:492] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4: No bootstrap required, opened a new log
I20250626 02:00:13.862696 22687 tablet_bootstrap.cc:492] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879: No bootstrap required, opened a new log
I20250626 02:00:13.863008 22685 ts_tablet_manager.cc:1397] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.013s
I20250626 02:00:13.863211 22687 ts_tablet_manager.cc:1397] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.009s
I20250626 02:00:13.882438 22687 raft_consensus.cc:357] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c554ad3849d45fda4e11514db7b2ae4" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 43387 } } peers { permanent_uuid: "6bb7e1dd53b344f49e42d1587eb3061d" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 46681 } } peers { permanent_uuid: "129a1a235aea4afd8bb9a66c64645879" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 34401 } }
I20250626 02:00:13.883387 22687 raft_consensus.cc:383] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:00:13.883697 22687 raft_consensus.cc:738] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129a1a235aea4afd8bb9a66c64645879, State: Initialized, Role: FOLLOWER
I20250626 02:00:13.884511 22687 consensus_queue.cc:260] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c554ad3849d45fda4e11514db7b2ae4" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 43387 } } peers { permanent_uuid: "6bb7e1dd53b344f49e42d1587eb3061d" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 46681 } } peers { permanent_uuid: "129a1a235aea4afd8bb9a66c64645879" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 34401 } }
I20250626 02:00:13.889245 22686 raft_consensus.cc:357] T 42a75e8727ba4c2b95126d27808996a5 P 6bb7e1dd53b344f49e42d1587eb3061d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c554ad3849d45fda4e11514db7b2ae4" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 43387 } } peers { permanent_uuid: "6bb7e1dd53b344f49e42d1587eb3061d" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 46681 } } peers { permanent_uuid: "129a1a235aea4afd8bb9a66c64645879" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 34401 } }
I20250626 02:00:13.890275 22686 raft_consensus.cc:383] T 42a75e8727ba4c2b95126d27808996a5 P 6bb7e1dd53b344f49e42d1587eb3061d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:00:13.890014 22685 raft_consensus.cc:357] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c554ad3849d45fda4e11514db7b2ae4" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 43387 } } peers { permanent_uuid: "6bb7e1dd53b344f49e42d1587eb3061d" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 46681 } } peers { permanent_uuid: "129a1a235aea4afd8bb9a66c64645879" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 34401 } }
I20250626 02:00:13.890604 22686 raft_consensus.cc:738] T 42a75e8727ba4c2b95126d27808996a5 P 6bb7e1dd53b344f49e42d1587eb3061d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bb7e1dd53b344f49e42d1587eb3061d, State: Initialized, Role: FOLLOWER
I20250626 02:00:13.891003 22685 raft_consensus.cc:383] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:00:13.891340 22685 raft_consensus.cc:738] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c554ad3849d45fda4e11514db7b2ae4, State: Initialized, Role: FOLLOWER
I20250626 02:00:13.891602 22686 consensus_queue.cc:260] T 42a75e8727ba4c2b95126d27808996a5 P 6bb7e1dd53b344f49e42d1587eb3061d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c554ad3849d45fda4e11514db7b2ae4" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 43387 } } peers { permanent_uuid: "6bb7e1dd53b344f49e42d1587eb3061d" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 46681 } } peers { permanent_uuid: "129a1a235aea4afd8bb9a66c64645879" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 34401 } }
I20250626 02:00:13.892823 22687 ts_tablet_manager.cc:1428] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879: Time spent starting tablet: real 0.029s user 0.027s sys 0.004s
I20250626 02:00:13.892271 22685 consensus_queue.cc:260] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c554ad3849d45fda4e11514db7b2ae4" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 43387 } } peers { permanent_uuid: "6bb7e1dd53b344f49e42d1587eb3061d" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 46681 } } peers { permanent_uuid: "129a1a235aea4afd8bb9a66c64645879" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 34401 } }
I20250626 02:00:13.897881 22685 ts_tablet_manager.cc:1428] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4: Time spent starting tablet: real 0.035s user 0.033s sys 0.000s
I20250626 02:00:13.900223 22661 heartbeater.cc:499] Master 127.16.19.254:32875 was elected leader, sending a full tablet report...
I20250626 02:00:13.901539 22686 ts_tablet_manager.cc:1428] T 42a75e8727ba4c2b95126d27808996a5 P 6bb7e1dd53b344f49e42d1587eb3061d: Time spent starting tablet: real 0.040s user 0.022s sys 0.016s
W20250626 02:00:13.996428 22396 tablet.cc:2378] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250626 02:00:14.067391 22529 tablet.cc:2378] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 02:00:14.134325 22691 raft_consensus.cc:491] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:00:14.134840 22691 raft_consensus.cc:513] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c554ad3849d45fda4e11514db7b2ae4" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 43387 } } peers { permanent_uuid: "6bb7e1dd53b344f49e42d1587eb3061d" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 46681 } } peers { permanent_uuid: "129a1a235aea4afd8bb9a66c64645879" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 34401 } }
I20250626 02:00:14.137336 22691 leader_election.cc:290] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c554ad3849d45fda4e11514db7b2ae4 (127.16.19.193:43387), 6bb7e1dd53b344f49e42d1587eb3061d (127.16.19.195:46681)
W20250626 02:00:14.144737 22662 tablet.cc:2378] T 42a75e8727ba4c2b95126d27808996a5 P 6bb7e1dd53b344f49e42d1587eb3061d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 02:00:14.149518 22616 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "42a75e8727ba4c2b95126d27808996a5" candidate_uuid: "129a1a235aea4afd8bb9a66c64645879" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bb7e1dd53b344f49e42d1587eb3061d" is_pre_election: true
I20250626 02:00:14.149518 22350 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "42a75e8727ba4c2b95126d27808996a5" candidate_uuid: "129a1a235aea4afd8bb9a66c64645879" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c554ad3849d45fda4e11514db7b2ae4" is_pre_election: true
I20250626 02:00:14.150213 22350 raft_consensus.cc:2466] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 129a1a235aea4afd8bb9a66c64645879 in term 0.
I20250626 02:00:14.150220 22616 raft_consensus.cc:2466] T 42a75e8727ba4c2b95126d27808996a5 P 6bb7e1dd53b344f49e42d1587eb3061d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 129a1a235aea4afd8bb9a66c64645879 in term 0.
I20250626 02:00:14.151338 22417 leader_election.cc:304] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 129a1a235aea4afd8bb9a66c64645879, 6bb7e1dd53b344f49e42d1587eb3061d; no voters:
I20250626 02:00:14.152055 22691 raft_consensus.cc:2802] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:00:14.152313 22691 raft_consensus.cc:491] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:00:14.152568 22691 raft_consensus.cc:3058] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:00:14.156919 22691 raft_consensus.cc:513] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c554ad3849d45fda4e11514db7b2ae4" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 43387 } } peers { permanent_uuid: "6bb7e1dd53b344f49e42d1587eb3061d" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 46681 } } peers { permanent_uuid: "129a1a235aea4afd8bb9a66c64645879" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 34401 } }
I20250626 02:00:14.158429 22691 leader_election.cc:290] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879 [CANDIDATE]: Term 1 election: Requested vote from peers 8c554ad3849d45fda4e11514db7b2ae4 (127.16.19.193:43387), 6bb7e1dd53b344f49e42d1587eb3061d (127.16.19.195:46681)
I20250626 02:00:14.159186 22350 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "42a75e8727ba4c2b95126d27808996a5" candidate_uuid: "129a1a235aea4afd8bb9a66c64645879" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c554ad3849d45fda4e11514db7b2ae4"
I20250626 02:00:14.159451 22616 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "42a75e8727ba4c2b95126d27808996a5" candidate_uuid: "129a1a235aea4afd8bb9a66c64645879" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bb7e1dd53b344f49e42d1587eb3061d"
I20250626 02:00:14.159749 22350 raft_consensus.cc:3058] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:00:14.160002 22616 raft_consensus.cc:3058] T 42a75e8727ba4c2b95126d27808996a5 P 6bb7e1dd53b344f49e42d1587eb3061d [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:00:14.166842 22616 raft_consensus.cc:2466] T 42a75e8727ba4c2b95126d27808996a5 P 6bb7e1dd53b344f49e42d1587eb3061d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 129a1a235aea4afd8bb9a66c64645879 in term 1.
I20250626 02:00:14.166842 22350 raft_consensus.cc:2466] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 129a1a235aea4afd8bb9a66c64645879 in term 1.
I20250626 02:00:14.167905 22417 leader_election.cc:304] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 129a1a235aea4afd8bb9a66c64645879, 6bb7e1dd53b344f49e42d1587eb3061d; no voters:
I20250626 02:00:14.168612 22691 raft_consensus.cc:2802] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:00:14.170222 22691 raft_consensus.cc:695] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879 [term 1 LEADER]: Becoming Leader. State: Replica: 129a1a235aea4afd8bb9a66c64645879, State: Running, Role: LEADER
I20250626 02:00:14.171074 22691 consensus_queue.cc:237] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c554ad3849d45fda4e11514db7b2ae4" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 43387 } } peers { permanent_uuid: "6bb7e1dd53b344f49e42d1587eb3061d" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 46681 } } peers { permanent_uuid: "129a1a235aea4afd8bb9a66c64645879" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 34401 } }
I20250626 02:00:14.181643 22204 catalog_manager.cc:5582] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879 reported cstate change: term changed from 0 to 1, leader changed from <none> to 129a1a235aea4afd8bb9a66c64645879 (127.16.19.194). New cstate: current_term: 1 leader_uuid: "129a1a235aea4afd8bb9a66c64645879" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c554ad3849d45fda4e11514db7b2ae4" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 43387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bb7e1dd53b344f49e42d1587eb3061d" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 46681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "129a1a235aea4afd8bb9a66c64645879" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 34401 } health_report { overall_health: HEALTHY } } }
I20250626 02:00:14.433097 22616 raft_consensus.cc:1273] T 42a75e8727ba4c2b95126d27808996a5 P 6bb7e1dd53b344f49e42d1587eb3061d [term 1 FOLLOWER]: Refusing update from remote peer 129a1a235aea4afd8bb9a66c64645879: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:00:14.433158 22350 raft_consensus.cc:1273] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4 [term 1 FOLLOWER]: Refusing update from remote peer 129a1a235aea4afd8bb9a66c64645879: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:00:14.434852 22691 consensus_queue.cc:1035] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c554ad3849d45fda4e11514db7b2ae4" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 43387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:00:14.435510 22696 consensus_queue.cc:1035] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bb7e1dd53b344f49e42d1587eb3061d" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 46681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:00:14.473580 22705 mvcc.cc:204] Tried to move back new op lower bound from 7171699566301425664 to 7171699565252476928. Current Snapshot: MvccSnapshot[applied={T|T < 7171699566301425664}]
I20250626 02:00:19.818337 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 22173
W20250626 02:00:20.215485 22747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 02:00:20.216558 22747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 02:00:20.250662 22747 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250626 02:00:20.566212 22661 heartbeater.cc:646] Failed to heartbeat to 127.16.19.254:32875 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.19.254:32875: connect: Connection refused (error 111)
W20250626 02:00:20.643471 22395 heartbeater.cc:646] Failed to heartbeat to 127.16.19.254:32875 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.19.254:32875: connect: Connection refused (error 111)
W20250626 02:00:20.647817 22528 heartbeater.cc:646] Failed to heartbeat to 127.16.19.254:32875 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.19.254:32875: connect: Connection refused (error 111)
W20250626 02:00:21.540445 22747 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.243s user 0.475s sys 0.764s
W20250626 02:00:21.540849 22747 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.244s user 0.479s sys 0.764s
I20250626 02:00:21.701488 22747 minidump.cc:252] Setting minidump size limit to 20M
I20250626 02:00:21.703913 22747 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 02:00:21.705399 22747 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:00:21.718309 22780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:00:21.722116 22783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:00:21.720726 22781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:00:21.893334 22747 server_base.cc:1048] running on GCE node
I20250626 02:00:21.894639 22747 hybrid_clock.cc:584] initializing the hybrid clock with 'system' time source
I20250626 02:00:21.895200 22747 hybrid_clock.cc:648] HybridClock initialized: now 1750903221895175 us; error 28901 us; skew 500 ppm
I20250626 02:00:21.896097 22747 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 02:00:21.901494 22747 webserver.cc:469] Webserver started at http://0.0.0.0:36723/ using document root <none> and password file <none>
I20250626 02:00:21.902601 22747 fs_manager.cc:362] Metadata directory not provided
I20250626 02:00:21.902899 22747 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:00:21.903425 22747 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 02:00:21.909952 22747 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/master-0/data/instance:
uuid: "d1c8512529a1431a84bc4888edf347bf"
format_stamp: "Formatted at 2025-06-26 02:00:21 on dist-test-slave-h4rq"
I20250626 02:00:21.911501 22747 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/master-0/wal/instance:
uuid: "d1c8512529a1431a84bc4888edf347bf"
format_stamp: "Formatted at 2025-06-26 02:00:21 on dist-test-slave-h4rq"
I20250626 02:00:21.919817 22747 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.004s
I20250626 02:00:21.926375 22788 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:00:21.927414 22747 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.004s
I20250626 02:00:21.927843 22747 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/master-0/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/master-0/wal
uuid: "d1c8512529a1431a84bc4888edf347bf"
format_stamp: "Formatted at 2025-06-26 02:00:21 on dist-test-slave-h4rq"
I20250626 02:00:21.928285 22747 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:00:22.025404 22747 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:00:22.027328 22747 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 02:00:22.027911 22747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:00:22.034111 22747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:00:22.056530 22747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf: Bootstrap starting.
I20250626 02:00:22.063731 22747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf: Neither blocks nor log segments found. Creating new log.
I20250626 02:00:22.066021 22747 log.cc:826] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf: Log is configured to *not* fsync() on all Append() calls
I20250626 02:00:22.071161 22747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf: No bootstrap required, opened a new log
I20250626 02:00:22.096964 22747 raft_consensus.cc:357] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c8512529a1431a84bc4888edf347bf" member_type: VOTER }
I20250626 02:00:22.097625 22747 raft_consensus.cc:383] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:00:22.097909 22747 raft_consensus.cc:738] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c8512529a1431a84bc4888edf347bf, State: Initialized, Role: FOLLOWER
I20250626 02:00:22.098737 22747 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c8512529a1431a84bc4888edf347bf" member_type: VOTER }
I20250626 02:00:22.099350 22747 raft_consensus.cc:397] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 02:00:22.099685 22747 raft_consensus.cc:491] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 02:00:22.100075 22747 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:00:22.107496 22747 raft_consensus.cc:513] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c8512529a1431a84bc4888edf347bf" member_type: VOTER }
I20250626 02:00:22.108541 22747 leader_election.cc:304] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1c8512529a1431a84bc4888edf347bf; no voters:
I20250626 02:00:22.111068 22747 leader_election.cc:290] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 02:00:22.111357 22795 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:00:22.115453 22795 raft_consensus.cc:695] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf [term 1 LEADER]: Becoming Leader. State: Replica: d1c8512529a1431a84bc4888edf347bf, State: Running, Role: LEADER
I20250626 02:00:22.116308 22795 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c8512529a1431a84bc4888edf347bf" member_type: VOTER }
I20250626 02:00:22.123463 22797 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1c8512529a1431a84bc4888edf347bf. Latest consensus state: current_term: 1 leader_uuid: "d1c8512529a1431a84bc4888edf347bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c8512529a1431a84bc4888edf347bf" member_type: VOTER } }
I20250626 02:00:22.124161 22797 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf [sys.catalog]: This master's current role is: LEADER
I20250626 02:00:22.123733 22796 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1c8512529a1431a84bc4888edf347bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c8512529a1431a84bc4888edf347bf" member_type: VOTER } }
I20250626 02:00:22.124634 22796 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf [sys.catalog]: This master's current role is: LEADER
I20250626 02:00:22.140074 22747 tablet_replica.cc:331] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf: stopping tablet replica
I20250626 02:00:22.140726 22747 raft_consensus.cc:2241] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf [term 1 LEADER]: Raft consensus shutting down.
I20250626 02:00:22.141216 22747 raft_consensus.cc:2270] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:00:22.143589 22747 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250626 02:00:22.144011 22747 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250626 02:00:22.173159 22747 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
I20250626 02:00:23.203006 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 22266
I20250626 02:00:23.239441 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 22399
I20250626 02:00:23.282502 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 22532
I20250626 02:00:23.318821 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.19.254:32875
--webserver_interface=127.16.19.254
--webserver_port=34315
--builtin_ntp_servers=127.16.19.212:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.19.254:32875 with env {}
W20250626 02:00:23.629721 22805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 02:00:23.630283 22805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 02:00:23.630704 22805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 02:00:23.662307 22805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 02:00:23.662602 22805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 02:00:23.662808 22805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 02:00:23.663003 22805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 02:00:23.698987 22805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.19.254:32875
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.19.254:32875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.19.254
--webserver_port=34315
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 02:00:23.700340 22805 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 02:00:23.701957 22805 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:00:23.717926 22812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:00:23.717955 22814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:00:23.718974 22811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:00:24.912171 22813 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 02:00:24.912303 22805 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 02:00:24.917724 22805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 02:00:24.920924 22805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 02:00:24.922533 22805 hybrid_clock.cc:648] HybridClock initialized: now 1750903224922481 us; error 62 us; skew 500 ppm
I20250626 02:00:24.923538 22805 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 02:00:24.930590 22805 webserver.cc:469] Webserver started at http://127.16.19.254:34315/ using document root <none> and password file <none>
I20250626 02:00:24.931712 22805 fs_manager.cc:362] Metadata directory not provided
I20250626 02:00:24.931949 22805 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:00:24.940166 22805 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20250626 02:00:24.944926 22821 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:00:24.946049 22805 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20250626 02:00:24.946393 22805 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/master-0/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/master-0/wal
uuid: "d1c8512529a1431a84bc4888edf347bf"
format_stamp: "Formatted at 2025-06-26 02:00:21 on dist-test-slave-h4rq"
I20250626 02:00:24.948341 22805 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:00:24.999444 22805 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:00:25.001005 22805 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 02:00:25.001477 22805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:00:25.072741 22805 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.254:32875
I20250626 02:00:25.072793 22872 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.254:32875 every 8 connection(s)
I20250626 02:00:25.075564 22805 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/master-0/data/info.pb
I20250626 02:00:25.085091 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 22805
I20250626 02:00:25.086329 22873 sys_catalog.cc:263] Verifying existing consensus state
I20250626 02:00:25.087283 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.19.193:43387
--local_ip_for_outbound_sockets=127.16.19.193
--tserver_master_addrs=127.16.19.254:32875
--webserver_port=34435
--webserver_interface=127.16.19.193
--builtin_ntp_servers=127.16.19.212:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250626 02:00:25.104171 22873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf: Bootstrap starting.
I20250626 02:00:25.116750 22873 log.cc:826] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf: Log is configured to *not* fsync() on all Append() calls
I20250626 02:00:25.129117 22873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:00:25.129926 22873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf: Bootstrap complete.
I20250626 02:00:25.150873 22873 raft_consensus.cc:357] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c8512529a1431a84bc4888edf347bf" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 32875 } }
I20250626 02:00:25.151629 22873 raft_consensus.cc:738] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c8512529a1431a84bc4888edf347bf, State: Initialized, Role: FOLLOWER
I20250626 02:00:25.152348 22873 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c8512529a1431a84bc4888edf347bf" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 32875 } }
I20250626 02:00:25.152899 22873 raft_consensus.cc:397] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 02:00:25.153172 22873 raft_consensus.cc:491] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 02:00:25.153466 22873 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:00:25.157634 22873 raft_consensus.cc:513] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c8512529a1431a84bc4888edf347bf" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 32875 } }
I20250626 02:00:25.158313 22873 leader_election.cc:304] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1c8512529a1431a84bc4888edf347bf; no voters:
I20250626 02:00:25.160588 22873 leader_election.cc:290] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf [CANDIDATE]: Term 2 election: Requested vote from peers
I20250626 02:00:25.160955 22877 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf [term 2 FOLLOWER]: Leader election won for term 2
I20250626 02:00:25.164336 22877 raft_consensus.cc:695] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf [term 2 LEADER]: Becoming Leader. State: Replica: d1c8512529a1431a84bc4888edf347bf, State: Running, Role: LEADER
I20250626 02:00:25.165231 22877 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c8512529a1431a84bc4888edf347bf" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 32875 } }
I20250626 02:00:25.166041 22873 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf [sys.catalog]: configured and running, proceeding with master startup.
I20250626 02:00:25.175115 22879 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1c8512529a1431a84bc4888edf347bf. Latest consensus state: current_term: 2 leader_uuid: "d1c8512529a1431a84bc4888edf347bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c8512529a1431a84bc4888edf347bf" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 32875 } } }
I20250626 02:00:25.176656 22878 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d1c8512529a1431a84bc4888edf347bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c8512529a1431a84bc4888edf347bf" member_type: VOTER last_known_addr { host: "127.16.19.254" port: 32875 } } }
I20250626 02:00:25.177302 22878 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf [sys.catalog]: This master's current role is: LEADER
I20250626 02:00:25.180147 22879 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf [sys.catalog]: This master's current role is: LEADER
I20250626 02:00:25.184068 22883 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 02:00:25.201951 22883 catalog_manager.cc:671] Loaded metadata for table pre_rebuild [id=8bbb77ce1dde43b9ae2df9d0b19acc57]
I20250626 02:00:25.209291 22883 tablet_loader.cc:96] loaded metadata for tablet 42a75e8727ba4c2b95126d27808996a5 (table pre_rebuild [id=8bbb77ce1dde43b9ae2df9d0b19acc57])
I20250626 02:00:25.211685 22883 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 02:00:25.242128 22883 catalog_manager.cc:1349] Generated new cluster ID: 1e0d62485cca4e7f9b08bbc38e0c7759
I20250626 02:00:25.242635 22883 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 02:00:25.262742 22883 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 02:00:25.264876 22883 catalog_manager.cc:1506] Loading token signing keys...
I20250626 02:00:25.289866 22883 catalog_manager.cc:5955] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf: Generated new TSK 0
I20250626 02:00:25.290973 22883 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250626 02:00:25.452168 22875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 02:00:25.452750 22875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 02:00:25.453336 22875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 02:00:25.486099 22875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 02:00:25.487020 22875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.19.193
I20250626 02:00:25.523525 22875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.19.193:43387
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.19.193
--webserver_port=34435
--tserver_master_addrs=127.16.19.254:32875
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.19.193
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 02:00:25.524948 22875 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 02:00:25.526618 22875 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:00:25.544689 22902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:00:25.546834 22904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:00:25.547832 22875 server_base.cc:1048] running on GCE node
W20250626 02:00:25.547363 22901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:00:26.723229 22875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 02:00:26.726224 22875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 02:00:26.727685 22875 hybrid_clock.cc:648] HybridClock initialized: now 1750903226727637 us; error 60 us; skew 500 ppm
I20250626 02:00:26.728775 22875 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 02:00:26.739780 22875 webserver.cc:469] Webserver started at http://127.16.19.193:34435/ using document root <none> and password file <none>
I20250626 02:00:26.741076 22875 fs_manager.cc:362] Metadata directory not provided
I20250626 02:00:26.741367 22875 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:00:26.752224 22875 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.006s sys 0.000s
I20250626 02:00:26.758098 22911 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:00:26.759305 22875 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250626 02:00:26.759708 22875 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-0/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-0/wal
uuid: "8c554ad3849d45fda4e11514db7b2ae4"
format_stamp: "Formatted at 2025-06-26 02:00:09 on dist-test-slave-h4rq"
I20250626 02:00:26.762105 22875 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:00:26.817890 22875 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:00:26.819419 22875 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 02:00:26.819908 22875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:00:26.823130 22875 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:00:26.829907 22918 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250626 02:00:26.841815 22875 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250626 02:00:26.842109 22875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.000s sys 0.002s
I20250626 02:00:26.842406 22875 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250626 02:00:26.847517 22875 ts_tablet_manager.cc:610] Registered 1 tablets
I20250626 02:00:26.847739 22875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.002s sys 0.001s
I20250626 02:00:26.848284 22918 tablet_bootstrap.cc:492] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4: Bootstrap starting.
I20250626 02:00:27.055720 22875 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.193:43387
I20250626 02:00:27.055888 23024 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.193:43387 every 8 connection(s)
I20250626 02:00:27.058627 22875 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-0/data/info.pb
I20250626 02:00:27.061165 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 22875
I20250626 02:00:27.063890 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.19.194:34401
--local_ip_for_outbound_sockets=127.16.19.194
--tserver_master_addrs=127.16.19.254:32875
--webserver_port=39303
--webserver_interface=127.16.19.194
--builtin_ntp_servers=127.16.19.212:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250626 02:00:27.117802 23025 heartbeater.cc:344] Connected to a master server at 127.16.19.254:32875
I20250626 02:00:27.118328 23025 heartbeater.cc:461] Registering TS with master...
I20250626 02:00:27.119550 23025 heartbeater.cc:507] Master 127.16.19.254:32875 requested a full tablet report, sending...
I20250626 02:00:27.124650 22838 ts_manager.cc:194] Registered new tserver with Master: 8c554ad3849d45fda4e11514db7b2ae4 (127.16.19.193:43387)
I20250626 02:00:27.131386 22838 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.19.193:59989
I20250626 02:00:27.183362 22918 log.cc:826] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4: Log is configured to *not* fsync() on all Append() calls
W20250626 02:00:27.445472 23026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 02:00:27.446025 23026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 02:00:27.446530 23026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 02:00:27.478825 23026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 02:00:27.479719 23026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.19.194
I20250626 02:00:27.515348 23026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.19.194:34401
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.19.194
--webserver_port=39303
--tserver_master_addrs=127.16.19.254:32875
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.19.194
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 02:00:27.516752 23026 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 02:00:27.518424 23026 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:00:27.535647 23036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:00:28.135447 23025 heartbeater.cc:499] Master 127.16.19.254:32875 was elected leader, sending a full tablet report...
W20250626 02:00:28.938586 23035 debug-util.cc:398] Leaking SignalData structure 0x7b0800000a80 after lost signal to thread 23026
W20250626 02:00:29.377569 23026 thread.cc:640] OpenStack (cloud detector) Time spent creating pthread: real 1.843s user 0.680s sys 1.068s
W20250626 02:00:27.535663 23037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:00:29.378067 23026 thread.cc:606] OpenStack (cloud detector) Time spent starting thread: real 1.844s user 0.680s sys 1.068s
W20250626 02:00:29.380273 23039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 02:00:29.386087 23038 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1849 milliseconds
I20250626 02:00:29.386133 23026 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 02:00:29.387573 23026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 02:00:29.390164 23026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 02:00:29.391642 23026 hybrid_clock.cc:648] HybridClock initialized: now 1750903229391588 us; error 53 us; skew 500 ppm
I20250626 02:00:29.392746 23026 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 02:00:29.400509 23026 webserver.cc:469] Webserver started at http://127.16.19.194:39303/ using document root <none> and password file <none>
I20250626 02:00:29.401810 23026 fs_manager.cc:362] Metadata directory not provided
I20250626 02:00:29.402112 23026 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:00:29.412510 23026 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.001s
I20250626 02:00:29.418210 23047 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:00:29.419310 23026 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250626 02:00:29.419615 23026 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-1/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-1/wal
uuid: "129a1a235aea4afd8bb9a66c64645879"
format_stamp: "Formatted at 2025-06-26 02:00:11 on dist-test-slave-h4rq"
I20250626 02:00:29.421566 23026 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:00:29.472597 23026 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:00:29.474149 23026 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 02:00:29.474578 23026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:00:29.477249 23026 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:00:29.483078 23054 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250626 02:00:29.494235 23026 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250626 02:00:29.494500 23026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.001s sys 0.001s
I20250626 02:00:29.494804 23026 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250626 02:00:29.499835 23026 ts_tablet_manager.cc:610] Registered 1 tablets
I20250626 02:00:29.500043 23026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.002s sys 0.000s
I20250626 02:00:29.500524 23054 tablet_bootstrap.cc:492] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879: Bootstrap starting.
I20250626 02:00:29.702068 23026 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.194:34401
I20250626 02:00:29.702203 23160 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.194:34401 every 8 connection(s)
I20250626 02:00:29.704921 23026 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-1/data/info.pb
I20250626 02:00:29.708874 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 23026
I20250626 02:00:29.711921 16463 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
/tmp/dist-test-taskifDKOX/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.19.195:46681
--local_ip_for_outbound_sockets=127.16.19.195
--tserver_master_addrs=127.16.19.254:32875
--webserver_port=41993
--webserver_interface=127.16.19.195
--builtin_ntp_servers=127.16.19.212:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250626 02:00:29.786609 23161 heartbeater.cc:344] Connected to a master server at 127.16.19.254:32875
I20250626 02:00:29.787143 23161 heartbeater.cc:461] Registering TS with master...
I20250626 02:00:29.788532 23161 heartbeater.cc:507] Master 127.16.19.254:32875 requested a full tablet report, sending...
I20250626 02:00:29.793385 22838 ts_manager.cc:194] Registered new tserver with Master: 129a1a235aea4afd8bb9a66c64645879 (127.16.19.194:34401)
I20250626 02:00:29.796792 22838 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.19.194:34169
I20250626 02:00:29.804289 23054 log.cc:826] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879: Log is configured to *not* fsync() on all Append() calls
W20250626 02:00:30.229887 23165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 02:00:30.230566 23165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 02:00:30.231360 23165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 02:00:30.288097 23165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 02:00:30.289564 23165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.19.195
I20250626 02:00:30.351558 23165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.19.212:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.19.195:46681
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.19.195
--webserver_port=41993
--tserver_master_addrs=127.16.19.254:32875
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.19.195
--log_dir=/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6773
TSAN enabled
I20250626 02:00:30.353253 23165 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 02:00:30.355429 23165 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 02:00:30.376152 23172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:00:30.801745 23161 heartbeater.cc:499] Master 127.16.19.254:32875 was elected leader, sending a full tablet report...
I20250626 02:00:30.981223 22918 tablet_bootstrap.cc:492] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:00:30.982800 22918 tablet_bootstrap.cc:492] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4: Bootstrap complete.
I20250626 02:00:30.985597 22918 ts_tablet_manager.cc:1397] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4: Time spent bootstrapping tablet: real 4.138s user 3.863s sys 0.076s
I20250626 02:00:31.011298 22918 raft_consensus.cc:357] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c554ad3849d45fda4e11514db7b2ae4" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 43387 } } peers { permanent_uuid: "6bb7e1dd53b344f49e42d1587eb3061d" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 46681 } } peers { permanent_uuid: "129a1a235aea4afd8bb9a66c64645879" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 34401 } }
I20250626 02:00:31.015901 22918 raft_consensus.cc:738] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c554ad3849d45fda4e11514db7b2ae4, State: Initialized, Role: FOLLOWER
I20250626 02:00:31.017388 22918 consensus_queue.cc:260] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c554ad3849d45fda4e11514db7b2ae4" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 43387 } } peers { permanent_uuid: "6bb7e1dd53b344f49e42d1587eb3061d" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 46681 } } peers { permanent_uuid: "129a1a235aea4afd8bb9a66c64645879" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 34401 } }
I20250626 02:00:31.028268 22918 ts_tablet_manager.cc:1428] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4: Time spent starting tablet: real 0.042s user 0.028s sys 0.011s
W20250626 02:00:31.772691 23171 debug-util.cc:398] Leaking SignalData structure 0x7b0800000a80 after lost signal to thread 23165
W20250626 02:00:32.099020 23165 thread.cc:640] GCE (cloud detector) Time spent creating pthread: real 1.725s user 0.627s sys 1.079s
W20250626 02:00:32.103314 23165 thread.cc:606] GCE (cloud detector) Time spent starting thread: real 1.729s user 0.627s sys 1.079s
W20250626 02:00:30.377517 23173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:00:32.116855 23165 server_base.cc:1048] running on GCE node
W20250626 02:00:32.117797 23178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 02:00:32.119390 23165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 02:00:32.122366 23165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 02:00:32.123898 23165 hybrid_clock.cc:648] HybridClock initialized: now 1750903232123806 us; error 92 us; skew 500 ppm
I20250626 02:00:32.125142 23165 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 02:00:32.132145 23165 webserver.cc:469] Webserver started at http://127.16.19.195:41993/ using document root <none> and password file <none>
I20250626 02:00:32.133284 23165 fs_manager.cc:362] Metadata directory not provided
I20250626 02:00:32.133621 23165 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 02:00:32.141968 23165 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.002s sys 0.002s
I20250626 02:00:32.147329 23183 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 02:00:32.148404 23165 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.003s
I20250626 02:00:32.148744 23165 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-2/data,/tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-2/wal
uuid: "6bb7e1dd53b344f49e42d1587eb3061d"
format_stamp: "Formatted at 2025-06-26 02:00:13 on dist-test-slave-h4rq"
I20250626 02:00:32.150653 23165 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 02:00:32.215015 23165 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 02:00:32.216495 23165 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 02:00:32.216977 23165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 02:00:32.219628 23165 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 02:00:32.226219 23190 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250626 02:00:32.237247 23165 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250626 02:00:32.237522 23165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.002s sys 0.000s
I20250626 02:00:32.237793 23165 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250626 02:00:32.242554 23165 ts_tablet_manager.cc:610] Registered 1 tablets
I20250626 02:00:32.242758 23165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.003s sys 0.000s
I20250626 02:00:32.243243 23190 tablet_bootstrap.cc:492] T 42a75e8727ba4c2b95126d27808996a5 P 6bb7e1dd53b344f49e42d1587eb3061d: Bootstrap starting.
I20250626 02:00:32.393719 23054 tablet_bootstrap.cc:492] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:00:32.394775 23054 tablet_bootstrap.cc:492] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879: Bootstrap complete.
I20250626 02:00:32.396566 23054 ts_tablet_manager.cc:1397] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879: Time spent bootstrapping tablet: real 2.896s user 2.756s sys 0.100s
I20250626 02:00:32.407590 23266 raft_consensus.cc:491] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:00:32.408601 23266 raft_consensus.cc:513] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c554ad3849d45fda4e11514db7b2ae4" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 43387 } } peers { permanent_uuid: "6bb7e1dd53b344f49e42d1587eb3061d" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 46681 } } peers { permanent_uuid: "129a1a235aea4afd8bb9a66c64645879" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 34401 } }
I20250626 02:00:32.414799 23054 raft_consensus.cc:357] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c554ad3849d45fda4e11514db7b2ae4" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 43387 } } peers { permanent_uuid: "6bb7e1dd53b344f49e42d1587eb3061d" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 46681 } } peers { permanent_uuid: "129a1a235aea4afd8bb9a66c64645879" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 34401 } }
I20250626 02:00:32.417934 23054 raft_consensus.cc:738] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129a1a235aea4afd8bb9a66c64645879, State: Initialized, Role: FOLLOWER
W20250626 02:00:32.418047 22913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.19.195:46681: connect: Connection refused (error 111)
I20250626 02:00:32.419070 23054 consensus_queue.cc:260] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c554ad3849d45fda4e11514db7b2ae4" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 43387 } } peers { permanent_uuid: "6bb7e1dd53b344f49e42d1587eb3061d" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 46681 } } peers { permanent_uuid: "129a1a235aea4afd8bb9a66c64645879" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 34401 } }
I20250626 02:00:32.423171 23266 leader_election.cc:290] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bb7e1dd53b344f49e42d1587eb3061d (127.16.19.195:46681), 129a1a235aea4afd8bb9a66c64645879 (127.16.19.194:34401)
I20250626 02:00:32.428751 23054 ts_tablet_manager.cc:1428] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879: Time spent starting tablet: real 0.032s user 0.028s sys 0.000s
W20250626 02:00:32.433691 22913 leader_election.cc:336] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bb7e1dd53b344f49e42d1587eb3061d (127.16.19.195:46681): Network error: Client connection negotiation failed: client connection to 127.16.19.195:46681: connect: Connection refused (error 111)
I20250626 02:00:32.446345 23116 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "42a75e8727ba4c2b95126d27808996a5" candidate_uuid: "8c554ad3849d45fda4e11514db7b2ae4" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "129a1a235aea4afd8bb9a66c64645879" is_pre_election: true
I20250626 02:00:32.447268 23116 raft_consensus.cc:2466] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c554ad3849d45fda4e11514db7b2ae4 in term 1.
I20250626 02:00:32.449198 22915 leader_election.cc:304] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 129a1a235aea4afd8bb9a66c64645879, 8c554ad3849d45fda4e11514db7b2ae4; no voters: 6bb7e1dd53b344f49e42d1587eb3061d
I20250626 02:00:32.450245 23266 raft_consensus.cc:2802] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 02:00:32.450620 23266 raft_consensus.cc:491] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:00:32.450954 23266 raft_consensus.cc:3058] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4 [term 1 FOLLOWER]: Advancing to term 2
I20250626 02:00:32.460965 23266 raft_consensus.cc:513] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c554ad3849d45fda4e11514db7b2ae4" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 43387 } } peers { permanent_uuid: "6bb7e1dd53b344f49e42d1587eb3061d" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 46681 } } peers { permanent_uuid: "129a1a235aea4afd8bb9a66c64645879" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 34401 } }
I20250626 02:00:32.465557 23266 leader_election.cc:290] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4 [CANDIDATE]: Term 2 election: Requested vote from peers 6bb7e1dd53b344f49e42d1587eb3061d (127.16.19.195:46681), 129a1a235aea4afd8bb9a66c64645879 (127.16.19.194:34401)
I20250626 02:00:32.465835 23116 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "42a75e8727ba4c2b95126d27808996a5" candidate_uuid: "8c554ad3849d45fda4e11514db7b2ae4" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "129a1a235aea4afd8bb9a66c64645879"
I20250626 02:00:32.466408 23116 raft_consensus.cc:3058] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879 [term 1 FOLLOWER]: Advancing to term 2
W20250626 02:00:32.469817 22913 leader_election.cc:336] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6bb7e1dd53b344f49e42d1587eb3061d (127.16.19.195:46681): Network error: Client connection negotiation failed: client connection to 127.16.19.195:46681: connect: Connection refused (error 111)
I20250626 02:00:32.478650 23116 raft_consensus.cc:2466] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c554ad3849d45fda4e11514db7b2ae4 in term 2.
I20250626 02:00:32.480163 22915 leader_election.cc:304] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 129a1a235aea4afd8bb9a66c64645879, 8c554ad3849d45fda4e11514db7b2ae4; no voters: 6bb7e1dd53b344f49e42d1587eb3061d
I20250626 02:00:32.481010 23266 raft_consensus.cc:2802] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 02:00:32.483240 23266 raft_consensus.cc:695] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4 [term 2 LEADER]: Becoming Leader. State: Replica: 8c554ad3849d45fda4e11514db7b2ae4, State: Running, Role: LEADER
I20250626 02:00:32.484328 23266 consensus_queue.cc:237] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c554ad3849d45fda4e11514db7b2ae4" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 43387 } } peers { permanent_uuid: "6bb7e1dd53b344f49e42d1587eb3061d" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 46681 } } peers { permanent_uuid: "129a1a235aea4afd8bb9a66c64645879" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 34401 } }
I20250626 02:00:32.498091 23165 rpc_server.cc:307] RPC server started. Bound to: 127.16.19.195:46681
I20250626 02:00:32.498457 23304 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.19.195:46681 every 8 connection(s)
I20250626 02:00:32.500746 22838 catalog_manager.cc:5582] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4 reported cstate change: term changed from 0 to 2, leader changed from <none> to 8c554ad3849d45fda4e11514db7b2ae4 (127.16.19.193), VOTER 129a1a235aea4afd8bb9a66c64645879 (127.16.19.194) added, VOTER 6bb7e1dd53b344f49e42d1587eb3061d (127.16.19.195) added, VOTER 8c554ad3849d45fda4e11514db7b2ae4 (127.16.19.193) added. New cstate: current_term: 2 leader_uuid: "8c554ad3849d45fda4e11514db7b2ae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c554ad3849d45fda4e11514db7b2ae4" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 43387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bb7e1dd53b344f49e42d1587eb3061d" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 46681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "129a1a235aea4afd8bb9a66c64645879" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 34401 } health_report { overall_health: UNKNOWN } } }
I20250626 02:00:32.501893 23165 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-2/data/info.pb
I20250626 02:00:32.512275 16463 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu as pid 23165
I20250626 02:00:32.555299 23305 heartbeater.cc:344] Connected to a master server at 127.16.19.254:32875
I20250626 02:00:32.555862 23305 heartbeater.cc:461] Registering TS with master...
I20250626 02:00:32.557312 23305 heartbeater.cc:507] Master 127.16.19.254:32875 requested a full tablet report, sending...
I20250626 02:00:32.562496 22837 ts_manager.cc:194] Registered new tserver with Master: 6bb7e1dd53b344f49e42d1587eb3061d (127.16.19.195:46681)
I20250626 02:00:32.568123 16463 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 02:00:32.569342 22837 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.19.195:38919
I20250626 02:00:32.576009 23190 log.cc:826] T 42a75e8727ba4c2b95126d27808996a5 P 6bb7e1dd53b344f49e42d1587eb3061d: Log is configured to *not* fsync() on all Append() calls
I20250626 02:00:32.921633 23116 raft_consensus.cc:1273] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879 [term 2 FOLLOWER]: Refusing update from remote peer 8c554ad3849d45fda4e11514db7b2ae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250626 02:00:32.923337 23266 consensus_queue.cc:1035] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a1a235aea4afd8bb9a66c64645879" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 34401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
I20250626 02:00:33.018669 22980 consensus_queue.cc:237] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 2.206, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "8c554ad3849d45fda4e11514db7b2ae4" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 43387 } } peers { permanent_uuid: "129a1a235aea4afd8bb9a66c64645879" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 34401 } }
I20250626 02:00:33.026700 23115 raft_consensus.cc:1273] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879 [term 2 FOLLOWER]: Refusing update from remote peer 8c554ad3849d45fda4e11514db7b2ae4: Log matching property violated. Preceding OpId in replica: term: 2 index: 206. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
I20250626 02:00:33.028165 23266 consensus_queue.cc:1035] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a1a235aea4afd8bb9a66c64645879" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 34401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.000s
I20250626 02:00:33.035284 23266 raft_consensus.cc:2953] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4 [term 2 LEADER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 6bb7e1dd53b344f49e42d1587eb3061d (127.16.19.195) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "8c554ad3849d45fda4e11514db7b2ae4" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 43387 } } peers { permanent_uuid: "129a1a235aea4afd8bb9a66c64645879" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 34401 } } }
I20250626 02:00:33.038018 23115 raft_consensus.cc:2953] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879 [term 2 FOLLOWER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 6bb7e1dd53b344f49e42d1587eb3061d (127.16.19.195) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "8c554ad3849d45fda4e11514db7b2ae4" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 43387 } } peers { permanent_uuid: "129a1a235aea4afd8bb9a66c64645879" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 34401 } } }
I20250626 02:00:33.055069 22824 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 42a75e8727ba4c2b95126d27808996a5 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 02:00:33.063143 22837 catalog_manager.cc:5582] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4 reported cstate change: config changed from index -1 to 207, VOTER 6bb7e1dd53b344f49e42d1587eb3061d (127.16.19.195) evicted. New cstate: current_term: 2 leader_uuid: "8c554ad3849d45fda4e11514db7b2ae4" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "8c554ad3849d45fda4e11514db7b2ae4" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 43387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "129a1a235aea4afd8bb9a66c64645879" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 34401 } health_report { overall_health: HEALTHY } } }
I20250626 02:00:33.111832 22980 consensus_queue.cc:237] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 2.207, Last appended by leader: 205, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "8c554ad3849d45fda4e11514db7b2ae4" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 43387 } }
I20250626 02:00:33.114553 23266 raft_consensus.cc:2953] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4 [term 2 LEADER]: Committing config change with OpId 2.208: config changed from index 207 to 208, VOTER 129a1a235aea4afd8bb9a66c64645879 (127.16.19.194) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "8c554ad3849d45fda4e11514db7b2ae4" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 43387 } } }
I20250626 02:00:33.119041 23232 tablet_service.cc:1515] Processing DeleteTablet for tablet 42a75e8727ba4c2b95126d27808996a5 with delete_type TABLET_DATA_TOMBSTONED (TS 6bb7e1dd53b344f49e42d1587eb3061d not found in new config with opid_index 207) from {username='slave'} at 127.0.0.1:39966
I20250626 02:00:33.131621 22824 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 42a75e8727ba4c2b95126d27808996a5 with cas_config_opid_index 207: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 02:00:33.136391 22836 catalog_manager.cc:5582] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4 reported cstate change: config changed from index 207 to 208, VOTER 129a1a235aea4afd8bb9a66c64645879 (127.16.19.194) evicted. New cstate: current_term: 2 leader_uuid: "8c554ad3849d45fda4e11514db7b2ae4" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "8c554ad3849d45fda4e11514db7b2ae4" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 43387 } health_report { overall_health: HEALTHY } } }
W20250626 02:00:33.141899 22823 catalog_manager.cc:4908] TS 6bb7e1dd53b344f49e42d1587eb3061d (127.16.19.195:46681): delete failed for tablet 42a75e8727ba4c2b95126d27808996a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42a75e8727ba4c2b95126d27808996a5 already in progress: opening tablet
I20250626 02:00:33.185010 23096 tablet_service.cc:1515] Processing DeleteTablet for tablet 42a75e8727ba4c2b95126d27808996a5 with delete_type TABLET_DATA_TOMBSTONED (TS 129a1a235aea4afd8bb9a66c64645879 not found in new config with opid_index 208) from {username='slave'} at 127.0.0.1:57074
I20250626 02:00:33.196328 23334 tablet_replica.cc:331] stopping tablet replica
I20250626 02:00:33.197268 23334 raft_consensus.cc:2241] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 02:00:33.197984 23334 raft_consensus.cc:2270] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:00:33.231860 23334 ts_tablet_manager.cc:1905] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 02:00:33.248134 23334 ts_tablet_manager.cc:1918] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.207
I20250626 02:00:33.248720 23334 log.cc:1199] T 42a75e8727ba4c2b95126d27808996a5 P 129a1a235aea4afd8bb9a66c64645879: Deleting WAL directory at /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-1/wal/wals/42a75e8727ba4c2b95126d27808996a5
I20250626 02:00:33.250782 22825 catalog_manager.cc:4928] TS 129a1a235aea4afd8bb9a66c64645879 (127.16.19.194:34401): tablet 42a75e8727ba4c2b95126d27808996a5 (table pre_rebuild [id=8bbb77ce1dde43b9ae2df9d0b19acc57]) successfully deleted
I20250626 02:00:33.574242 23305 heartbeater.cc:499] Master 127.16.19.254:32875 was elected leader, sending a full tablet report...
I20250626 02:00:33.580598 23232 tablet_service.cc:1515] Processing DeleteTablet for tablet 42a75e8727ba4c2b95126d27808996a5 with delete_type TABLET_DATA_TOMBSTONED (Replica has no consensus available (current committed config index is 208)) from {username='slave'} at 127.0.0.1:39966
W20250626 02:00:33.582751 22823 catalog_manager.cc:4908] TS 6bb7e1dd53b344f49e42d1587eb3061d (127.16.19.195:46681): delete failed for tablet 42a75e8727ba4c2b95126d27808996a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42a75e8727ba4c2b95126d27808996a5 already in progress: opening tablet
I20250626 02:00:34.009398 23232 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 02:00:34.017014 23096 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 02:00:34.066946 22960 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d1c8512529a1431a84bc4888edf347bf | 127.16.19.254:32875 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.16.19.212:42727 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
129a1a235aea4afd8bb9a66c64645879 | 127.16.19.194:34401 | HEALTHY | <none> | 0 | 0
6bb7e1dd53b344f49e42d1587eb3061d | 127.16.19.195:46681 | HEALTHY | <none> | 0 | 0
8c554ad3849d45fda4e11514db7b2ae4 | 127.16.19.193:43387 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.16.19.193 | experimental | 127.16.19.193:43387
local_ip_for_outbound_sockets | 127.16.19.194 | experimental | 127.16.19.194:34401
local_ip_for_outbound_sockets | 127.16.19.195 | experimental | 127.16.19.195:46681
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.19.193:43387
server_dump_info_path | /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.19.194:34401
server_dump_info_path | /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-2/data/info.pb | hidden | 127.16.19.195:46681
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.16.19.212:42727 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------+----+---------+---------------+---------+------------+------------------+-------------
pre_rebuild | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 0
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 1
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250626 02:00:34.478590 16463 log_verifier.cc:126] Checking tablet 42a75e8727ba4c2b95126d27808996a5
I20250626 02:00:35.274533 16463 log_verifier.cc:177] Verified matching terms for 208 ops in tablet 42a75e8727ba4c2b95126d27808996a5
I20250626 02:00:35.277694 22837 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36268:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250626 02:00:35.278388 22837 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36268:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250626 02:00:35.294169 22837 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d1c8512529a1431a84bc4888edf347bf: Sending DeleteTablet for 1 replicas of tablet 42a75e8727ba4c2b95126d27808996a5
I20250626 02:00:35.296422 16463 test_util.cc:276] Using random seed: -312460151
I20250626 02:00:35.296351 22960 tablet_service.cc:1515] Processing DeleteTablet for tablet 42a75e8727ba4c2b95126d27808996a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 02:00:35 UTC) from {username='slave'} at 127.0.0.1:48078
I20250626 02:00:35.313742 23367 tablet_replica.cc:331] stopping tablet replica
I20250626 02:00:35.314738 23367 raft_consensus.cc:2241] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4 [term 2 LEADER]: Raft consensus shutting down.
I20250626 02:00:35.315485 23367 raft_consensus.cc:2270] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 02:00:35.356190 23190 tablet_bootstrap.cc:492] T 42a75e8727ba4c2b95126d27808996a5 P 6bb7e1dd53b344f49e42d1587eb3061d: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 02:00:35.357332 23190 tablet_bootstrap.cc:492] T 42a75e8727ba4c2b95126d27808996a5 P 6bb7e1dd53b344f49e42d1587eb3061d: Bootstrap complete.
I20250626 02:00:35.359149 23190 ts_tablet_manager.cc:1397] T 42a75e8727ba4c2b95126d27808996a5 P 6bb7e1dd53b344f49e42d1587eb3061d: Time spent bootstrapping tablet: real 3.116s user 2.776s sys 0.080s
I20250626 02:00:35.364153 23367 ts_tablet_manager.cc:1905] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 02:00:35.367393 22837 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36296:
name: "post_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 02:00:35.371416 22837 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table post_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 02:00:35.377803 23190 raft_consensus.cc:357] T 42a75e8727ba4c2b95126d27808996a5 P 6bb7e1dd53b344f49e42d1587eb3061d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c554ad3849d45fda4e11514db7b2ae4" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 43387 } } peers { permanent_uuid: "6bb7e1dd53b344f49e42d1587eb3061d" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 46681 } } peers { permanent_uuid: "129a1a235aea4afd8bb9a66c64645879" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 34401 } }
I20250626 02:00:35.379966 23190 raft_consensus.cc:738] T 42a75e8727ba4c2b95126d27808996a5 P 6bb7e1dd53b344f49e42d1587eb3061d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bb7e1dd53b344f49e42d1587eb3061d, State: Initialized, Role: FOLLOWER
I20250626 02:00:35.380949 23190 consensus_queue.cc:260] T 42a75e8727ba4c2b95126d27808996a5 P 6bb7e1dd53b344f49e42d1587eb3061d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c554ad3849d45fda4e11514db7b2ae4" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 43387 } } peers { permanent_uuid: "6bb7e1dd53b344f49e42d1587eb3061d" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 46681 } } peers { permanent_uuid: "129a1a235aea4afd8bb9a66c64645879" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 34401 } }
I20250626 02:00:35.385035 23367 ts_tablet_manager.cc:1918] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.208
I20250626 02:00:35.385607 23367 log.cc:1199] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4: Deleting WAL directory at /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-0/wal/wals/42a75e8727ba4c2b95126d27808996a5
I20250626 02:00:35.386817 23367 ts_tablet_manager.cc:1939] T 42a75e8727ba4c2b95126d27808996a5 P 8c554ad3849d45fda4e11514db7b2ae4: Deleting consensus metadata
I20250626 02:00:35.390424 22824 catalog_manager.cc:4928] TS 8c554ad3849d45fda4e11514db7b2ae4 (127.16.19.193:43387): tablet 42a75e8727ba4c2b95126d27808996a5 (table pre_rebuild [id=8bbb77ce1dde43b9ae2df9d0b19acc57]) successfully deleted
I20250626 02:00:35.395589 23190 ts_tablet_manager.cc:1428] T 42a75e8727ba4c2b95126d27808996a5 P 6bb7e1dd53b344f49e42d1587eb3061d: Time spent starting tablet: real 0.036s user 0.029s sys 0.004s
I20250626 02:00:35.399812 23232 tablet_service.cc:1515] Processing DeleteTablet for tablet 42a75e8727ba4c2b95126d27808996a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 02:00:35 UTC) from {username='slave'} at 127.0.0.1:39966
I20250626 02:00:35.402745 23375 tablet_replica.cc:331] stopping tablet replica
I20250626 02:00:35.403640 23375 raft_consensus.cc:2241] T 42a75e8727ba4c2b95126d27808996a5 P 6bb7e1dd53b344f49e42d1587eb3061d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 02:00:35.406494 23096 tablet_service.cc:1468] Processing CreateTablet for tablet a8316f76224943028d5f632162341c8f (DEFAULT_TABLE table=post_rebuild [id=16d98dd97b874db59adcbdd5d1f431b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 02:00:35.406790 23375 raft_consensus.cc:2270] T 42a75e8727ba4c2b95126d27808996a5 P 6bb7e1dd53b344f49e42d1587eb3061d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 02:00:35.407891 23096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8316f76224943028d5f632162341c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:00:35.408435 22960 tablet_service.cc:1468] Processing CreateTablet for tablet a8316f76224943028d5f632162341c8f (DEFAULT_TABLE table=post_rebuild [id=16d98dd97b874db59adcbdd5d1f431b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 02:00:35.409955 22960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8316f76224943028d5f632162341c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:00:35.421787 23232 tablet_service.cc:1468] Processing CreateTablet for tablet a8316f76224943028d5f632162341c8f (DEFAULT_TABLE table=post_rebuild [id=16d98dd97b874db59adcbdd5d1f431b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 02:00:35.423772 23232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8316f76224943028d5f632162341c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 02:00:35.430886 23376 tablet_bootstrap.cc:492] T a8316f76224943028d5f632162341c8f P 129a1a235aea4afd8bb9a66c64645879: Bootstrap starting.
I20250626 02:00:35.438627 23376 tablet_bootstrap.cc:654] T a8316f76224943028d5f632162341c8f P 129a1a235aea4afd8bb9a66c64645879: Neither blocks nor log segments found. Creating new log.
I20250626 02:00:35.441421 23375 ts_tablet_manager.cc:1905] T 42a75e8727ba4c2b95126d27808996a5 P 6bb7e1dd53b344f49e42d1587eb3061d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 02:00:35.459447 23375 ts_tablet_manager.cc:1918] T 42a75e8727ba4c2b95126d27808996a5 P 6bb7e1dd53b344f49e42d1587eb3061d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.205
I20250626 02:00:35.460076 23375 log.cc:1199] T 42a75e8727ba4c2b95126d27808996a5 P 6bb7e1dd53b344f49e42d1587eb3061d: Deleting WAL directory at /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-2/wal/wals/42a75e8727ba4c2b95126d27808996a5
I20250626 02:00:35.461242 23375 ts_tablet_manager.cc:1939] T 42a75e8727ba4c2b95126d27808996a5 P 6bb7e1dd53b344f49e42d1587eb3061d: Deleting consensus metadata
I20250626 02:00:35.464296 23376 tablet_bootstrap.cc:492] T a8316f76224943028d5f632162341c8f P 129a1a235aea4afd8bb9a66c64645879: No bootstrap required, opened a new log
I20250626 02:00:35.464565 22823 catalog_manager.cc:4928] TS 6bb7e1dd53b344f49e42d1587eb3061d (127.16.19.195:46681): tablet 42a75e8727ba4c2b95126d27808996a5 (table pre_rebuild [id=8bbb77ce1dde43b9ae2df9d0b19acc57]) successfully deleted
I20250626 02:00:35.464823 23376 ts_tablet_manager.cc:1397] T a8316f76224943028d5f632162341c8f P 129a1a235aea4afd8bb9a66c64645879: Time spent bootstrapping tablet: real 0.034s user 0.015s sys 0.017s
I20250626 02:00:35.467764 23376 raft_consensus.cc:357] T a8316f76224943028d5f632162341c8f P 129a1a235aea4afd8bb9a66c64645879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a1a235aea4afd8bb9a66c64645879" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 34401 } } peers { permanent_uuid: "8c554ad3849d45fda4e11514db7b2ae4" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 43387 } } peers { permanent_uuid: "6bb7e1dd53b344f49e42d1587eb3061d" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 46681 } }
I20250626 02:00:35.468546 23376 raft_consensus.cc:383] T a8316f76224943028d5f632162341c8f P 129a1a235aea4afd8bb9a66c64645879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:00:35.468919 23376 raft_consensus.cc:738] T a8316f76224943028d5f632162341c8f P 129a1a235aea4afd8bb9a66c64645879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129a1a235aea4afd8bb9a66c64645879, State: Initialized, Role: FOLLOWER
I20250626 02:00:35.469820 23376 consensus_queue.cc:260] T a8316f76224943028d5f632162341c8f P 129a1a235aea4afd8bb9a66c64645879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a1a235aea4afd8bb9a66c64645879" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 34401 } } peers { permanent_uuid: "8c554ad3849d45fda4e11514db7b2ae4" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 43387 } } peers { permanent_uuid: "6bb7e1dd53b344f49e42d1587eb3061d" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 46681 } }
I20250626 02:00:35.473750 23190 tablet_bootstrap.cc:492] T a8316f76224943028d5f632162341c8f P 6bb7e1dd53b344f49e42d1587eb3061d: Bootstrap starting.
I20250626 02:00:35.478215 23376 ts_tablet_manager.cc:1428] T a8316f76224943028d5f632162341c8f P 129a1a235aea4afd8bb9a66c64645879: Time spent starting tablet: real 0.013s user 0.007s sys 0.003s
I20250626 02:00:35.479373 23190 tablet_bootstrap.cc:654] T a8316f76224943028d5f632162341c8f P 6bb7e1dd53b344f49e42d1587eb3061d: Neither blocks nor log segments found. Creating new log.
I20250626 02:00:35.480134 23379 tablet_bootstrap.cc:492] T a8316f76224943028d5f632162341c8f P 8c554ad3849d45fda4e11514db7b2ae4: Bootstrap starting.
I20250626 02:00:35.486651 23190 tablet_bootstrap.cc:492] T a8316f76224943028d5f632162341c8f P 6bb7e1dd53b344f49e42d1587eb3061d: No bootstrap required, opened a new log
I20250626 02:00:35.487180 23190 ts_tablet_manager.cc:1397] T a8316f76224943028d5f632162341c8f P 6bb7e1dd53b344f49e42d1587eb3061d: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20250626 02:00:35.488963 23190 raft_consensus.cc:357] T a8316f76224943028d5f632162341c8f P 6bb7e1dd53b344f49e42d1587eb3061d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a1a235aea4afd8bb9a66c64645879" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 34401 } } peers { permanent_uuid: "8c554ad3849d45fda4e11514db7b2ae4" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 43387 } } peers { permanent_uuid: "6bb7e1dd53b344f49e42d1587eb3061d" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 46681 } }
I20250626 02:00:35.489394 23190 raft_consensus.cc:383] T a8316f76224943028d5f632162341c8f P 6bb7e1dd53b344f49e42d1587eb3061d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:00:35.489646 23190 raft_consensus.cc:738] T a8316f76224943028d5f632162341c8f P 6bb7e1dd53b344f49e42d1587eb3061d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bb7e1dd53b344f49e42d1587eb3061d, State: Initialized, Role: FOLLOWER
I20250626 02:00:35.490124 23190 consensus_queue.cc:260] T a8316f76224943028d5f632162341c8f P 6bb7e1dd53b344f49e42d1587eb3061d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a1a235aea4afd8bb9a66c64645879" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 34401 } } peers { permanent_uuid: "8c554ad3849d45fda4e11514db7b2ae4" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 43387 } } peers { permanent_uuid: "6bb7e1dd53b344f49e42d1587eb3061d" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 46681 } }
I20250626 02:00:35.491701 23190 ts_tablet_manager.cc:1428] T a8316f76224943028d5f632162341c8f P 6bb7e1dd53b344f49e42d1587eb3061d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250626 02:00:35.507308 23379 tablet_bootstrap.cc:654] T a8316f76224943028d5f632162341c8f P 8c554ad3849d45fda4e11514db7b2ae4: Neither blocks nor log segments found. Creating new log.
I20250626 02:00:35.512877 23379 tablet_bootstrap.cc:492] T a8316f76224943028d5f632162341c8f P 8c554ad3849d45fda4e11514db7b2ae4: No bootstrap required, opened a new log
I20250626 02:00:35.513933 23379 ts_tablet_manager.cc:1397] T a8316f76224943028d5f632162341c8f P 8c554ad3849d45fda4e11514db7b2ae4: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.011s
I20250626 02:00:35.517474 23379 raft_consensus.cc:357] T a8316f76224943028d5f632162341c8f P 8c554ad3849d45fda4e11514db7b2ae4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a1a235aea4afd8bb9a66c64645879" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 34401 } } peers { permanent_uuid: "8c554ad3849d45fda4e11514db7b2ae4" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 43387 } } peers { permanent_uuid: "6bb7e1dd53b344f49e42d1587eb3061d" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 46681 } }
I20250626 02:00:35.518333 23379 raft_consensus.cc:383] T a8316f76224943028d5f632162341c8f P 8c554ad3849d45fda4e11514db7b2ae4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 02:00:35.518603 23379 raft_consensus.cc:738] T a8316f76224943028d5f632162341c8f P 8c554ad3849d45fda4e11514db7b2ae4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c554ad3849d45fda4e11514db7b2ae4, State: Initialized, Role: FOLLOWER
I20250626 02:00:35.519296 23379 consensus_queue.cc:260] T a8316f76224943028d5f632162341c8f P 8c554ad3849d45fda4e11514db7b2ae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a1a235aea4afd8bb9a66c64645879" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 34401 } } peers { permanent_uuid: "8c554ad3849d45fda4e11514db7b2ae4" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 43387 } } peers { permanent_uuid: "6bb7e1dd53b344f49e42d1587eb3061d" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 46681 } }
I20250626 02:00:35.523173 23379 ts_tablet_manager.cc:1428] T a8316f76224943028d5f632162341c8f P 8c554ad3849d45fda4e11514db7b2ae4: Time spent starting tablet: real 0.009s user 0.000s sys 0.007s
W20250626 02:00:35.523839 23162 tablet.cc:2378] T a8316f76224943028d5f632162341c8f P 129a1a235aea4afd8bb9a66c64645879: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250626 02:00:35.540761 23306 tablet.cc:2378] T a8316f76224943028d5f632162341c8f P 6bb7e1dd53b344f49e42d1587eb3061d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250626 02:00:35.612113 23027 tablet.cc:2378] T a8316f76224943028d5f632162341c8f P 8c554ad3849d45fda4e11514db7b2ae4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 02:00:35.634991 23374 raft_consensus.cc:491] T a8316f76224943028d5f632162341c8f P 6bb7e1dd53b344f49e42d1587eb3061d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 02:00:35.635480 23374 raft_consensus.cc:513] T a8316f76224943028d5f632162341c8f P 6bb7e1dd53b344f49e42d1587eb3061d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a1a235aea4afd8bb9a66c64645879" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 34401 } } peers { permanent_uuid: "8c554ad3849d45fda4e11514db7b2ae4" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 43387 } } peers { permanent_uuid: "6bb7e1dd53b344f49e42d1587eb3061d" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 46681 } }
I20250626 02:00:35.637837 23374 leader_election.cc:290] T a8316f76224943028d5f632162341c8f P 6bb7e1dd53b344f49e42d1587eb3061d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 129a1a235aea4afd8bb9a66c64645879 (127.16.19.194:34401), 8c554ad3849d45fda4e11514db7b2ae4 (127.16.19.193:43387)
I20250626 02:00:35.653486 23115 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8316f76224943028d5f632162341c8f" candidate_uuid: "6bb7e1dd53b344f49e42d1587eb3061d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "129a1a235aea4afd8bb9a66c64645879" is_pre_election: true
I20250626 02:00:35.653757 22980 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8316f76224943028d5f632162341c8f" candidate_uuid: "6bb7e1dd53b344f49e42d1587eb3061d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c554ad3849d45fda4e11514db7b2ae4" is_pre_election: true
I20250626 02:00:35.654013 23115 raft_consensus.cc:2466] T a8316f76224943028d5f632162341c8f P 129a1a235aea4afd8bb9a66c64645879 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bb7e1dd53b344f49e42d1587eb3061d in term 0.
I20250626 02:00:35.654481 22980 raft_consensus.cc:2466] T a8316f76224943028d5f632162341c8f P 8c554ad3849d45fda4e11514db7b2ae4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bb7e1dd53b344f49e42d1587eb3061d in term 0.
I20250626 02:00:35.655104 23187 leader_election.cc:304] T a8316f76224943028d5f632162341c8f P 6bb7e1dd53b344f49e42d1587eb3061d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 129a1a235aea4afd8bb9a66c64645879, 6bb7e1dd53b344f49e42d1587eb3061d; no voters:
I20250626 02:00:35.655989 23374 raft_consensus.cc:2802] T a8316f76224943028d5f632162341c8f P 6bb7e1dd53b344f49e42d1587eb3061d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 02:00:35.656267 23374 raft_consensus.cc:491] T a8316f76224943028d5f632162341c8f P 6bb7e1dd53b344f49e42d1587eb3061d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 02:00:35.656526 23374 raft_consensus.cc:3058] T a8316f76224943028d5f632162341c8f P 6bb7e1dd53b344f49e42d1587eb3061d [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:00:35.661052 23374 raft_consensus.cc:513] T a8316f76224943028d5f632162341c8f P 6bb7e1dd53b344f49e42d1587eb3061d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a1a235aea4afd8bb9a66c64645879" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 34401 } } peers { permanent_uuid: "8c554ad3849d45fda4e11514db7b2ae4" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 43387 } } peers { permanent_uuid: "6bb7e1dd53b344f49e42d1587eb3061d" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 46681 } }
I20250626 02:00:35.662550 23374 leader_election.cc:290] T a8316f76224943028d5f632162341c8f P 6bb7e1dd53b344f49e42d1587eb3061d [CANDIDATE]: Term 1 election: Requested vote from peers 129a1a235aea4afd8bb9a66c64645879 (127.16.19.194:34401), 8c554ad3849d45fda4e11514db7b2ae4 (127.16.19.193:43387)
I20250626 02:00:35.663496 23115 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8316f76224943028d5f632162341c8f" candidate_uuid: "6bb7e1dd53b344f49e42d1587eb3061d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "129a1a235aea4afd8bb9a66c64645879"
I20250626 02:00:35.663532 22980 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8316f76224943028d5f632162341c8f" candidate_uuid: "6bb7e1dd53b344f49e42d1587eb3061d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c554ad3849d45fda4e11514db7b2ae4"
I20250626 02:00:35.663938 23115 raft_consensus.cc:3058] T a8316f76224943028d5f632162341c8f P 129a1a235aea4afd8bb9a66c64645879 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:00:35.663942 22980 raft_consensus.cc:3058] T a8316f76224943028d5f632162341c8f P 8c554ad3849d45fda4e11514db7b2ae4 [term 0 FOLLOWER]: Advancing to term 1
I20250626 02:00:35.668453 23115 raft_consensus.cc:2466] T a8316f76224943028d5f632162341c8f P 129a1a235aea4afd8bb9a66c64645879 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bb7e1dd53b344f49e42d1587eb3061d in term 1.
I20250626 02:00:35.668977 22980 raft_consensus.cc:2466] T a8316f76224943028d5f632162341c8f P 8c554ad3849d45fda4e11514db7b2ae4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bb7e1dd53b344f49e42d1587eb3061d in term 1.
I20250626 02:00:35.669406 23187 leader_election.cc:304] T a8316f76224943028d5f632162341c8f P 6bb7e1dd53b344f49e42d1587eb3061d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 129a1a235aea4afd8bb9a66c64645879, 6bb7e1dd53b344f49e42d1587eb3061d; no voters:
I20250626 02:00:35.670159 23374 raft_consensus.cc:2802] T a8316f76224943028d5f632162341c8f P 6bb7e1dd53b344f49e42d1587eb3061d [term 1 FOLLOWER]: Leader election won for term 1
I20250626 02:00:35.671725 23374 raft_consensus.cc:695] T a8316f76224943028d5f632162341c8f P 6bb7e1dd53b344f49e42d1587eb3061d [term 1 LEADER]: Becoming Leader. State: Replica: 6bb7e1dd53b344f49e42d1587eb3061d, State: Running, Role: LEADER
I20250626 02:00:35.672431 23374 consensus_queue.cc:237] T a8316f76224943028d5f632162341c8f P 6bb7e1dd53b344f49e42d1587eb3061d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a1a235aea4afd8bb9a66c64645879" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 34401 } } peers { permanent_uuid: "8c554ad3849d45fda4e11514db7b2ae4" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 43387 } } peers { permanent_uuid: "6bb7e1dd53b344f49e42d1587eb3061d" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 46681 } }
I20250626 02:00:35.682353 22837 catalog_manager.cc:5582] T a8316f76224943028d5f632162341c8f P 6bb7e1dd53b344f49e42d1587eb3061d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bb7e1dd53b344f49e42d1587eb3061d (127.16.19.195). New cstate: current_term: 1 leader_uuid: "6bb7e1dd53b344f49e42d1587eb3061d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129a1a235aea4afd8bb9a66c64645879" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 34401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c554ad3849d45fda4e11514db7b2ae4" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 43387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bb7e1dd53b344f49e42d1587eb3061d" member_type: VOTER last_known_addr { host: "127.16.19.195" port: 46681 } health_report { overall_health: HEALTHY } } }
I20250626 02:00:35.815460 22980 raft_consensus.cc:1273] T a8316f76224943028d5f632162341c8f P 8c554ad3849d45fda4e11514db7b2ae4 [term 1 FOLLOWER]: Refusing update from remote peer 6bb7e1dd53b344f49e42d1587eb3061d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:00:35.815462 23115 raft_consensus.cc:1273] T a8316f76224943028d5f632162341c8f P 129a1a235aea4afd8bb9a66c64645879 [term 1 FOLLOWER]: Refusing update from remote peer 6bb7e1dd53b344f49e42d1587eb3061d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 02:00:35.817351 23388 consensus_queue.cc:1035] T a8316f76224943028d5f632162341c8f P 6bb7e1dd53b344f49e42d1587eb3061d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c554ad3849d45fda4e11514db7b2ae4" member_type: VOTER last_known_addr { host: "127.16.19.193" port: 43387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 02:00:35.818135 23374 consensus_queue.cc:1035] T a8316f76224943028d5f632162341c8f P 6bb7e1dd53b344f49e42d1587eb3061d [LEADER]: Connected to new peer: Peer: permanent_uuid: "129a1a235aea4afd8bb9a66c64645879" member_type: VOTER last_known_addr { host: "127.16.19.194" port: 34401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 02:00:35.851096 23397 mvcc.cc:204] Tried to move back new op lower bound from 7171699653884985344 to 7171699653320527872. Current Snapshot: MvccSnapshot[applied={T|T < 7171699653884985344}]
W20250626 02:00:37.337669 23157 debug-util.cc:398] Leaking SignalData structure 0x7b08000c6a80 after lost signal to thread 23031
W20250626 02:00:37.338759 23157 debug-util.cc:398] Leaking SignalData structure 0x7b08000c7020 after lost signal to thread 23160
I20250626 02:00:41.794957 23232 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250626 02:00:41.795565 23096 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250626 02:00:41.799958 22960 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d1c8512529a1431a84bc4888edf347bf | 127.16.19.254:32875 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.16.19.212:42727 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
129a1a235aea4afd8bb9a66c64645879 | 127.16.19.194:34401 | HEALTHY | <none> | 0 | 0
6bb7e1dd53b344f49e42d1587eb3061d | 127.16.19.195:46681 | HEALTHY | <none> | 1 | 0
8c554ad3849d45fda4e11514db7b2ae4 | 127.16.19.193:43387 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.16.19.193 | experimental | 127.16.19.193:43387
local_ip_for_outbound_sockets | 127.16.19.194 | experimental | 127.16.19.194:34401
local_ip_for_outbound_sockets | 127.16.19.195 | experimental | 127.16.19.195:46681
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-0/data/info.pb | hidden | 127.16.19.193:43387
server_dump_info_path | /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-1/data/info.pb | hidden | 127.16.19.194:34401
server_dump_info_path | /tmp/dist-test-taskifDKOX/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750903057700330-16463-0/minicluster-data/ts-2/data/info.pb | hidden | 127.16.19.195:46681
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.16.19.212:42727 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
--------------+----+---------+---------------+---------+------------+------------------+-------------
post_rebuild | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250626 02:00:42.013103 16463 log_verifier.cc:126] Checking tablet 42a75e8727ba4c2b95126d27808996a5
I20250626 02:00:42.013530 16463 log_verifier.cc:177] Verified matching terms for 0 ops in tablet 42a75e8727ba4c2b95126d27808996a5
I20250626 02:00:42.013702 16463 log_verifier.cc:126] Checking tablet a8316f76224943028d5f632162341c8f
W20250626 02:00:42.306610 23021 debug-util.cc:398] Leaking SignalData structure 0x7b08000b90c0 after lost signal to thread 22896
I20250626 02:00:42.850370 16463 log_verifier.cc:177] Verified matching terms for 205 ops in tablet a8316f76224943028d5f632162341c8f
I20250626 02:00:42.878847 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 22875
I20250626 02:00:42.923576 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 23026
W20250626 02:00:42.962980 22824 connection.cc:537] server connection from 127.16.19.194:34169 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250626 02:00:42.963495 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 23165
I20250626 02:00:43.009199 16463 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskifDKOX/build/tsan/bin/kudu with pid 22805
2025-06-26T02:00:43Z chronyd exiting
[ OK ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0 (36951 ms)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest (36951 ms total)
[----------] Global test environment tear-down
[==========] 9 tests from 5 test suites ran. (185295 ms total)
[ PASSED ] 8 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AdminCliTest.TestRebuildTables
1 FAILED TEST